Interface control testing method and apparatus

Abstract

The application provides an interface control test method and device. The method comprises the following steps: obtaining a character and an operation type of a to-be-tested control of a to-be-tested interface; searching for a character of a target control matched with the character of the to-be-tested control in an associated library of the to-be-tested interface, if the character of the target control is found, obtaining a coordinate of the to-be-tested control, the associated library comprising an association relationship between a character and a coordinate of a control in the to-be-tested interface, and the coordinate of the to-be-tested control being the same as that of the target control; if the character of the target control is not found, performing character recognition on the to-be-tested interface based on the character of the to-be-tested control to obtain the coordinate of the to-be-tested control; and performing a test operation in the to-be-tested interface based on the coordinate of the to-be-tested control and the operation type to obtain a first operation result. The interface control test is realized by replacing a picture with a character, the influence of a control style is reduced, and the coordinate of the interface control can be recognized at a high speed based on character recognition.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a method and device for testing interface controls. Background Technology

[0002] Currently, applications developed using graphical form tools, such as those based on Java Swing, often lack a unique identifier for their internal controls—a handle (a unique identifier for an object or data structure within the operating system). Due to this lack of handles, these applications cannot obtain or generate window handles by calling the Windows API, thus hindering the ability to send message events to specific controls. Consequently, ensuring operational accuracy is difficult during GUI (Graphical User Interface) testing.

[0003] Currently, each operation step of the application is abstracted, and the application is opened in advance in the test execution machine to record screenshots of the scene library. Then, during automated testing, by comparing the image corresponding to the target operation with the real-time screenshot of the current interface, the coordinates of the successfully matched area are returned, and the internal controls of the application are identified and clicked based on the coordinates.

[0004] However, the above method requires 100% matching in the image comparison process, which is easily affected by resolution and control background.

[0005] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0006] This application provides a method and device for testing interface controls. By recognizing the coordinates of interface controls through character recognition, the recognition speed can be improved.

[0007] In a first aspect, embodiments of this application provide a method for testing interface controls, comprising: obtaining the characters and operation types of the control to be tested on the interface to be tested; searching in an association library of the interface to be tested based on the characters of the control to be tested; if a character of a target control matching the characters of the control to be tested is found in the association library, then obtaining the coordinates of the control to be tested; the association library includes the association relationship between the characters and coordinates of controls in the interface to be tested, and the coordinates of the control to be tested are the same as the coordinates of the target control; if no character of a target control matching the characters of the control to be tested is found in the association library, then character recognition is performed on the interface to be tested based on the characters of the control to be tested to obtain the coordinates of the control to be tested; and performing a test operation on the interface to be tested based on the coordinates of the control to be tested and the operation type of the control to be tested to obtain a first operation result.

[0008] In this solution, UI control testing is achieved by replacing images with characters, which reduces the impact of control styles on the image recognition process. In addition, recognizing the coordinates of UI controls based on characters can improve the recognition speed.

[0009] In one possible implementation, the method further includes: obtaining the expected operation result of the control to be tested on the interface to be tested; when the first operation result is inconsistent with the expected operation result, performing character recognition on the interface to be tested based on the characters of the control to be tested, and determining the corrected coordinates of the control to be tested in the interface to be tested; and replacing the coordinates of the target control in the associated library with the corrected coordinates.

[0010] In this solution, the associated database is automatically maintained by updating the coordinates.

[0011] In one possible implementation, character recognition is performed on the interface under test based on the characters of the control under test to obtain the coordinates of the control under test, including: when multiple candidate coordinates are obtained by character recognition on the interface under test based on the characters of the control under test, the coordinates of the control under test are determined from the multiple candidate coordinates.

[0012] In this solution, when the position of the character of the control to be tested on the interface to be tested is not unique, the true position of the control to be tested can be determined from multiple positions.

[0013] In one example, determining the coordinates of the control to be tested from multiple candidate coordinates includes: determining extended characters; wherein the extended characters include the characters of the control to be tested and other characters besides them; performing character recognition on the first interface based on the extended characters to determine the coordinates of the control to be tested from multiple candidate coordinates.

[0014] In this solution, when the position of the characters of the control under test on the interface under test is not unique, the position of the control under test on the interface under test can be accurately located by enlarging the characters of the control under test.

[0015] In one example, determining the coordinates of the control under test from multiple candidate coordinates includes: determining a user action; wherein the user action indicates the selection of multiple candidate coordinates; and in response to the user action, using the candidate coordinates selected by the user as the coordinates of the control under test.

[0016] In this solution, a manual selection method is used to accurately locate the actual position of the control to be tested on the interface to be tested.

[0017] In one possible implementation, the characters of the control under test include the tooltip characters of the control under test displayed on the interface under test, and the first coordinate indicates the position of the tooltip characters of the control under test on the interface under test.

[0018] In one example, the text for the control under test also includes the tooltip text for the area where the control is located.

[0019] In this solution, considering that the position of the prompt characters for operable characters in the interface under test is not unique, in order to make the character identifier of operable controls unique, additional characters can be added to the prompt characters of operable characters to achieve unique identification.

[0020] In one possible implementation, the method further includes: if no character matching the character of the target control to be tested is found in the association library, character recognition is performed on the interface to be tested based on the character of the control to be tested to obtain the coordinates of the control to be tested.

[0021] In one example, the characters and coordinates of the control to be tested are associated and added to the associated library.

[0022] In this solution, newly appearing characters can be added to the association database, enabling the database to be updated and maintained.

[0023] Secondly, embodiments of the present invention provide an interface control testing device, which includes several modules. Each module is used to execute various steps in the display method provided in the first aspect of the present invention. The division of modules is not limited here. For the specific functions performed by each module of the display device and the beneficial effects achieved, please refer to the functions of each step in the display method provided in the first aspect of the present invention; further details will not be repeated here.

[0024] For example, an embodiment of this application provides a UI control testing apparatus, including:

[0025] The acquisition module is used to acquire the characters and operation types of the controls to be tested on the interface under test;

[0026] The search module is used to search the association library of the interface under test based on the characters of the control under test. If the characters of the target control that match the characters of the control under test are found in the association library, the coordinates of the control under test are obtained. The association library includes the association relationship between the characters and coordinates of the controls in the interface under test. The coordinates of the control under test are the same as the coordinates of the target control.

[0027] The recognition module is used to perform character recognition on the interface under test based on the characters of the target control if no character matching the character of the target control is found in the association library, and obtain the coordinates of the control under test.

[0028] The operation test module is used to perform test operations on the interface under test based on the coordinates of the control under test and the operation type of the control under test, and obtain the first operation result.

[0029] In one possible implementation, the device further includes:

[0030] The recognition module is used to recognize characters in the interface under test based on the characters of the control under test when the result of the first operation is inconsistent with the expected result of the operation, and to determine the corrected coordinates of the control under test in the interface under test.

[0031] The update module is used to replace the coordinates of the target control in the associated library with the corrected coordinates.

[0032] In one possible implementation, the recognition module is used to determine the coordinates of the control under test from the multiple candidate coordinates when character recognition of the interface under test is performed based on the characters of the control under test.

[0033] In one example, the recognition module includes:

[0034] A character extension unit is used to determine the extended characters; wherein, the extended characters include the characters of the control to be tested and other characters besides them;

[0035] The recognition and selection unit is used to perform character recognition on the first interface based on extended characters and determine the coordinates of the control to be tested from multiple candidate coordinates.

[0036] In one example, the recognition module includes:

[0037] An operation determination unit is used to determine a user operation; wherein the user operation indicates the selection of multiple candidate coordinates;

[0038] The position determination unit is used to respond to user operations and use the candidate coordinates selected by the user as the coordinates of the control to be tested.

[0039] In one possible implementation, the characters of the control under test include the tooltip characters of the control under test displayed on the interface under test, and the first coordinate indicates the position of the tooltip characters of the control under test on the interface under test.

[0040] In one example, the text for the control under test also includes the tooltip text for the area where the control is located.

[0041] In one possible implementation, the recognition module is further used to perform character recognition on the interface under test based on the characters of the control under test, and obtain the coordinates of the control under test, if no character matching the character of the target control is found in the association library.

[0042] In one example, the device also includes:

[0043] The add module is used to associate the characters and coordinates of the control to be tested and add them to the association library.

[0044] Thirdly, embodiments of this application provide an interface control testing apparatus, characterized in that the apparatus executes computer program instructions to perform the method provided in the first aspect. For example, the apparatus may be a chip or a processor.

[0045] In one example, the device may include a processor that can be coupled to memory, read instructions from the memory, and execute the methods provided in the first aspect according to those instructions. The memory may be integrated into the chip or processor, or it may be independent of the chip or processor.

[0046] Fourthly, embodiments of this application provide an interface control testing device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method provided in the first aspect.

[0047] Fifthly, embodiments of this application provide a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method provided in the first aspect.

[0048] In a sixth aspect, embodiments of this application provide a computer program product containing instructions that, when executed on a computer, cause the computer to perform the method provided in the first aspect. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of a process for testing an interface control provided in an embodiment of this application;

[0050] Figure 2 This is a schematic diagram of a first interface provided in an embodiment of this application. Figure 1 ;

[0051] Figure 3 This is a schematic diagram of a first interface provided in an embodiment of this application. Figure 2 ;

[0052] Figure 4a yes Figure 3 Display illustration of interface changes Figure 1 ;

[0053] Figure 4b yes Figure 3 Display illustration of interface changes Figure 2 ;

[0054] Figure 5 yes Figure 1 Flowchart of step 107 Figure 1 ;

[0055] Figure 6 yes Figure 1 Flowchart of step 107 Figure 2 ;

[0056] Figure 7 This is a flowchart illustrating another interface control testing method provided in an embodiment of this application;

[0057] Figure 8a This is a schematic diagram of a specific example of a UI control testing method provided in an embodiment of the present invention. Figure 1 ;

[0058] Figure 8b This is a schematic diagram of a specific example of a UI control testing method provided in an embodiment of the present invention. Figure 2 ;

[0059] Figure 9 This is a schematic diagram of the structure of an interface control testing device provided in an embodiment of the present invention;

[0060] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.

[0062] In the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0063] In the description of the embodiments in this application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple terminals refer to two or more terminals.

[0064] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0065] Current UI control recognition solutions have the following drawbacks:

[0066] 1. The image comparison process requires 100% matching and is easily affected by resolution and control background.

[0067] 2. The update frequency of the image scene library changes with the version of the application. When the size of the control, text style, background style, etc. change, it is necessary to take new screenshots and update them to the scene library. If compatibility with the old version is required, the scene library will become more and more bloated, and the maintenance cost will become higher and higher.

[0068] Based on this, the present invention proposes a method for testing interface controls.

[0069] This method improves recognition speed by recognizing the coordinates of UI controls through character recognition. This is only a brief overview of the method; for more details, please see the description below.

[0070] Next, a detailed description of a UI control testing method provided by an embodiment of the present invention will be given.

[0071] Figure 1This is a flowchart illustrating the interface control testing method provided in this application. This embodiment can be applied to electronic devices, specifically servers or general computers. Figure 1 As shown, the interface control testing method provided in this application embodiment includes at least the following steps:

[0072] Step 101: Establish the association library for the interface under test; wherein, the association library includes: the association relationship between the characters of operable controls and the coordinates of operable controls; wherein, operable controls are controls displayed in the interface under test of the target application (or application, etc.), the characters of operable controls are the characters (or keywords, etc.) displayed by the control on the interface under test, and the coordinates of operable controls are the positions or coordinates of the operable controls on the interface under test, etc.

[0073] The target application includes multiple application interfaces. Each application interface includes several operable controls (which can also be called controls). Additionally, each application interface can display the operable controls and the characters associated with those controls. These displayed characters can be understood as character arrays, representing either a single character or a string of multiple characters. For example, such as... Figure 2 As shown, assuming the operable control is an account input box, and the characters on this operable control are "Please enter your account" and / or "Account", the application interface can display this input box, showing "Please enter your account" within the input box, and displaying "Account" or similar characters outside the input box. For example, such as... Figure 3 As shown, assuming the operable control is a "Learn More" button, the text displayed on the application interface will be "Learn More".

[0074] Any application interface of the target application can be used as the interface to be tested, and a relational library for the interface to be tested can be built. This relational library includes the association relationship between the characters of the operable controls in the interface to be tested and the coordinates of the operable controls.

[0075] The characters of the operable control may include the tooltip characters of the operable control. Optionally, the tooltip characters may be some or all of the characters displayed by the operable control on the interface under test.

[0076] The coordinates of the operable control are used to represent the position of the operable control in the interface under test. In one example, it can be the area where the prompt text of the operable control is located.

[0077] It's worth noting that the association library can store data using a tuple of [character of the operable control, coordinates of the operable control]. Alternatively, it can store data using a triple of [interface identifier of the interface where the operable control is located, character of the operable control, coordinates of the operable control]. The coordinates in the triple indicate the area where the prompt character of the operable control is located within the interface indicated by the interface identifier. Here, the interface identifier is used to indicate the interface under test. In practical applications, key-value pairs can also be used to record the character and coordinates of the operable control. Here, the key represents the character of the operable control, and the value represents the coordinates of the operable control.

[0078] It should be noted that when the tooltip of an operable control in the interface uniquely identifies the operable control, for example, Figure 2 The example shown includes a login button, a username input field, and a password input field. In this case, the characters on the operable controls can be prompt characters. However, when the prompt character of an operable control in the interface is not a unique identifier for that control, for example... Figure 3 Three "Learn More" buttons are shown; at this point, the characters of the operable controls also need to include associated characters of the tooltip characters of the operable controls. Here, the associated characters are used to uniquely distinguish different operable controls and to indicate the tooltip characters of the operable controls in the area of ​​the interface under test. For example, for... Figure 3 Three "Learn More" buttons are shown, with the associated text being "Hot Recommendations," "Recommended Products," and "Success Stories," respectively. This results in three separate phrases: "Learn More Through Hot Recommendations," "Learn More Through Recommended Products," and "Learn More Through Success Stories," thus distinguishing the "Learn More" buttons in three different locations. For example... Figure 3 The code shows two "View Details" buttons, associated with the characters "Product A" and "Success Case," resulting in the two text options: "View Details for Product A" and "View Details for Success Case." Here, because the prompt and associated characters are not consecutive but spaced, they can be distinguished using the spacebar, allowing for the identification of the key and prompt characters.

[0079] It is worth noting that the associated library includes characters for multiple operable controls, and each character is unique.

[0080] According to one feasible implementation method, all characters in the association library can be prompt characters. This allows for the identification of the prompt characters and coordinates of operable controls in the target application's interface under test, constructing an association library in key-value pairs. It is important to note that the prompt characters and coordinates of operable controls in the target application's interface under test are unique.

[0081] Step 102: Obtain the characters and operation types of the controls to be tested on the interface to be tested.

[0082] In step 102, the acquisition methods may include, but are not limited to, manual input or parsing of the test script. In practical applications, test cases for the interface under test need to be constructed. Test cases may include one or more controls of the application under test, as well as the characters and operation types of each control. Test cases can be pre-written scripts, received via network transmission, or retrieved from storage or a database. Afterward, based on the test cases, the controls and operation types of the interface under test can be obtained and tested.

[0083] The operation type can include, but is not limited to, a combination of one or more operations such as clicking, long-pressing and swiping, inputting, dragging, copying, and shortcut keys. This operation type can represent the operation used by the user in the actual process.

[0084] Step 103: Based on the characters of the control to be tested, search in the association library of the interface to be tested. If the characters of the target control that match the characters of the control to be tested are found in the association library, determine the coordinates of the control to be tested. The coordinates of the control to be tested are the same as the coordinates of the target control. The target control is an operable control with the same characters as the control to be tested.

[0085] According to one feasible implementation, if the characters of the control under test match the characters of the target control, it can be considered that the characters of the control under test are at least partially or completely the same as the characters of the target control. For example, if the characters of the control under test are completely identical to the characters of the target control, then the characters of the control under test are considered to match the characters of the target control.

[0086] Step 104: Based on the coordinates of the control to be tested and the operation type of the control to be tested, perform the test operation in the interface to be tested and obtain the first operation result.

[0087] For example, a test operation may include positioning the screen cursor at the coordinates of the control to be tested, and performing a test operation on the cursor according to the operation type of the control to be tested, such as clicking, dragging, inputting, copying, and / or shortcut key operations or combinations thereof.

[0088] According to a feasible implementation method, the area indicated by the coordinates of the control under test can be horizontally expanded to obtain the operation area in the interface under test. Here, by expanding the coordinates of the control under test, the possibility of accidental operation is minimized. Furthermore, the test operation indicated by the operation type is executed in the operation area of ​​the interface under test to obtain the first operation result.

[0089] like Figure 2As shown, assuming the character of the control to be tested is "Login" and the operation type is "Click", the first operation result can be the interface identifier of the application interface after the jump.

[0090] like Figure 3 As shown, assuming the text of the control to be tested is "Hot Recommendations Learn More" and the operation type is click, during the test operation, if the "Learn More" operation is clicked under "Hot Recommendations", the operation result can be the interface identifier of the application interface after the jump.

[0091] like Figure 3 As shown, assuming the text of the control to be tested is "View details for product A" and the operation type is click, during the test operation, if the click operation of "Learn more" under product A is performed, the operation result can be the interface identifier of the redirected interface.

[0092] As shown in 4a, when the mouse clicks on product b, the area of ​​product b is enlarged to display "View Details," while product a no longer displays "View Details." Assuming the text of the control under test is "Product b View Details" and the operation type is click, then in the specific test, assume the interface under test is... Figure 3 The interface shown requires you to first move the mouse to the area where product b is located during testing, and then perform the test operation, i.e., click "View Details" under product b. The result of the operation will be the interface icon of the application interface after the jump.

[0093] As shown in 4b, when the mouse clicks on image B, image A switches to image B, and image A no longer displays "View Details". Assuming the text of the control under test is "View Details for Image B" and the operation type is click, then in the specific test, assume the interface under test is... Figure 3 The interface shown requires you to first move the mouse to the area where image B is located and click it during the test operation. After image A is replaced with image B, you can perform the test operation by clicking "View Details". The first result of the operation can be the interface icon of the application interface after the jump.

[0094] In this embodiment of the invention, the testing of interface controls is achieved by replacing images with characters, thereby reducing the impact of control styles on the image recognition process.

[0095] In one possible scenario, the test case includes the expected operation result for each control to be tested. When the operation result of the control to be tested is inconsistent with the expected operation result, it indicates that the coordinates of the operable control may be incorrect. In this case, it is necessary to correct the coordinates associated with the target control in the association library.

[0096] This invention provides another method for testing interface controls. Figure 1 This illustration shows a flowchart of another interface control testing method provided in an embodiment of this application. Figure 1 As shown, in Figure 1 Based on steps 101 to 104 shown, this embodiment of the application further includes at least the following steps:

[0097] Step 105: When the result of the first operation is inconsistent with the expected result of the operation, character recognition is performed on the interface under test based on the characters of the control to be tested, and the candidate coordinates of the control to be tested on the interface under test are determined.

[0098] Assuming the first operation result is the interface identifier of the redirected interface, and the expected operation result is the interface identifier of the redirected interface, if the interface identifiers in the first operation result and the expected operation result are different, or if the first operation result does not contain the interface identifier of the redirected interface, then the first operation result and the expected operation result can be considered inconsistent.

[0099] like Figure 2 As shown, assuming the character of the control to be tested is "Login", the expected operation result is to jump to another page. If the interface identifiers in the first operation result and the expected operation result are different, then it can be considered that the first operation result and the expected operation result are inconsistent.

[0100] According to a feasible implementation method, the interface to be tested can be divided into character regions to obtain multiple character regions containing characters; then, the characters in the multiple character regions are identified, and it is determined whether the character of the control to be tested exists in the multiple character regions. If it exists, the identification is successful, the candidate coordinates of the control to be tested are obtained, and the character recognition result is obtained, which includes at least one candidate coordinate.

[0101] In one example, character localization of the interface under test can be performed in the following way:

[0102] Using the area block containing the characters in the interface under test as the dividing line, multiple character regions containing the characters are obtained; for example, such as Figure 3 As shown, if the region block where the character is located is used as the basis for regionalization, then character region 307, character region 308 and character region 305 can be obtained.

[0103] Correspondingly, characters in multiple character regions can be identified in the following ways:

[0104] For each character region, a window is used to traverse the region and retrieve the characters within it. It is then determined whether the character of the control under test exists within the window; if it does, the recognition is successful. The coordinates of the prompt character within the character of the control under test are obtained as candidate coordinates. During the traversal, the window size can be continuously increased to ensure the accuracy of the recognition results.

[0105] In one example, character localization of the interface under test can be performed in the following way:

[0106] Using consecutive characters as the dividing line, the area containing consecutive characters in the interface under test is divided into multiple character regions. Here, the interface under test is regionalized according to consecutive characters, that is, consecutively appearing characters in the interface are treated as a whole to obtain the first character region; for example, such as... Figure 2 As shown, if consecutive characters are used as the basis for regionalization, then the first character region 201 and the first character region 202 can be obtained.

[0107] Correspondingly, characters in multiple character regions can be identified in the following ways:

[0108] The process involves character recognition within a character region to determine if the character of the control under test exists within that region. If it does, the character region containing the character of the control under test (referred to as the target character region for ease of description and distinction) is extracted from multiple character regions. The coordinates of the prompt character within the target character region are then determined and used as candidate coordinates. Next, multiple character regions outside the target character region are combined according to their position on the interface under test and their row number. For example, character regions on the same row are combined into one region, resulting in multiple character regions. Character recognition is then performed on each character region to determine if the character of the control under test exists within that region. This process can be repeated by increasing the row number and continuously combining character regions outside the target character region to obtain character regions containing more characters. This process continuously searches for the character of the control under test, yielding at least one candidate coordinate.

[0109] According to another feasible implementation method, the interface to be tested can be traversed in a window manner to determine whether the characters in the window are characters of the control to be tested. If so, the coordinates of the prompt character in the characters of the control to be tested in the window can be determined and used as candidate coordinates. After traversing the interface to be tested, the size of the window can be increased and the interface to be tested can be continued to slide to determine whether the characters in the window are characters of the control to be tested. Here, if multiple candidate coordinates are similar, they need to be merged to finally obtain the character recognition result.

[0110] Step 106: When there is one candidate coordinate, use the candidate coordinate as the corrected coordinate of the control to be tested.

[0111] Here, when there is only one candidate coordinate, it means that the character of the control to be tested in the test interface is unique. In this case, the candidate coordinate can be used as the correction coordinate.

[0112] For example, suppose the text of the control under test is "Hot Recommendations Learn More", and the interface under test is as follows: Figure 3As shown, character recognition is performed on the interface to be tested, and region 301 can be obtained. The candidate coordinates of region 302 in region 301 are then used as correction coordinates.

[0113] For example, suppose the text in the control to be tested is "Product A View Details", and the interface to be tested is as follows: Figure 3 As shown, character recognition is performed on the interface to be tested, resulting in region 303. The candidate coordinates of region 304, where the details are viewed, are used as correction coordinates.

[0114] For example, suppose the text in the control to be tested is "View details of successful cases", and the interface to be tested is as follows: Figure 3 As shown, character recognition is performed on the interface to be tested, and region 305 is obtained. The candidate coordinates of region 306, where the details are viewed, are used as the correction coordinates.

[0115] Step 107: When there are multiple candidate coordinates, determine the corrected coordinates from the multiple candidate coordinates.

[0116] Here, when there are multiple candidate coordinates, it means that the character of the control to be tested in the interface to be tested is not unique. In this case, it is necessary to locate the corrected coordinates representing the control to be tested from multiple candidate coordinates.

[0117] For example, suppose the characters in the control to be tested are an account number, and the interface to be tested is... Figure 2 The login interface shown is for... Figure 2 The login interface shown performs character recognition and can identify the areas where the two accounts are located, obtaining the coordinates A1 of area 201 and the coordinates A2 of area 202. At this point, it is necessary to choose one of the coordinates A1 and A2 as the correction coordinate.

[0118] Step 108: Replace the coordinates associated with the target control in the associated library with the corrected coordinates.

[0119] The embodiments of the present invention can automatically recognize the characters of the control under test on the interface under test, which can not only improve the recognition speed, but also update the coordinates associated with the control under test and maintain the association library of the interface under test.

[0120] Figure 5 As shown Figure 1 The flowchart of step 107 in the illustrated embodiment is shown. Figure 1 .like Figure 5 As shown, step 107 includes the following steps:

[0121] Step 501: Determine the extended characters; wherein, the extended characters include some or all of the characters in the control to be tested and other characters outside of them.

[0122] In practical applications, extended characters can include key characters extracted from the characters of the control under test.

[0123] For example, if the character of the control to be tested is an account, then the extended character can be "Please enter your account".

[0124] For example, if the character of the control under test is "account", then the extended character can be "account". That is, there are spaces before and after the account.

[0125] According to a feasible implementation method, a character replacement library can be constructed. The database can pre-store the characters of operable controls in the associated library, as well as the replacement characters containing those characters. Then, the replacement characters of the characters of the control to be tested can be determined from the pre-set character replacement library, and the replacement characters are used as extended characters.

[0126] According to one feasible implementation method, the characters of the control under test can be displayed, and the user can modify the characters to obtain extended characters.

[0127] Step 502: Perform character recognition on the interface under test based on extended characters, and determine the corrected coordinates from multiple candidate coordinates.

[0128] For details, please refer to the description of the interface regionization in step 105 above. Specifically, character recognition is performed on multiple regions of the interface after it has been divided based on extended characters to determine the regions containing the extended characters, and the candidate coordinates in these regions are used as correction coordinates.

[0129] For example, suppose the characters in the control to be tested are an account number, and the interface to be tested is... Figure 2 The login interface shown is for... Figure 2 The login interface shown undergoes character recognition, yielding coordinates A1 for region 201 and A2 for region 202. Since there are two coordinates, the "account" character can be updated to "Please enter your account" and re-recognized. Figure 2 The login interface shown yields region 203, and the coordinates A2 located in region 203 are taken as the final coordinates.

[0130] According to a feasible implementation, assuming character recognition is performed on the interface under test based on extended characters, at least one region of the extended character can be determined. If there is only one region where the extended character is located, the candidate coordinates located at the extended character can be used as correction coordinates. If there are multiple regions where the extended character is located, then... Figure 6 The method shown determines the corrected coordinates.

[0131] Furthermore, in some possible implementations, users can decide whether to replace the characters of the target controls in the association library with extended characters, thereby maintaining the association library.

[0132] Figure 6 As shown Figure 1 The flowchart of step 107 in the illustrated embodiment is shown. Figure 1 .like Figure 6 As shown, step 107 specifically includes the following steps:

[0133] Step 601: Determine the user action; wherein the user action indicates the selection of multiple candidate coordinates.

[0134] In practical applications, multiple candidate coordinate areas can be marked on the interface to be tested. For example, a rectangle or origin can be marked on the interface, with each rectangle or origin representing a candidate coordinate. The user knows what needs to be tested, so they can accurately select the rectangle or origin; for example, the user can click on the rectangle or origin.

[0135] Step 602: In response to the user's operation, use the candidate coordinates selected by the user as the correction coordinates.

[0136] For example, suppose the characters in the control to be tested are an account number, and the interface to be tested is... Figure 2 The login interface shown is for... Figure 2 The login interface shown performs character recognition to obtain the coordinates A1 of region 201 and A2 of region 202; then the user can select coordinate A2 as the final corrected coordinate.

[0137] Furthermore, in some possible implementations, users can update the characters of the control to be tested, for example, by adding additional associated characters. Afterward, users can decide whether to replace the characters of the target control in the association library with the updated characters, thereby maintaining the association library.

[0138] Based on Figure 5 and Figure 6 The following is a detailed description of how to determine the associated database.

[0139] In the initial stage of building the associated library, users can pre-set test text. This test text includes test content corresponding to several functions, describing the operation flow of the target application. This operation flow enables the target application's functionality. Specifically, the operation flow indicates which operable control in which application interface is operated on, and the result of the operation. Correspondingly, the test content needs to include the interface identifier of the target application's interface, as well as the characters of the operable controls within the application interface. It is worth noting that when the function involves multiple application interfaces, the test text needs to arrange the operation flow between different application interfaces. In one example, assuming the target application is a website with tab bars and title names, the tab names in the tab bar and some title names can serve as interface identifiers.

[0140] Subsequently, the test text can be recognized to extract the characters of the interface logo and operable controls in the application interface; then, the characters and the coordinates of the area where the prompt characters are located can be identified from the application interface indicated by the interface logo; finally, the interface logo, characters, and the coordinates of the area where the prompt characters are located are associated and added to the association library.

[0141] If the same character appears in different areas of the application interface, there will be multiple coordinates. These coordinates can be handled in the following two ways.

[0142] Implementation Method 1: Different coordinates can be displayed to the user, who can then select the final coordinates. See details below. Figure 6 The description.

[0143] Implementation Method 2: Additional associated characters can be added to the existing characters to update the existing characters. See details below. Figure 5 The description will not be repeated here.

[0144] For example, suppose the function to be tested is login. The test content for login could be entering the username in the username input text box and the password in the password input box on the login page, and then clicking the login button. Here, the operable controls are the username input box, the password input box, and the login button on the login page. The username, password, and login are the characters of the operable controls. The login icon on the login page can be used as the interface identifier. Then, the interface identifier "login" and the characters of the operable controls in the interface, namely "login," "username," and "password," can be extracted from the test content. Subsequently, the characters of the operable controls and the coordinates of the areas where the characters are located can be identified from the application interface indicated by the interface identifier. Finally, the interface identifier, the characters, and the coordinates of the areas where the characters are located are associated and added to the association library.

[0145] In one possible scenario, if no matching character is found in the database for the target control, it is necessary to determine the coordinates of the target control within the interface to facilitate subsequent testing.

[0146] Based on this, embodiments of the present invention provide another method for testing interface controls. Figure 7 This illustration shows a flowchart of another interface control testing method provided in an embodiment of this application. Figure 7 As shown, in Figure 1 Based on steps 101 to 102 shown, in this embodiment of the application, at least the following steps are included before performing step 104:

[0147] Step 109: If no character matching the character of the target control to be tested is found in the association library, then character recognition is performed on the interface to be tested based on the character of the control to be tested to obtain the coordinates of the control to be tested.

[0148] Here, by considering the actual operable controls in the interface under test, the coordinates of the controls in the interface under test can be determined more accurately.

[0149] Here, the coordinates of the control under test on the interface under test can be determined using the method described above for determining corrected coordinates. See steps 105 to 107 for details. Figure 5 , Figure 6 The description will not be repeated here.

[0150] Furthermore, to maintain and update the associated database, the following steps may also be included:

[0151] Step 110: Add part or all of the characters in the control to be tested and the coordinates of the control to be tested to the associated library.

[0152] According to one feasible implementation method, if the characters of the control under test describe in detail the location information of the operable control, that is, how to find the operable control, such as which area and which position it is in, the key characters can be extracted from the characters of the control under test, and the key characters and the coordinates of the control under test can be associated and added to the association library.

[0153] It is worth noting that, assuming the adoption Figure 2 When determining the coordinates of the control under test using the method shown in step 106, in some possible implementations, the user can decide whether to associate the characters of the control under test with its coordinates and then add it to the association library. Alternatively, the user can modify the characters of the control under test, associate the modified characters with its coordinates, and then add it to the association library.

[0154] It is worth noting that, assuming the adoption Figure 5When determining the coordinates of the control under test using the method shown, in some possible implementations, the user can decide whether to associate the extended characters with the coordinates of the control under test and add them to the association library.

[0155] For example, suppose the characters in the associated library do not include "Learn More", the characters of the control under test are "Learn More", and the test case includes the associated character "Success Cases" for "Learn More". Then, the extended characters "Success Cases Learn More" can be constructed and associated with the coordinates of the control under test and added to the associated library.

[0156] It is worth noting that, assuming the adoption Figure 6 When determining the coordinates of the control under test using the method shown, in some possible implementations, the user can decide whether to associate the characters of the control under test with its coordinates and then add them to the association library.

[0157] For example, suppose the characters in the associated library do not include "Learn More", the characters of the control under test are "Learn More", and the test case includes the associated character "Success Cases" for "Learn More". Then, the user can modify the characters "Learn More" of the control under test to "Success Cases Learn More". At this time, the modified characters "Success Cases Learn More" of the control under test and the coordinates of the control under test can be associated and added to the associated library.

[0158] The following provides two specific examples in conjunction with the above embodiments. In the following examples, the target application is FusionServer Tools, the characters of the control to be tested are called target characters, the operation type is called target operation type, the interface to be tested is called target interface, and it is any application interface in the target application.

[0159] like Figure 8a As shown, the first specific example of the present invention is provided, the details of which are as follows:

[0160] For each application interface in the latest version of FusionServer Tools, obtain the tooltips and coordinates of all operable controls for different functions within FusionServerTools, and construct an association library for that application interface in key-value pairs. Here, the tooltips for operable controls of different functions are unique, and when the tooltips are used as characters for operable controls, the characters (tooltips) in the association library are unique.

[0161] Based on the functions required by FusionServer Tools, multiple test cases are constructed. For any test case (referred to as the target test case for ease of description and distinction), the target character, target operation type, and expected operation result (referred to as the target expected operation result) of the control to be tested in the target interface of the target test case are input.

[0162] Then, the target characters of the control under test in the target interface of the target test case are matched with the characters of the target control in the associated library of the target interface.

[0163] According to a feasible implementation, if a matching character exists, the coordinates A (corresponding to the coordinates of the operable control mentioned above) associated with the matching character are further determined from the association library of the target interface. Then, based on the coordinates A and the target operation type, the operation is performed on the target interface to obtain the first operation result.

[0164] When the result of the first operation does not match the expected result of the target operation in the target test case, the target interface can be divided into character regions to obtain multiple character regions containing characters. Within each character region, a certain window is traversed to obtain the characters within the window. It is determined whether the target character exists within the window; if it exists, the recognition is successful; otherwise, the recognition fails. After traversing all character regions, the character recognition result is obtained (see the description of step 105 above for details). Here, the window size can be based on the width and height of the smallest and largest characters in the actual interface. If only one coordinate C (corresponding to the candidate coordinates of the control under test) is identified in the character recognition result, this coordinate is used as coordinate B (corresponding to the corrected coordinates of the control under test). If multiple coordinates C are identified, they can be... Figure 5 and Figure 6 As shown, coordinate B is obtained from multiple coordinates C; then, an operation is performed on the target interface based on coordinate B and the target operation type to obtain a second operation result. If the second operation result matches the expected target operation result, optionally, the coordinate A associated with the matched character is replaced with coordinate B, thereby automatically updating the association database. Furthermore, users can modify the matched character, replacing the original matched character with the modified one, and simultaneously modifying the target character in the test case to the modified matched character, thus facilitating subsequent testing.

[0165] According to a feasible implementation, if no matching character exists, coordinate B can be obtained by following the handling method for when the first operation result and the expected target operation result in the target test case do not match. Then, based on coordinate B and the target operation type, an operation is performed on the target interface to obtain the second operation result. If the second operation result matches the expected target operation result, optionally, the target character and coordinate B are associated and added to an association library to achieve automatic update and maintenance. Additionally, users can modify the target character, replacing the original target character with the modified one, and simultaneously modifying the target character in the target test case to the modified target character, thus facilitating subsequent testing.

[0166] like Figure 8b As shown, the present invention provides a second specific example, the details of which are as follows:

[0167] For each application interface of the target application, construct multiple characters for multiple operable controls of that application interface. Some characters are combinations of prompt characters and associated characters. Obtain the prompt characters and coordinates of all operable controls of the application interface, and then associate the coordinates with the characters of the pre-constructed operable controls to construct the association library of the application in the form of key-value pairs. The characters in the association library are unique.

[0168] Multiple test cases are constructed based on the functions that the target application needs to implement. For any test case (referred to as the target test case for ease of description and distinction), the target character, target operation type, and expected operation result (referred to as the target expected operation result) of the control to be tested in the target interface of the target test case are input.

[0169] Then, the target characters in the target test case are matched with the characters in the associated library of the target interface.

[0170] According to a feasible implementation, if a matching character exists, the coordinates A (corresponding to the coordinates of the operable control mentioned above) associated with the matching character are further determined from the association library of the target interface. Then, based on the coordinates A and the target operation type, the operation is performed on the target interface to obtain the first operation result.

[0171] When the result of the first operation does not match the expected result of the target operation in the target test case, the target interface can be localized to obtain the character recognition result. The character recognition result can be a single coordinate B (the candidate coordinate of the control under test mentioned above) or multiple coordinates C (the corrected coordinates of the control under test mentioned above). When there are multiple coordinates C, the above method can be used. Figure 5 or Figure 6The method shown determines coordinate B from multiple coordinates C; then, based on coordinate B and the target operation type, an operation is performed on the target interface to obtain a second operation result. If the second operation result matches the expected target operation result, optionally, the coordinate A associated with the matched character is replaced with coordinate B, thereby automatically updating the association database. Furthermore, the user can modify the matched character, replacing the original matched character with the modified one, while simultaneously changing the target character to the modified matched character, thus facilitating subsequent testing.

[0172] According to another feasible implementation, if no matching character exists, coordinate B can be obtained by handling the mismatch between the first operation result and the expected operation result in the target test case. Then, based on coordinate B and the operation type, an operation is performed on the target interface to obtain the third operation result. Coordinate B, the target interface identifier of the target interface, and the target character are then associated and added to the association library. Optionally, the target character and coordinate B are associated and added to the association library for automatic updates and maintenance. Furthermore, users can modify the target character, replacing the original target character with the modified one, and simultaneously modifying the target character in the target test case to the modified target character, thus facilitating subsequent testing.

[0173] It is worth noting that the operable control mentioned in the embodiments of the present invention can also be referred to as a control.

[0174] Based on the same concept as the embodiments of the method of the present invention, the embodiments of the present invention also provide an interface control testing device. The interface control testing device includes several modules, each module being used to execute various steps in the personal information recognition method provided in the embodiments of the present invention. The division of modules is not limited here. Those skilled in the art will clearly understand that in practical applications, the various steps in the interface control testing method provided in the embodiments of the present invention can be assigned to different modules as needed, that is, the internal structure of the device can be divided into different modules to complete all or part of the functions described above. The modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more modules can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the modules are only for easy differentiation and are not intended to limit the scope of protection of the present invention. The specific working process of the modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0175] For example, the interface control testing device is used to execute the interface control testing method provided in the embodiments of the present invention. Figure 9 This is a schematic diagram of the interface control testing device provided in an embodiment of this application. Figure 9As shown in the embodiment of this application, the interface control testing device includes:

[0176] The acquisition module 901 is used to acquire the characters and operation types of the controls to be tested on the interface to be tested;

[0177] The lookup module 902 is used to search in the association library of the interface under test based on the characters of the control to be tested. If the characters of the target control that match the characters of the control to be tested are found in the association library, the coordinates of the control to be tested are obtained. The association library includes the association relationship between the characters and coordinates of the controls in the interface under test. The coordinates of the control to be tested are the same as the coordinates of the target control.

[0178] The recognition module 903 is used to perform character recognition on the interface under test based on the character of the control under test if no character matching the character of the target control is found in the association library, and to obtain the coordinates of the control under test.

[0179] The operation test module 904 is used to perform test operations on the interface under test based on the coordinates of the control under test and the operation type of the control under test, and obtain a first operation result.

[0180] Exemplary electronic devices

[0181] Based on the same concept as the embodiments of the method of the present invention, the embodiments of the present invention also provide an electronic device. This electronic device is also known as a personal information identification device. The electronic device can be a server or a terminal device. The terminal device can be a mobile phone, tablet computer, wearable device, smart TV, Huawei smart screen, smart speaker, in-vehicle system, etc.

[0182] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0183] like Figure 10 As shown, the electronic device 1000 includes a processor 1001, a memory 1002, and a network interface 1003.

[0184] The processor 1001 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0185] The memory 1002 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0186] For example, a computer program may be stored on the memory 1002, and when the processor 1001 executes the computer program, it implements the steps in the above-described interface control testing method embodiment, for example... Figure 2Steps 101 to 104 are shown. Alternatively, when the processor 1001 executes the computer program, it implements the functions of each module in the above-described device embodiment. For example, the computer program can be divided into one or more modules / units, which can be a series of computer program instruction segments capable of performing specific functions. One or more modules / units are stored in the memory 1002 and executed by the processor 1001 to complete the present invention. For example, the computer program can be divided into an acquisition module 901, a search module 902, an identification module 903, and an operation test module 904; the specific functions of each module are described above.

[0187] The network interface 1003 is used to send and receive data, for example, to send data processed by the processor 1001 to other electronic devices, or to receive data sent by other electronic devices.

[0188] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device 1000 relevant to this application are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1000 may include any other suitable components depending on the specific application. Furthermore, the electronic device may be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Those skilled in the art will understand that... Figure 10 This is merely an example of electronic device 1000 and does not constitute a limitation on the electronic device. It may include more or fewer components than illustrated, or combine certain components, or use different components. For example, the electronic device may also include input devices, output devices, network access devices, buses, etc. For example, the input device may be a microphone array, and may also include, for example, a keyboard, mouse, etc. For example, the output device may output various information to the outside, and may include, for example, a display, speaker, printer, and communication network and its connected remote output devices, etc.

[0189] In addition to the methods, apparatus, and electronic devices described above, embodiments of this application may also provide a computer program product, comprising computer program instructions. When executed by a processor, these computer program instructions cause the processor to perform the steps of the interface control testing methods of various embodiments of this application described in the "Methods" section of this specification. The computer program product can be written in any combination of one or more programming languages ​​to execute computer program code for performing the operations of embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The computer program code can be in source code form, object code form, executable file, or some intermediate form. The computer program code can be executed entirely on a user's computing device, partially on a user's device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0190] Furthermore, embodiments of this application may also provide a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the interface control testing methods according to various embodiments of this disclosure as described in the "Method" section above. The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. It should be noted that the content contained in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, a computer-readable medium may not include electrical carrier signals and telecommunication signals.

[0191] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0192] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0193] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0194] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0195] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of the various embodiments of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the specific details described above.

[0196] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0197] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0198] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

[0199] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

Claims

1. A method for testing user interface controls, characterized in that, include: Obtain the characters and operation type of the control to be tested in the interface to be tested. The characters are the content displayed in the interface to be tested. The characters of the control to be tested include the prompt characters of the control to be tested displayed in the interface to be tested and the prompt characters indicating the area where the control to be tested is located. The characters are used to uniquely identify the control to be tested. The interface to be tested is any application interface of the target application. Based on the characters of the control to be tested, a search is performed in the association library. If a character of a target control that matches the character of the control to be tested is found in the association library, the coordinates of the control to be tested are obtained. The association library includes the association relationship between the characters and coordinates of the controls in the interface to be tested. The coordinates of the control to be tested are the same as the coordinates of the target control. The association library is constructed based on the character recognition results of any application interface of the target application. The character recognition results include the prompt characters of the controls displayed in any application interface and the prompt characters of the area indicating the location of the controls displayed in any application interface. If no character matching the character of the control to be tested is found in the associated database, then character recognition is performed on the interface to be tested based on the character of the control to be tested to obtain the coordinates of the control to be tested. Based on the coordinates of the control to be tested and the operation type of the control to be tested, a test operation is performed on the interface to be tested to obtain a first operation result; The test cases include expected operational results, and the method further includes: Obtain the expected operation result of the control to be tested on the interface to be tested; When a character of a target control that matches the character of the control to be tested is found in the association library, and the first operation result is inconsistent with the expected operation result, character recognition is performed on the interface to be tested based on the character of the control to be tested, and the corrected coordinates of the control to be tested in the interface to be tested are determined. Replace the coordinates of the target control in the associated library with the corrected coordinates; If no character matching the character of the control to be tested is found in the association library, and the first operation result is consistent with the expected operation result, at least a portion of the character of the control to be tested and the coordinates of the control to be tested obtained by character recognition of the interface to be tested based on the character of the control to be tested are added to the association library.

2. The method according to claim 1, characterized in that, The step of performing character recognition on the interface under test based on the characters of the control under test to obtain the coordinates of the control under test includes: When multiple candidate coordinates are obtained by character recognition of the interface under test based on the characters of the control under test, the coordinates of the control under test are determined from the multiple candidate coordinates.

3. The method according to claim 2, characterized in that, Determining the coordinates of the control to be tested from the plurality of candidate coordinates includes: Determine the extended characters; wherein, the extended characters include the characters of the control to be tested and other characters besides them; Based on the extended characters, character recognition is performed on the interface to be tested, and the coordinates of the control to be tested are determined from the multiple candidate coordinates.

4. The method according to claim 2, characterized in that, Determining the coordinates of the control to be tested from the plurality of candidate coordinates includes: Determine the user action; wherein the user action indicates the selection of the plurality of candidate coordinates; In response to the user's action, the candidate coordinates selected by the user are used as the coordinates of the control to be tested.

5. The method according to claim 1, characterized in that, The coordinates of the control to be tested indicate the position of the prompt character of the control to be tested on the interface to be tested.

6. A control testing device, characterized in that, include: At least one memory for storing programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1-5.

Images