Automatic test method and device for user interface, equipment, medium and product
By determining the test script based on the operation environment and calling the corresponding test tool set for automated testing in applications used in multiple operation environments, the test difficulty and compatibility problems of different operation environments are solved, and rapid development and efficient users are achieved Interface automation testing.
Patent Information
- Application Number
- CN202411875235.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-18
- Publication Date
- 2025-02-11
AI Technical Summary
When performing automated testing with the same application used in multiple operating environments, testers need to learn a variety of automated testing tools, making it difficult to quickly realize the development of automated testing programs in multiple operating environments, and test programs in different operating environments are difficult to be compatible, and the addition of Test difficulty and time.
An automated testing method for user interface is proposed. By determining the test script based on the running environment of the application to be tested, the test script includes environment parameters, and calling the test tool set corresponding to the running environment to automatically test the user interface.
It reduces the difficulty and time for applications used in various operating environments to perform user interface automation testing, improves the portability of test code, reduces the repeated development of test code, and simplifies the test code development process.
Smart Images
Figure CN119415404A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, specifically to technical fields such as application testing and user interface testing, and more particularly to an automated user interface testing method, an automated user interface testing device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art
[0002] At present, it is becoming more and more common for the same application to be used in multiple operating environments. For example, the same game can have a PC version, a mobile version, and a web version. The PC version can include Windows version and macOS version, and the mobile version can include Android version and iOS version. Different operating environments usually have different automated testing tools.
[0003] For automated testing of the same application used in multiple operating environments, testers need to learn the corresponding automated testing tools for each operating environment. It is difficult to quickly develop automated testing programs for multiple operating environments, and automated testing programs for different operating environments are usually difficult to be compatible. This increases the difficulty and time of automated testing for applications used in multiple operating environments. Summary of the invention
[0004] The embodiments of the present disclosure provide a method for automated testing of user interfaces, an apparatus for automated testing of user interfaces, an electronic device, a computer-readable storage medium, and a computer program product, which can reduce the difficulty and time of automated testing of user interfaces for applications used in a variety of operating environments.
[0005] In a first aspect, an embodiment of the present disclosure proposes an automated testing method for a user interface, comprising: determining a test script based on the operating environment of an application to be tested, wherein the test script includes environmental parameters for indicating the operating environment; based on the environmental parameters in the test script, calling a test tool set corresponding to the operating environment to perform automated testing on the user interface of the application to be tested.
[0006] In the second aspect, an embodiment of the present disclosure proposes an automated testing device for a user interface, comprising: a script determination module, configured to determine a test script based on the operating environment of an application to be tested, wherein the test script includes environmental parameters for indicating the operating environment; a script testing module, configured to call a test tool set corresponding to the operating environment to perform automated testing on the user interface of the application to be tested based on the environmental parameters in the test script.
[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the automated testing method of the user interface described in any implementation manner in the first aspect.
[0008] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement an automated testing method for a user interface as described in any implementation method of the first aspect when executed.
[0009] In a fifth aspect, an embodiment of the present disclosure provides a computer program product including a computer program, which, when executed by a processor, can implement the automated testing method for a user interface as described in any implementation manner in the first aspect.
[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:
[0012] Figure 1 is an exemplary system architecture diagram to which the present disclosure may be applied;
[0013] Figure 2 A flowchart of an automated testing method for a user interface provided by an embodiment of the present disclosure;
[0014] Figure 3 A flowchart of performing automated testing on a user interface of an application to be tested according to an embodiment of the present disclosure;
[0015] Figure 4 A flowchart of determining the position information of a functional control by a positioning method configured by a test script according to an embodiment of the present disclosure;
[0016] Figure 5 A flowchart of determining position information of a functional control through text recognition and / or image recognition in an embodiment of the present disclosure;
[0017] Figure 6 A flowchart of performing automated testing on a user interface of an application to be tested according to another embodiment of the present disclosure;
[0018] Figure 7 A schematic diagram for describing the packaging of the test tool set corresponding to each operating environment according to the embodiment of the present disclosure;
[0019] Figure 8 A schematic diagram for describing the encapsulation of screenshot instructions and text recognition or image recognition operations corresponding to each operating environment according to an embodiment of the present disclosure;
[0020] Fig. 9 A structural block diagram of an automated testing device for a user interface provided in an embodiment of the present disclosure;
[0021] Fig.10 A schematic diagram of the structure of an electronic device suitable for executing an automated testing method for a user interface provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.
[0023] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0024] Figure 1 An exemplary system architecture 100 is shown to which an automated testing method for a user interface according to an embodiment of the present disclosure can be applied.
[0025] like Figure 1As shown, the system architecture 100 may include a terminal device 101 and a test device 102. Among them, the test device 102 can be connected to the terminal device 101 by wired or wireless means, such as a USB data cable, wireless fidelity (Wireless Fidelity, referred to as Wi-Fi) or Bluetooth. The terminal device 101 can be various electronic devices with a display screen, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, augmented reality (Augmented Reality, referred to as AR) glasses and virtual reality (Virtual Reality, referred to as VR) glasses. Various applications can be installed on the terminal device 101, such as music applications, game applications, instant messaging applications, etc. The test device 102 can be various electronic devices with a display screen, including but not limited to laptop portable computers, tablet computers and desktop computers, etc. A test tool can be installed on the test device 102, and various test scripts used by the test tool can be stored. The tester can run the test script through the test tool on the test device 102 to perform automated testing on the user interfaces of various applications running on the terminal device 101 , and can display the user interfaces of the applications running on the terminal device 101 on the display screen of the test device 102 .
[0026] It should be understood that Figure 1 The number of terminal devices and test devices in the embodiment is only for illustration. Any number of terminal devices and test devices may be provided according to implementation requirements.
[0027] Please refer to Figure 2 , Figure 2 A flowchart of an automated testing method for a user interface provided by an embodiment of the present disclosure, wherein process 200 may include the following steps:
[0028] Step 201: determining a test script based on the operating environment of the application to be tested, wherein the test script includes environment parameters for indicating the operating environment;
[0029] This step is intended to be performed by the execution subject of the user interface automated testing method (e.g. Figure 1 The test device 102 shown in the figure determines a test script based on the operating environment of the application to be tested, and the test script includes environmental parameters for indicating the operating environment. The application to be tested can be, for example, Figure 1The applications running on the terminal device 101 shown are, for example, music applications, game applications, instant messaging applications, etc. The implementation methods of the present disclosure do not limit the type of the application to be tested. The operating environment of the application to be tested may include, but is not limited to: at least one of the Android system, iOS system, Windows system, macOS system, and web browser. The operating environment of the application to be tested can be determined based on the terminal device 101, and the implementation methods of the present disclosure do not limit this. For example, the terminal device 101 is a smartphone with an Android system, and the operating environment of the application to be tested is the Android system; the terminal device 101 is a smartphone with an iOS system, and the operating environment of the application to be tested is the iOS system.
[0030] The test script may be a script for automated testing of the user interface of the application to be tested, and the code of the test script may be written according to the interface function provided by the test tool on the test device 102. The test script includes environmental parameters for indicating the operating environment, and different test scripts are provided for different operating environments. The test script may be determined according to the operating environment of the application to be tested, so that the environmental parameters in the test script are consistent with the operating environment of the application to be tested. For example, if the operating environment of the application to be tested is a Windows system, a test script including environmental parameters for indicating the Windows system may be determined; if the operating environment of the application to be tested is a macOS system, a test script including environmental parameters for indicating the macOS system may be determined.
[0031] Step 202: Based on the environment parameters in the test script, call the test tool set corresponding to the running environment to perform automated testing on the user interface of the application to be tested.
[0032] On the basis of step 201, this step is intended to be performed by the above-mentioned execution subject, based on the environment parameters in the test script, to call the test tool set corresponding to the operating environment to perform automated testing on the user interface of the application to be tested. Among them, the test tool set corresponding to the operating environment can be an existing user interface automation test tool set applied to each operating environment, and can include but is not limited to: at least one of the test tool set under the Android system, the test tool set under the iOS system, the test tool set under the Windows system, the test tool set under the macOS system, and the test tool set of the web browser. For example, the test tool set under the Android system can be uiautomator2, the test tool set under the iOS system can be pocoui, the test tool set under the Windows system can be pyautowin, the test tool set under the macOS system can be xcode, and the test tool set of the web browser can be selenium, and the embodiments of the present disclosure are not limited to this.
[0033] The interface functions provided by the test tool on the test device 102 can be adapted to the test tool sets corresponding to each operating environment, and the test tool sets corresponding to each operating environment can be encapsulated through the interface functions provided by the test tool on the test device 102, so that the test tool sets corresponding to each operating environment provide a unified external interface through the interface functions provided by the test tool on the test device 102. The interface functions provided by the test tool on the test device 102 can be used to write the code of the test script, and the test tool sets corresponding to the operating environment encapsulated in each interface function can be called through the environment parameters of the test script. It is also possible to adapt to the interface functions provided by the test tool on the test device 102 as needed, and encapsulate through the interface functions provided by the test tool on the test device 102, increase the test tool set corresponding to the operating environment, and expand the application scenario of the test tool, for example, increase the test tool set under the Ubuntu system.
[0034] The automated testing method for user interfaces provided by the embodiments of the present disclosure integrates test tool sets corresponding to multiple operating environments, determines the environmental parameters of the test script through the operating environment of the application to be tested, and calls the test tool set corresponding to the operating environment, thereby utilizing the test tool set to implement automated testing of the user interface of the application to be tested. The automated testing is performed entirely using the commands and / or functions of the existing test tool set corresponding to the operating environment of the application to be tested, thereby ensuring the performance of the test and ignoring the differences in the operating environments of the application to be tested. The formats of test scripts with the same operating logic in different operating environments can be kept basically consistent, making it easy to learn and master, and easy to maintain. The portability of the test code can be improved, the repeated development of the test code can be reduced, the difficulty of the test code development can be reduced, and the efficiency of the test code development can be improved, thereby reducing the difficulty and time of automated user interface testing for applications used in multiple operating environments.
[0035] Please refer to Figure 3 , Figure 3 This is a flow chart of performing automated testing on a user interface of an application to be tested according to an embodiment of the present disclosure, wherein process 300 may include the following steps:
[0036] Step 301: In response to the user interface being a multimedia user interface, determining position information of a function control on the user interface based on a positioning method configured in a test script;
[0037] This step is intended to be performed by the execution subject of the user interface automated testing method (e.g. Figure 1The test device 102 shown in the figure responds to the user interface being a multimedia user interface, and determines the position information of the functional controls on the user interface based on the positioning method configured by the test script. Since the positioning method configured by the conventional user interface automation test tool set cannot meet the requirement of the multimedia user interface to determine the position information of the functional controls on the user interface, by determining the position information of the functional controls on the user interface according to the positioning method configured by the test script when the user interface is a multimedia user interface, the requirement of the multimedia user interface to determine the position information of the functional controls on the user interface can be met.
[0038] During specific execution, the execution subject may first determine whether the user interface of the application to be tested is a multimedia user interface, and in response to the user interface being a multimedia user interface, determine the position information of the function control on the user interface according to the positioning method configured by the test script. Among them, a multimedia user interface may refer to a user interface that combines two or more media such as text, pictures, photos, sounds, animations, and films, such as the user interface of a video application and the user interface of a game application. A function control may refer to an element on the user interface that allows the user to operate to interact with the application, and the function control may include buttons, text boxes, drop-down menus, sliders, etc. The position information of the function control may refer to the coordinates of the function control on the user interface. For example, a coordinate system is established with the upper left corner of the user interface as the coordinate origin, the horizontal rightward direction as the X-axis, and the vertical downward direction as the Y-axis.
[0039] The embodiments of the present disclosure do not limit the positioning method of the test script configuration. In some optional embodiments, such as Figure 4 As shown, step 301 determines the position information of the functional controls on the user interface based on the positioning method configured by the test script, which may include: step 401: based on the environmental parameters, calling the screenshot instruction corresponding to the operating environment to obtain a screenshot of the user interface of the application to be tested; step 402: performing text recognition and / or image recognition on the screenshot of the user interface to obtain the position information of the functional controls on the user interface. Among them, the positioning method configured by the test script is to take a screenshot of the user interface of the application to be tested, and to determine the position information of the functional controls on the user interface by performing text recognition and / or image recognition on the screenshot of the user interface. For example, the screenshot of the user interface can be image recognized by the open source computer vision library OpenCV, and the screenshot of the user interface can be text recognized by the optical character recognition tool PaddleOCR based on deep learning, which is not limited by the implementation party of the present disclosure. By performing text recognition and / or image recognition on the user interface of the application to be tested to determine the position information of the functional controls on the user interface, the position information of the functional controls on the multimedia user interface can be accurately determined when the multimedia user interface is quickly updated.
[0040] Because different operating environments have different instructions for taking screenshots of the user interface. For example, the screenshot instruction under the Android system can be a screenshot command, the screenshot instruction under the iOS system can be a tidevice command, the screenshot instruction under the Windows system can be a pywinauto command, the screenshot instruction under the macOS system can be a shortcut key screenshot, and the screenshot instruction of the web browser can be pywinauto and a shortcut key screenshot. Therefore, the screenshot instructions corresponding to each operating environment can be encapsulated through the interface function provided by the test tool on the test device 102, and the screenshot instructions corresponding to the operating environment encapsulated in the interface function can be called through the environment parameters of the test script to take a screenshot of the user interface of the application to be tested, thereby obtaining a screenshot of the user interface of the application to be tested.
[0041] In some optional implementations, after obtaining the screenshot of the user interface, the execution subject may also obtain the display resolution of the device running the application to be tested, and adjust the resolution of the screenshot of the user interface according to the display resolution. Figure 5 As shown, step 402 performs text recognition and / or image recognition on the screenshot of the user interface to obtain the position information of the functional controls on the user interface, which may include: step 501: based on the environmental parameters, calling the resolution acquisition instruction corresponding to the operating environment to obtain the display resolution of the device running the application to be tested; step 502: based on the display resolution and a preset resolution threshold, adjusting the resolution of the screenshot of the user interface; step 503: performing text recognition and / or image recognition on the adjusted screenshot of the user interface to obtain the position information of the functional controls on the user interface.
[0042] Among them, the preset resolution threshold can be a resolution that ensures successful text recognition and image recognition, and can be set according to a specific application scenario, and the embodiments of the present disclosure do not limit this. For example, the preset resolution threshold can be 1080P, and the display resolution of the device running the application to be tested is 720P, then the resolution of the screenshot of the user interface is 720P, which is lower than the preset resolution threshold. If the resolution of the screenshot of the user interface is not adjusted, and text recognition and / or image recognition are directly performed on the screenshot of the user interface, it may cause recognition failure, and the text and / or image in the screenshot of the user interface cannot be recognized. The resolution of the screenshot of the user interface can be increased by enlarging the image, etc., so that the resolution of the screenshot of the user interface reaches 1080P, thereby meeting the needs of text recognition and / or image recognition.
[0043] Since different operating environments have different instructions for obtaining the display resolution of the device, the resolution obtaining instructions corresponding to each operating environment can be encapsulated through the interface function provided by the test tool on the test device 102, and the resolution obtaining instructions corresponding to the operating environment encapsulated in the interface function can be called through the environment parameters of the test script to obtain the display resolution of the device running the application to be tested. After obtaining a screenshot of the user interface, by obtaining the display resolution of the device running the application to be tested, the resolution of the screenshot of the user interface is automatically adjusted according to the display resolution and the preset resolution threshold, which can effectively avoid the test failure problem caused by the resolution change such as changing the test device, and can improve the versatility and stability of the test code.
[0044] Step 302: Based on the location information and environmental parameters of the functional control, call the operation mode configured by the test tool set to perform an operation test on the functional control;
[0045] On the basis of step 301, this step is intended to have the above-mentioned execution subject call the operation mode configured by the test tool set to perform an operation test on the function control based on the location information and environmental parameters of the function control. Among them, the operation mode of the function control configured by the test tool set can be determined according to the type of the function control, for example, it can include but is not limited to: click, input, slide, etc. The instruction to take a screenshot of the user interface of the application to be tested, the operation of text recognition and / or image recognition on the screenshot of the user interface, and the operation of the function control based on the location information of the function control can be encapsulated in the interface function provided by the test tool on the test device 102.
[0046] For example, the test tool provides an interface function clickWord(word, exactMatch=False, wordOrder=-1, lang='ch', xOffset=0, yOffset=0, waitTime=1, windowName=”) for clicking on a specified word in the user interface. Among them, word, exactMatch, wordOrder, lang, xOffset, yOffset, waitTime, and windowName are all inputs. Word represents the text to be clicked; exactMatch indicates whether it is an exact match. The default is no. As long as the information of word is included, it is considered a match. An exact match needs to be equal to word; wordOrder indicates when there are multiple The matching number when there is a match, that is, an exact match, such as -1 is the last one, -2 is the second to last, 0 is the first, and 1, 2, and 3 correspond to the list number in sequence; lang indicates whether to search in Chinese or other languages, and en is used for English; xOffset indicates the offset between the actual click coordinates and the search text on the x-axis; yOffset indicates the offset between the actual click coordinates and the search text on the y-axis; waitTime indicates the waiting time after clicking; windowName indicates the name of the window to be clicked, and the default is the current window. Among them, word, exactMatch, wordOrder, and lang are inputs related to text recognition, and xOffset, yOffset, waitTime, and windowName are inputs related to click operations.
[0047] For another example, the test tool provides an interface function clickIcon(self, iconPath, threshold=0.8, iconOrder=0, orientation=”, xOffset=0, yOffset=0, waitTime=1, windowName=”) for clicking a specified icon in the user interface. Among them, iconPath, threshold, iconOrder, orientation, xOffset, yOffset, waitTime, and windowName are all inputs. iconPath indicates the path of the icon; threshold indicates the accuracy of the mark image search, and the default value is 0.8; iconOrder indicates the matching sequence number when there are multiple matches, that is, an exact match, such as -1 is the last one, -2 is the second to last, and 0 is the first one. 1, 2, and 3 correspond to the list sequence number; orientation indicates the direction of the screenshot, and landscape / portrait indicates that if it is specified, some special processing is required; xOffset indicates the offset between the actual click coordinates and the search icon on the x-axis; yOffset indicates the offset between the actual click coordinates and the search icon on the y-axis; waitTime indicates the waiting time after clicking; windowName indicates the name of the window to be clicked, and the default value is the current window. Among them, iconPath, threshold, iconOrder, and orientation are inputs related to image recognition, and xOffset, yOffset, waitTime, and windowName are inputs related to click operations.
[0048] In an optional example, the function clickWord() can be used to write a test script for logging in to a website through a Chrome browser under Android, macOS, and iOS systems. Among them, most of the operations of the test scripts for Android, macOS, and iOS are similar. For example, at the beginning of the three test scripts, control=DeviceControl(testParas,'ANDROID'), control=DeviceControl(testParas,'MACOS'), and control=DeviceControl(testParas,'IOS') are used to indicate the operating environment of the application under test. In the code after the three test scripts, the text of searching or entering a URL is clicked through control.clickWord('Search or enter a URL'), the text of logging in is clicked through control.clickWord('Login'), the text of please entering a mobile phone number is clicked through control.clickWord('Please enter a mobile phone number'), and the text of password is clicked through control.clickWord('Password', wordOrder=0) and control.clickWord('Password', wordOrder=1). For those who are familiar with the development of automated test code for Android systems, they can quickly master the development of automated test code for macOS and iOS systems, which can lower the threshold for developing automated test code across operating environments.
[0049] Optionally, the process 300 of performing automated testing on the user interface of the application to be tested may further include: step 303 and step 304 .
[0050] Step 303: Determine the position information of the component elements of the user interface based on the positioning method configured by the test script;
[0051] On the basis of step 302, this step is intended to determine the position information of the components of the user interface by the above-mentioned execution subject based on the positioning method configured by the test script. Among them, the user interface for determining the position information of the components can be the user interface obtained after operating the functional controls in step 302. In addition to the functional controls of the user interface, the components of the user interface can also include other elements on the user interface that are not used for operation. The execution subject can use the same positioning method as step 301 to determine the position information of the components of the user interface. For example, step 301 is to determine the position information of the functional controls on the user interface by performing text recognition on the screenshot of the user interface, and step 303 is to determine the position information of the components of the user interface by performing text recognition on the screenshot of the user interface. The execution subject can also use a positioning method different from step 301 to determine the position information of the components of the user interface. For example, step 301 is to determine the position information of the functional controls on the user interface by performing text recognition on the screenshot of the user interface, and step 303 is to determine the position information of the components of the user interface by performing text recognition and image recognition on the screenshot of the user interface. The embodiments of the present disclosure are not limited to this.
[0052] Step 304: Detect whether the user interface is an expected user interface based on the position information of the component elements of the user interface.
[0053] On the basis of step 303, this step is intended for the execution subject to detect whether the user interface is the expected user interface based on the position information of the components of the user interface. The expected user interface may be the user interface expected to be obtained after operating the function control in step 302. The execution subject can detect whether the user interface obtained after operating the function control is the expected user interface by matching the position information of the components of the user interface obtained after operating the function control with the position information of the components of the expected user interface.
[0054] Please refer to Figure 6 , Figure 6 This is a flowchart of another embodiment of the present disclosure for performing automated testing on the user interface of an application to be tested, wherein the process 600 may include the following steps:
[0055] Step 601: In response to the user interface being a non-multimedia user interface, calling a positioning method configured in a test tool set based on environmental parameters to determine position information of a functional control on the user interface;
[0056] This step is intended to be performed by the execution subject of the user interface automated testing method (e.g. Figure 1The test device 102 shown in the figure responds to the user interface being a non-multimedia user interface, and calls the positioning method configured by the test tool set based on the environmental parameters to determine the position information of the function control on the user interface. Usually for a non-multimedia user interface, the position information of the function control on the user interface can be quickly determined by the positioning method configured by the user interface automation test tool set. Therefore, when the execution subject responds to the user interface being a non-multimedia user interface, the positioning method configured by the test tool set can be called according to the environmental parameters to determine the position information of the function control on the user interface. Among them, the positioning method configured by the test tool set is usually to determine the position information of the function control on the user interface according to the properties of the function control. For example, the properties of the function control may include: identifier ID, name Name, etc. In addition to being able to directly determine the position information of the function control according to the properties of the function control, the positioning method configured by the test tool set can also determine the position information of the function control according to the path expression XPath or the style expression CSS.
[0057] Step 602: Based on the location information and environmental parameters of the functional control, call the operation mode configured by the test tool set to perform an operation test on the functional control;
[0058] Regarding step 602 and Figure 3 The same as step 302 shown in the figure, please refer to the same part Figure 3 The corresponding parts of the illustrated implementation manner are not described in detail here.
[0059] Optionally, the process 600 of performing automated testing on the user interface of the application to be tested may further include: step 603 and step 604 .
[0060] Step 603: Based on the environmental parameters, call the positioning method configured in the test tool set to determine the position information of the component elements of the user interface;
[0061] On the basis of step 602, this step is intended to determine the position information of the constituent elements of the user interface by the above-mentioned execution subject based on the environmental parameters and calling the positioning method configured by the test tool set. Among them, the user interface for determining the position information of the constituent elements can be the user interface obtained after operating the functional controls in step 602. In addition to the functional controls of the user interface, the constituent elements of the user interface can also include other elements on the user interface that are not used for operation. The execution subject can use the same positioning method as step 601 to determine the position information of the constituent elements of the user interface. For example, step 601 is to determine the position information of the functional controls on the user interface through the identifier ID of the functional controls, and step 603 can determine the position information of the constituent elements of the user interface through the identifier ID of the constituent elements of the user interface.
[0062] Step 604: Detect whether the user interface is an expected user interface based on the position information of the component elements of the user interface.
[0063] Regarding step 604 and Figure 3 The same as step 304 shown in the figure, please refer to the same part Figure 3 The corresponding parts of the illustrated implementation manner are not described in detail here.
[0064] To deepen understanding, this disclosure also provides a specific implementation solution in combination with a specific application scenario, see Figure 7 and Figure 8 shown. Figure 7 To describe the schematic diagram of packaging the test tool set corresponding to each operating environment according to the embodiment of the present disclosure, Figure 8 A schematic diagram is provided to describe the encapsulation of screenshot instructions and text recognition or image recognition operations corresponding to each operating environment according to the implementation method of the present disclosure. This implementation scheme provides a test tool that integrates a test tool set of multiple operating environments. Among them, the UI positioning external interface is an external interface function provided by the test tool, and the code of the test script can be written using the external interface function provided by the test tool. The external interface function provided by the test tool is adapted by the test tool set corresponding to each operating environment, and the test tool set corresponding to each operating environment is encapsulated, so that the test tool set corresponding to each operating environment can provide a unified external interface through the external interface function provided by the test tool. Among them, each operating environment is an Android system, an iOS system, a Windows system, an H5web browser, and a macOS system, and the test tool set encapsulated in each operating environment is uiautomator2, pocoui, pyautowin, selenium, and xcode, etc. The picture and text positioning external interface is an external interface function provided by the test tool, and the picture and text positioning external interface is the same external interface as the UI positioning external interface. The external interface functions provided by the test tool can encapsulate the screenshot commands and text recognition or image recognition operations corresponding to each operating environment, so that each operating environment can provide a unified external interface through the external interface functions provided by the test tool. Among them, the operating environments are Android system, iOS system, Windows system, H5web browser and macOS system, and the screenshot commands encapsulated in each operating environment are screenshot command, tidevice command, pywinauto command, pywinauto and shortcut key screenshot and shortcut key screenshot.
[0065] Further references Fig. 9 As an implementation of the methods shown in the above figures, the present disclosure also provides an automated testing device for a user interface. The implementation method of the automated testing device for a user interface is similar to Figure 2 The embodiments of the method shown correspond to those described above.
[0066] like Fig. 9 As shown, the user interface automation testing device 900 of this embodiment may include: a script determination module 901 and a script testing module 902 .
[0067] In the embodiment of the automated testing device 900 of the user interface Figure 2 The case corresponds to the implementation method shown.
[0068] The script determination module 901 is configured to determine a test script based on the operating environment of the application to be tested, wherein the test script includes environment parameters for indicating the operating environment. The script testing module 902 is configured to call a test tool set corresponding to the operating environment to perform automated testing on the user interface of the application to be tested based on the environment parameters in the test script.
[0069] In this embodiment, the specific processing of the script determination module 901 and the script testing module 902 in the user interface automation testing device 900 and the technical effects thereof can be referred to in Figure 2 The relevant descriptions of steps 201-202 in the corresponding implementation manner are omitted here.
[0070] In some optional implementations of this embodiment, the script testing module 902 includes:
[0071] A first positioning submodule is configured to determine, in response to the user interface being a multimedia user interface, position information of a function control on the user interface based on a positioning method configured by the test script;
[0072] The operation submodule is configured to call the operation mode configured in the test tool set to perform an operation test on the function control based on the position information of the function control and the environmental parameters.
[0073] In some optional implementations of this embodiment, the first positioning submodule includes:
[0074] A screenshot unit, configured to call a screenshot instruction corresponding to the operating environment based on the environment parameter to obtain a screenshot of the user interface of the application to be tested;
[0075] The processing unit is configured to perform text recognition and / or image recognition on the screenshot of the user interface to obtain position information of the function controls on the user interface.
[0076] In some optional implementations of this embodiment, the processing unit includes:
[0077] A resolution acquisition subunit is configured to call a resolution acquisition instruction corresponding to the running environment to acquire a display resolution of a device running the application to be tested based on the environment parameters;
[0078] a resolution adjustment subunit, configured to adjust the resolution of the screenshot of the user interface based on the display resolution and a preset resolution threshold;
[0079] The recognition subunit is configured to perform text recognition and / or image recognition on the screenshot of the adjusted user interface to obtain position information of the functional controls on the user interface.
[0080] In some optional implementations of this embodiment, the first positioning submodule is further configured to determine the position information of the component elements of the user interface based on the positioning method configured by the test script;
[0081] The script testing module further includes:
[0082] The detection submodule is configured to detect whether the user interface is an expected user interface based on position information of component elements of the user interface.
[0083] In some optional implementations of this embodiment, the script testing module 902 includes:
[0084] A second positioning submodule is configured to, in response to the user interface being a non-multimedia user interface, call the positioning method configured by the test tool set based on the environmental parameters to determine the position information of the functional control on the user interface;
[0085] The operation submodule is configured to call the operation mode configured in the test tool set to perform an operation test on the function control based on the position information of the function control and the environmental parameters.
[0086] In some optional implementations of this embodiment, the second positioning submodule is further configured to call the positioning method configured in the test tool set to determine the position information of the component elements of the user interface based on the environmental parameters;
[0087] The script testing module further includes:
[0088] The detection submodule is configured to determine whether the user interface is an expected user interface based on position information of component elements of the user interface.
[0089] In some optional implementations of this embodiment, the operating environment includes: at least one of: Android system, iOS system, Windows system, macOS system, and a web browser.
[0090] In some optional implementations of this embodiment, the test toolset includes: at least one of a test toolset under an Android system, a test toolset under an iOS system, a test toolset under a Windows system, a test toolset under a macOS system, and a test toolset for a web browser.
[0091] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the data migration method described in any of the above embodiments can be implemented when the at least one processor executes.
[0092] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the data migration method described in any of the above embodiments when executed.
[0093] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, which, when executed by a processor, can implement the data migration method described in any of the above embodiments.
[0094] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0095] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.
[0096] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0097] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as the automated testing method for the user interface. For example, in some embodiments, the automated testing method for the user interface may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the automated testing method for the user interface described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the automated testing method of the user interface in any other appropriate manner (eg, by means of firmware).
[0098] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0099] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0100] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0101] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0102] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0103] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.
[0104] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0105] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A method for automated testing of a user interface, comprising: Determining a test script based on the operating environment of the application to be tested, wherein the test script includes an environmental parameter for indicating the operating environment; Based on the environment parameters in the test script, a test tool set corresponding to the operating environment is called to perform automated testing on the user interface of the application to be tested.
2. The method according to claim 1, wherein: The step of calling a test tool set corresponding to the operating environment to perform automated testing on the user interface of the application to be tested based on the environment parameters in the test script includes: In response to the user interface being a multimedia user interface, determining position information of a functional control on the user interface based on a positioning method configured in the test script; Based on the position information of the functional control and the environmental parameters, the operation mode configured by the test tool set is called to perform an operation test on the functional control.
3. The method according to claim 2, wherein: The determining the position information of the functional control on the user interface based on the positioning method configured by the test script includes: Based on the environment parameters, calling a screenshot instruction corresponding to the operating environment to obtain a screenshot of the user interface of the application to be tested; Perform text recognition and / or image recognition on the screenshot of the user interface to obtain position information of function controls on the user interface.
4. The method according to claim 3, wherein: The performing text recognition and / or image recognition on the screenshot of the user interface to obtain the position information of the functional control on the user interface includes: Based on the environment parameters, calling a resolution acquisition instruction corresponding to the running environment to acquire a display resolution of a device running the application to be tested; Adjusting the resolution of the screenshot of the user interface based on the display resolution and a preset resolution threshold; Perform text recognition and / or image recognition on the screenshot of the adjusted user interface to obtain position information of the functional controls on the user interface.
5. The method according to any one of claims 2 to 4, wherein: The step of calling a test tool set corresponding to the operating environment to perform automated testing on the user interface of the application to be tested based on the environment parameters in the test script further includes: Determining position information of component elements of the user interface based on the positioning method configured by the test script; Whether the user interface is a desired user interface is detected based on position information of constituent elements of the user interface.
6. The method according to claim 1, wherein: The step of calling a test tool set corresponding to the operating environment to perform automated testing on the user interface of the application to be tested based on the environment parameters in the test script includes: In response to the user interface being a non-multimedia user interface, calling the positioning method configured by the test tool set based on the environmental parameters to determine the position information of the functional controls on the user interface; Based on the position information of the functional control and the environmental parameters, the operation mode configured by the test tool set is called to perform an operation test on the functional control.
7. The method according to claim 6, wherein: The step of calling a test tool set corresponding to the operating environment to perform automated testing on the user interface of the application to be tested based on the environment parameters in the test script further includes: Based on the environmental parameters, calling the positioning method configured in the test tool set to determine the position information of the component elements of the user interface; Whether the user interface is a desired user interface is detected based on position information of constituent elements of the user interface.
8. The method according to claim 1, wherein: The operating environment includes: at least one of an Android system, an iOS system, a Windows system, a macOS system, and a web browser.
9. The method according to claim 1, wherein: The test tool set includes: at least one of a test tool set under an Android system, a test tool set under an iOS system, a test tool set under a Windows system, a test tool set under a macOS system, and a test tool set for a web browser.
10. An automated testing device for a user interface, comprising: A script determination module is configured to determine a test script based on the operating environment of the application to be tested, wherein the test script includes an environment parameter for indicating the operating environment; The script testing module is configured to call a testing tool set corresponding to the operating environment to perform automated testing on the user interface of the application to be tested based on the environmental parameters in the test script.
11. The device according to claim 10, wherein the script test module comprises: A first positioning submodule is configured to determine, in response to the user interface being a multimedia user interface, position information of a function control on the user interface based on a positioning method configured by the test script; The operation submodule is configured to call the operation mode configured in the test tool set to perform an operation test on the function control based on the position information of the function control and the environmental parameters.
12. The device according to claim 11, wherein the first positioning submodule comprises: A screenshot unit, configured to call a screenshot instruction corresponding to the operating environment based on the environment parameter to obtain a screenshot of the user interface of the application to be tested; The processing unit is configured to perform text recognition and / or image recognition on the screenshot of the user interface to obtain position information of the function controls on the user interface.
13. The device according to claim 12, wherein the processing unit comprises: A resolution acquisition subunit is configured to call a resolution acquisition instruction corresponding to the running environment to acquire a display resolution of a device running the application to be tested based on the environment parameters; a resolution adjustment subunit, configured to adjust the resolution of the screenshot of the user interface based on the display resolution and a preset resolution threshold; The recognition subunit is configured to perform text recognition and / or image recognition on the screenshot of the adjusted user interface to obtain position information of the functional controls on the user interface.
14. The device according to any one of claims 11 to 13, wherein the first positioning submodule is further configured to determine the position information of the component elements of the user interface based on the positioning method configured in the test script; The script testing module further includes: The detection submodule is configured to detect whether the user interface is an expected user interface based on position information of component elements of the user interface.
15. The device according to claim 10, wherein the script test module comprises: A second positioning submodule is configured to, in response to the user interface being a non-multimedia user interface, call the positioning method configured by the test tool set based on the environmental parameters to determine the position information of the functional control on the user interface; The operation submodule is configured to call the operation mode configured in the test tool set to perform an operation test on the function control based on the position information of the function control and the environmental parameters.
16. The apparatus according to claim 11, wherein the second positioning submodule is further configured to determine the position information of the component elements of the user interface by calling the positioning method configured in the test tool set based on the environmental parameters; The script testing module further includes: The detection submodule is configured to determine whether the user interface is an expected user interface based on position information of component elements of the user interface.
17. The device according to claim 10, wherein: The operating environment includes: at least one of an Android system, an iOS system, a Windows system, a macOS system, and a web browser.
18. The device according to claim 10, wherein: The test tool set includes: at least one of a test tool set under an Android system, a test tool set under an iOS system, a test tool set under a Windows system, a test tool set under a macOS system, and a test tool set for a web browser.
19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the automatic testing method for the user interface according to any one of claims 1 to 9.
20. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the automated testing method for a user interface according to any one of claims 1 to 9.
21. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the automated testing method for a user interface according to any one of claims 1 to 9.