UI automatic test method and device, equipment, medium and product

Abstract

The invention relates to a UI automatic testing method. The method comprises the steps that element images and page structure information of all elements without child nodes in a current interface of a tested UI are obtained; inputting the element image into an image classification model to output a category label and confidence of each element; receiving a category label required by a test, and matching a corresponding element; and returning one or more matched elements based on the page structure information. The invention further relates to a UI automated testing device, electronic equipment, a computer readable medium and a computer program product.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an artificial intelligence-based UI automation testing method, device, electronic device, computer-readable medium, and computer program product. Background Art

[0002] The current mainstream mobile APP automation testing frameworks (such as Appium) are based on the page document object model (DOM) structure to obtain the control element attributes to identify and locate and operate the page controls to achieve UI automation testing. Common positioning methods include resource-id positioning, className positioning, xpath positioning, etc. Once the page element attribute value or DOM structure changes, all positioning methods and test cases need to be readjusted, and the maintenance cost of the automation script is high.

[0003] And with the popularity of the cross-platform mobile application development framework ReactNative, more and more developers choose to use JavaScript and React to quickly build native mobile applications through bridge conversion technology. Since the component states in the applications converted by ReactNative change frequently, the DOM structure is complex and often lacks necessary accessible attributes, which further reduces the stability of traditional solutions for locating page elements. Summary of the invention

[0004] In order to solve the problems of low versatility and high maintenance cost of existing UI automation testing, the present application proposes a UI automation testing method, including: obtaining element images and page structure information of all elements without child nodes in the current interface of the UI under test; inputting the element images into an image classification model to output the category label and confidence of each of the elements; receiving the category label required for the test and matching the corresponding elements; and returning one or more matching elements based on the page structure information.

[0005] Optionally, the obtaining of element images and page structure information of all elements without child nodes in the current interface of the tested UI includes:

[0006] Taking a screenshot of the current interface of the UI under test;

[0007] Reading page structure information of the element;

[0008] Based on the page structure information, the element image is captured in the screenshot.

[0009] Optionally, the page structure information includes position coordinate information and size information of the element image.

[0010] Optionally, the image classification model is trained by the following method:

[0011] Obtain training images of training elements based on design documents of training UI;

[0012] Adding training labels to the training images;

[0013] The training images and the training labels are used as training data sets to train the artificial intelligence model to be trained, so as to obtain the image classification model.

[0014] Optionally, when the confidence of the element does not reach a confidence threshold, the element is marked as an unclassified element.

[0015] Optionally, matching one or more of the elements are returned in matching order.

[0016] Optionally, when the test requires only one matching element, the first matching element is returned.

[0017] Optionally, obtaining element images and page structure information of all elements without child nodes in the current interface of the UI under test includes: obtaining the element images and the page structure information through Appium.

[0018] This application also proposes a UI automation testing device, including:

[0019] The data acquisition module is used to obtain the element images and page structure information of all elements without child nodes in the current interface of the tested UI;

[0020] An artificial intelligence module, used for inputting the element image into an image classification model to output a category label and confidence score of each element;

[0021] A data matching module, used for receiving the category labels required for testing and matching the corresponding elements;

[0022] The data return module is used to return one or more matching elements based on the page structure information.

[0023] The present application also proposes an electronic device, comprising a memory storing computer-executable instructions and a processor; when the instructions are executed by the processor, the device implements the method according to any one of claims 1 to 8.

[0024] The present application also proposes a computer-readable medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the method described in any one of claims 1 to 8.

[0025] The present application also proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

[0026] The UI automation test method proposed in this application is not the traditional feature matching based on image recognition technology, and then the elements are located. Instead, the elements in the user interface (UI) page are classified and processed through an image classification model, and then the elements are located based on the classification results and the page document structure combined analysis, which greatly enhances the controllable performance of identification and positioning, and can better handle the scene with multiple identical elements. In addition, the UI automation test method proposed in this application is highly versatile. Since the method proposed in this application does not rely on the special syntax of different programming languages ​​or the feature information of special elements when locating elements, but dynamically reads the real-time page structure information of the tested UI, and then locates it in combination with the image classification algorithm proposed in this article, and the UI of applications (APP) on different platforms is usually unified and standardized, and there is a universal page structure document tree that can be parsed, so the computer program code corresponding to this method can be realized without modification. Stable operation across platforms and devices, for example, can be stably operated on many software and hardware platforms such as IOS platform, Android platform or applet platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flowchart obtained by UI automation testing according to the implementation method of this application.

[0028] Figure 2 It is a flowchart of obtaining element images and page structure information according to an implementation of the present application.

[0029] Figure 3 It is a flowchart of training an image classification model according to an embodiment of the present application.

[0030] Figure 4 is a schematic diagram of a UI automation testing device according to an embodiment of the present application.

[0031] Figure 5 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0033] First reference Figure 1 The flowchart is used to illustrate the steps of the method of the present application.

[0034] In step 101, the element images and page structure information of all elements without child nodes in the current interface of the tested UI are obtained. In this application, the smallest element of the tested UI is tested, that is, the element corresponding to the bottom node in the page structure document tree of the tested UI is tested. The element of the node at the next level does not exist in the structure document. Such an element is an element without child nodes. In addition, the test framework Appium can be used to obtain the page data of the APP across platforms.

[0035] In step 102, the element image is input into the image classification model to output the category label and confidence of each element. The category label is the category of the element image artificially divided in the training data set used when training the image classification model, and the category label corresponding to each element and the confidence indicating the corresponding accuracy are automatically identified by the trained image classification model. In one embodiment, the confidence threshold can be set artificially. When the confidence corresponding to some elements exceeds the confidence threshold range, it indicates that its correspondence with the category label is unreliable, and the element is marked as an unclassified element to prompt the tester that the current test does not meet expectations or that the image classification model needs to be adjusted.

[0036] In step 103, the category tags required for the test are received and the corresponding elements are matched. The tester can match the elements belonging to the category tag by inputting the value corresponding to the required category tag into the findElement() API for locating elements in the test framework Appium. Specifically, different number of elements may be matched for different category tags. For example, if there is a category tag of "add button", it may correspond to multiple elements representing add buttons; if there is a category tag of "exit button", it may correspond to one element representing the exit button.

[0037] In step 104, based on the page structure information, one or more matching elements are returned. When a single category tag matches a single element, the element is returned. When a single category tag matches multiple elements, each element can be returned in sequence according to the matching order. The matching order can be, for example, the order in which the elements appear from top to bottom in the page structure document tree, or the order in which the elements appear from top to bottom, from left to right in the diagram, etc. In addition, the tester can also declare in advance the number of elements required for the test. For example, when the tester only needs one element, only the first element that appears in the matching order is returned. When two elements are needed, the first two elements that appear are returned, etc.

[0038] This application tests the positioning of elements, which can be mainly used to test the integrity of elements in the current UI page, that is, to determine whether there are missing elements by positioning the elements. Figure 2Flowchart to obtain element images and page structure information.

[0039] In step 201, a screenshot is taken of the current interface of the UI under test.

[0040] In step 202, the page structure information of the element is read. The element read is the childless element mentioned above, and its page structure information is the coordinate position information and size information of the element image corresponding to the element, and the size information may include, for example, the length and width of the element image. It can be understood that the page structure information includes a variety of information, such as node information used to determine that the element is a childless element, and this article only describes the page structure information related to the method of the present application.

[0041] In step 203, the element image is captured in the screenshot based on the page structure information. The specific position of the element image in the screenshot can be obtained through the page structure information, so that it can be automatically captured.

[0042] Steps 201 to 203 implement real-time dynamic reading of the element images and page structure information of the UI under test. Figure 3 The flowchart shown is used to explain in detail the training of the image classification model.

[0043] In step 301, a training image of a training element is obtained based on the design document of the training UI. It is understandable that the training UI, design document, training element, and training image described in this step are different from the UI, page structure document, element, and element image used for testing described above. The training UI is a standard UI for training an image classification model, and all its information is stored in the design document. Therefore, the training elements and their corresponding training images used for training an image classification model can be obtained by accessing the design document.

[0044] In step 302, training labels are added to the training images. As mentioned above, the training labels are added to the training images by the tester based on information such as the category of the image. For example, different training images can have the same training labels, which means that the training elements corresponding to these training images are the same type of elements. Specifically, as mentioned above, the training labels can be labels with actual meanings such as "add button", "exit button", "food" and "activity", or they can be meaningless labels such as "100", "200", "300" and "400" that are only used for indication. Category labels and training labels are only used to classify different elements and element images.

[0045] In step 303, the training images and training labels are used as training data sets to train the artificial intelligence model to obtain an image classification model. Specifically, the MobileNet model architecture can be modified to use it to train the image classification model.

[0046] In the traditional element image acquisition method that uses the extracted image features to perform feature matching in the screenshot, when there are multiple identical element images, the tester cannot control which element the corresponding image recognition algorithm will identify, and thus cannot control the test process, especially the element positioning process. The traditional detection process is an unpredictable black box process, which leads to the instability and inaccuracy of the test results. In addition, the traditional feature matching technology needs to find similar areas in the image, so it is necessary to prepare different UI page screenshots or use different image recognition algorithms for hardware devices with different resolutions and different screen ratios, so the versatility is poor. However, through the above-mentioned element image acquisition method, combined with the image classification model proposed in this application, only the artificial intelligence model is used to classify the image, and the element positioning process is realized by considering the element category and the page document structure as a whole, avoiding the uncertainty of element positioning in the traditional method. At the same time, the method of this application does not need to consider the hardware device parameters, and can be positioned only by the category label and page document information, and different program languages ​​have a common page structure document tree, so when running the method proposed in this application across platforms or devices, there is no need to modify the corresponding program code, etc., which has good versatility. Therefore, the present application can well cope with the situation where there are multiple identical elements in the displayed UI interface.

[0047] The present application also proposes a UI automated testing device 400, which includes:

[0048] The data acquisition module 401 is used to acquire element images and page structure information of all elements without child nodes in the current interface of the UI under test.

[0049] The artificial intelligence module 402 is used to input the element image into the image classification model to output the category label and confidence of each element. When training the image classification model, the artificial intelligence module can use the Python image processing library Pillow to rotate, crop, scale, Gaussian blur, add noise and background to each element image in the training UI to generate images of different categories to improve the stability of the trained image classification model.

[0050] The data matching module 403 is used to receive the category labels required for the test and match the corresponding elements. The module may have an application monitoring function, that is, a monitoring window is provided, and the page image of the current UI of the APP to which the device is applied is displayed in the monitoring window. The tester can obtain the category label corresponding to the element by clicking on the element of the page in the monitoring window. Furthermore, when the tester clicks on the corresponding element, the corresponding element image can be intercepted based on the page structure information, and classified to display the obtained category label in the element information display column.

[0051] The data return module 404 is used to return one or more matching elements based on the page structure information.

[0052] The UI automation testing device 400 can utilize the aforementioned UI automation testing method to perform automated testing on the UI to detect the status of each element in the UI, and the device 400 can be easily embedded in various software and hardware platforms, such as various applications and various smart phones, and has good versatility.

[0053] Reference now Figure 5 , which is a block diagram of an electronic device 500 according to an embodiment of the present application. The electronic device 500 may include one or more processors 502, a system control logic 508 connected to at least one of the processors 502, a system memory 504 connected to the system control logic 508, a non-volatile memory (NVM) 506 connected to the system control logic 508, and a network interface 510 connected to the system control logic 508.

[0054] The processor 502 may include one or more single-core or multi-core processors. The processor 502 may include any combination of a general-purpose processor and a dedicated processor (e.g., a graphics processor, an application processor, a baseband processor, etc.). In the embodiments of this document, the processor 502 may be configured to execute one or more embodiments of the various embodiments proposed in this application.

[0055] In some embodiments, system control logic 508 may include any suitable interface controller to provide any suitable interface to at least one of processors 502 and / or any suitable device or component in communication with system control logic 508 .

[0056] In some embodiments, the system control logic 508 may include one or more memory controllers to provide an interface to the system memory 504. The system memory 504 may be used to load and store data and / or instructions. In some embodiments, the system memory 504 of the electronic device 500 may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM).

[0057] The non-volatile memory 506 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory 506 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of a HDD (Hard Disk Drive), a CD (Compact Disc) drive, and a DVD (Digital Versatile Disc) drive.

[0058] The non-volatile memory 506 may include a portion of storage resources installed on a device of the electronic device 500, or it may be accessible by the device but not necessarily a portion of the device. For example, the non-volatile memory 506 may be accessed over a network via the network interface 510.

[0059] In particular, the system memory 504 and the non-volatile memory 506 may include: a temporary copy and a permanent copy of the instruction 520, respectively. The instruction 520 may include: an instruction that causes the electronic device 500 to implement the method provided by the present application when executed by at least one of the processors 502. In some embodiments, the instruction 520, hardware, firmware and / or software components thereof may be additionally / alternatively placed in the system control logic 508, the network interface 510 and / or the processor 502.

[0060] In some embodiments, the network interface 510 may be integrated with other components of the electronic device 500. For example, the network interface 510 may be integrated with at least one of the processor 502, the system memory 504, the non-volatile memory 506, and a firmware device (not shown) having instructions, and when at least one of the processors 502 executes the instructions, the electronic device 500 implements one or more of the various embodiments described herein. The network interface 510 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface.

[0061] In one embodiment, at least one of the processors 502 may be packaged together with logic for one or more controllers of the system control logic 508 to form a system in package (SiP). In one embodiment, at least one of the processors 502 may be integrated on the same die with logic for one or more controllers of the system control logic 508 to form a system on a chip (SoC).

[0062] The electronic device 500 may further include an input / output (I / O) device 512. The input / output (I / O) device 512 may include a user interface, so that a user can interact with the device 500; the design of the peripheral component interface enables peripheral components to interact with the electronic device 500.

[0063] In some embodiments, the user interface may include, but is not limited to, a display (e.g., an LCD display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., an LED flash), and a keyboard.

[0064] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0065] It will be understood by those skilled in the art that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 500. In other embodiments of the present application, the electronic device 500 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0066] Program code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0067] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described herein is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0068] One or more aspects of at least one embodiment may be implemented by representative instructions stored on a computer-readable storage medium, which represent various logic in a processor, and when the instructions are read by a machine, cause the machine to fabricate logic for performing the techniques described herein. These representations, known as "IP cores," may be stored on a tangible computer-readable storage medium and supplied to multiple customers or manufacturing facilities to load into a manufacturing machine that actually manufactures the logic or processor.

[0069] An embodiment of the present application discloses a computer-readable medium storing one or more programs executable by one or more processors to implement the method of the present application.

[0070] An embodiment of the present application discloses a computer program product, including a computer program, which implements the method of the present application when executed by a processor.

[0071] The above specific embodiments illustrate the implementation of the present application, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. Although the description of the present application will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to the implementation. In addition, in order to avoid confusion or blurring the focus of the present application, some specific details will be omitted in the description. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0072] Furthermore, various operations will be described as multiple discrete operations in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be construed as implying that these operations are necessarily order dependent. In particular, these operations do not need to be performed in the order presented.

[0073] Unless the context dictates otherwise, the terms "comprising," "having," and "including" are synonymous. The phrase "A / B" means "A or B." The phrase "A and / or B" means "(A and B) or (A or B)."

[0074] As used herein, the term "module" or "unit" may refer to, be or include: an application specific integrated circuit (ASIC), an electronic circuit, a (shared, dedicated or group) processor and / or memory that executes one or more software or firmware programs, a combinational logic circuit and / or other suitable components that provide the described functionality.

[0075] In the accompanying drawings, some structural or method features are shown in a specific arrangement and / or order. However, it should be understood that such specific arrangement and / or ordering may not be required. In some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a specific figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0076] It should be understood that although the terms "first", "second", etc. may be used here to describe various units or data, these units or data should not be limited by these terms. These terms are used only to distinguish one feature from another. For example, without departing from the scope of the exemplary embodiment, a first feature may be referred to as a second feature, and similarly, a second feature may be referred to as a first feature.

[0077] It should be noted that in this specification, similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0078] While the present invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.

Claims

1. A UI automation testing method, characterized in that: include: Get the element images and page structure information of all elements without child nodes in the current interface of the tested UI; Inputting the element image into an image classification model to output a category label and confidence score for each element; Receiving the category label required for the test, and matching the corresponding element; Based on the page structure information, one or more matching elements are returned.

2. The method according to claim 1, characterized in that The step of obtaining element images and page structure information of all elements without child nodes in the current interface of the tested UI includes: Taking a screenshot of the current interface of the UI under test; Reading page structure information of the element; Based on the page structure information, the element image is captured in the screenshot.

3. The method according to claim 2, characterized in that The page structure information includes position coordinate information and size information of the element image.

4. The method according to claim 1, characterized in that The image classification model is trained by the following method: Obtain training images of training elements based on design documents of training UI; Adding training labels to the training images; The training images and the training labels are used as training data sets to train the artificial intelligence model to be trained, so as to obtain the image classification model.

5. The method according to claim 1, characterized in that When the confidence of the element does not reach a confidence threshold, the element is marked as an unclassified element.

6. The method according to claim 1, characterized in that Returns the matching one or more of the elements in the order they were matched.

7. The method according to claim 6, characterized in that When the test requires only one matching element, the first matching element is returned.

8. The method according to claim 1, characterized in that The step of obtaining element images and page structure information of all elements without child nodes in the current interface of the tested UI includes: The element image and the page structure information are obtained through Appium.

9. A UI automated testing device, characterized in that: include: The data acquisition module is used to obtain the element images and page structure information of all elements without child nodes in the current interface of the tested UI; An artificial intelligence module, used for inputting the element image into an image classification model to output a category label and confidence score of each element; A data matching module, used for receiving the category labels required for testing and matching the corresponding elements; The data return module is used to return one or more matching elements based on the page structure information.

10. An electronic device, characterized in that: The device comprises a memory storing computer executable instructions and a processor; when the instructions are executed by the processor, the device implements the method according to any one of claims 1 to 8.

11. A computer readable medium, characterized in that The computer-readable medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images