A test method, device and equipment of an operation interface and a storage medium

Abstract

Embodiments of the present application provide a test method, device and equipment of an operation interface and a storage medium to solve the problem of high maintenance cost and low test efficiency in the prior art when the UI interface is automatically tested. The method comprises: determining a to-be-tested interface of a to-be-tested application in a target device based on a preset interface test requirement; determining a connection path between the to-be-tested interface and a main operation interface based on a hierarchical connection relationship between interface identifiers; after starting the to-be-tested application, starting from the obtained main operation interface, the following operations are executed in a loop until the to-be-tested interface is obtained: performing control recognition on a current operation interface to obtain control categories and display positions of controls in the current operation interface; obtaining an interface identifier of a next operation interface and determining a control category matched with the interface identifier from the obtained control categories; and sending a click command based on the associated display position to obtain the next operation interface.

Description

Technical Field

[0001] This application relates to the field of application testing technology, and in particular to a method, apparatus, device and storage medium for testing user interfaces. Background Technology

[0002] Currently, some applications that focus on the user interface (UI) require repeated verification on different devices and environments during development and maintenance to ensure that different versions of the application can display the UI correctly during use. Due to the limited number of test machines and the complex installation process for different versions, testing is usually the responsibility of testers.

[0003] To reduce the workload of testers, automated testing methods can be used to test the UI interface, given the relevant technology. Specifically, the following two methods can be used, but are limited to:

[0004] Method 1: Automated testing of UI interfaces through behavior trees.

[0005] A behavior tree is a representation of the execution logic of a test through the connections between nodes. Each node represents a single click action on a UI control, and each edge represents the execution order of the click actions between two connected nodes.

[0006] For example, in the behavior tree, the control click behavior represented by node 1 is: click on control 1 at position (400, 400) on the screen. Then, when the position of control 1 is updated to (238, 320), the control click behavior represented by node 1 in the behavior tree needs to be modified accordingly.

[0007] Therefore, whenever the position of a control in the UI changes, the click behavior of the corresponding node in the behavior tree needs to be corrected and modified. As a result, when testing different versions of the application, the behavior tree needs to be updated according to different UI interfaces, making the testing process too cumbersome and affecting testing efficiency.

[0008] Method 2: Implement automated testing of the UI interface through an intrusive automated testing system.

[0009] Intrusive automated testing systems need to utilize interfaces provided by the operating system or framework to obtain the control hierarchy tree and build locators based on control attributes to achieve control identification and location. Furthermore, intrusive automated testing systems can implement control clicks by calling specific functions. Thus, during testing, the intrusive automated testing system can determine whether the control click behavior was completed correctly by monitoring click events (i.e., determining whether a specific function was called).

[0010] However, invasive automated testing systems require maintaining a control hierarchy tree. The display position and size of the same control may differ on different versions of the application's UI. Therefore, invasive automated testing systems also need to update the execution logic according to the specific position of the control on different versions of the UI to ensure accurate monitoring of control click behavior. This significantly increases the maintenance cost of invasive automated testing systems and limits testing efficiency to some extent.

[0011] Therefore, there is an urgent need to provide an automated testing method to overcome the above-mentioned shortcomings. Summary of the Invention

[0012] This application provides a method, apparatus, device, and storage medium for testing user interfaces, in order to solve the problems of high maintenance costs and low testing efficiency when automating UI testing in the prior art.

[0013] In a first aspect, embodiments of this application provide a method for testing a user interface, including:

[0014] Based on the preset interface testing requirements, determine the interface to be tested for the application to be tested on the target device;

[0015] Based on the hierarchical connection relationship between the interface identifiers preset for the application under test, the connection path between the interface under test and the main operation interface is determined; the connection path includes: the interface identifiers of each operation interface presented during the process of switching from the main operation interface to the interface under test.

[0016] After the application under test is launched on the target device, the following operations are performed in a loop, starting from the main operation interface obtained, until the interface under test is obtained:

[0017] The current operation interface is identified to obtain the control category and display position of at least one control in the current operation interface.

[0018] Based on the connection path, obtain the interface identifier of the next operation interface, and determine the control category that matches the interface identifier from at least one obtained control category;

[0019] Based on the display position associated with the matched control category, a click command is sent to the target device to obtain the next operation interface, and the next operation interface is used as the new current operation interface.

[0020] Secondly, embodiments of this application provide a testing device for an operating interface, the device comprising:

[0021] The first processing module is used to determine the interface to be tested of the application to be tested on the target device based on the preset interface testing requirements.

[0022] The second processing module is used to determine the connection path between the interface to be tested and the main operation interface based on the hierarchical connection relationship between the interface identifiers preset for the application to be tested; the connection path includes: the interface identifiers of each operation interface presented during the process of switching from the main operation interface to the interface to be tested.

[0023] The third processing module is used to perform the following operations in a loop, starting from the main operation interface obtained, after the application to be tested is launched on the target device, until the interface to be tested is obtained:

[0024] The current operation interface is identified to obtain the control category and display position of at least one control in the current operation interface.

[0025] Based on the connection path, obtain the interface identifier of the next operation interface, and determine the control category that matches the interface identifier from at least one obtained control category;

[0026] Based on the display position associated with the matched control category, a click command is sent to the target device to obtain the next operation interface, and the next operation interface is used as the new current operation interface.

[0027] In one possible implementation, the third processing module, when performing control identification on the acquired current operation interface and determining the control category and display position of at least one control in the current operation interface, is specifically used for:

[0028] According to multiple preset scales, features are extracted from the current operation interface to obtain corresponding feature maps;

[0029] For each feature map, perform the following operations: select at least one matching detection box according to the resolution of the feature map, and perform control recognition on the feature map;

[0030] Based on the control recognition results of each detection box, the control category and display position of at least one control in the current operation interface are determined.

[0031] In one possible implementation, the third processing module, when selecting at least one matching detection box according to the resolution of the feature map and performing control recognition on the feature map, is specifically used for:

[0032] Based on a preset correlation between resolution and detection box size, the target size of at least one detection box associated with the resolution of the feature map is obtained;

[0033] The feature map is divided into multiple grids, each grid representing a pixel feature at a spatial location in the feature map;

[0034] For each of the grids, using each of the detection boxes, the following operations are performed respectively: taking one grid as the center of a detection box, determining each grid covered by the target size of the detection box, and performing control recognition in the detection box based on the pixel features represented by each grid.

[0035] In one possible implementation, each control identification result includes: the bounding box parameters of the corresponding detection box, the confidence score, and the control category probability; the confidence score is used to represent the predicted probability that a control exists within the detection box;

[0036] The third processing module, when determining the control category and display position of at least one control in the current operation interface based on the control recognition results of each detection box, is specifically used for:

[0037] Based on the control category probability of at least one detection box in each feature map, the control category of at least one control included in the current operation interface is determined.

[0038] Aggregate the confidence scores of at least one checkbox with the same control category to obtain the confidence score set corresponding to each control category.

[0039] For each control category, perform the following operations: Based on the confidence set corresponding to a control category, select a detection box that meets the confidence condition from the detection boxes corresponding to each confidence level in the confidence set, and determine the display position of the control represented by the control category based on the bounding box parameters of the detection box.

[0040] In one possible implementation, the third processing module obtains the current operation interface in the following way:

[0041] Determine the device type of the target device, and determine the corresponding interface acquisition instruction based on the device type;

[0042] The interface acquisition instruction is sent to the target device so that the target device can send the current operation interface after receiving the interface acquisition instruction.

[0043] In one possible implementation, the interface testing requirements include a device identifier, an application identifier, and an interface identifier;

[0044] The first processing module, when determining the interface to be tested of the application to be tested on the target device based on preset interface testing requirements, is specifically used for:

[0045] The target device for which interface testing needs to be performed is determined based on the device identifier;

[0046] Based on the application identifier, the application to be tested is determined in the target device;

[0047] Based on the interface identifier, the interface to be tested in the application to be tested is determined from the interface configuration data of the application to be tested.

[0048] In one possible implementation, the second processing module is further configured to:

[0049] Before the target device launches the application under test, an application launch command is generated based on the application identifier of the application under test.

[0050] Send the application launch command to the target device.

[0051] Thirdly, embodiments of this application provide an electronic device, including:

[0052] Memory, used to store program instructions;

[0053] A processor is configured to invoke program instructions stored in the memory and execute the method described in the first aspect and different implementations of the first aspect according to the obtained program instructions.

[0054] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the methods described in the first aspect and different implementations of the first aspect.

[0055] Fifthly, embodiments of this application provide a computer program product, including a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of the method described in the first aspect and different implementations of the first aspect.

[0056] The beneficial effects of this application are as follows:

[0057] In this application, only the connection paths between each level of the interface need to be maintained. After determining the testing requirements, it is only necessary to identify the controls and their display positions in the interface, and then send click commands to the target device based on the connection paths between the interfaces and their display positions, thereby achieving automatic testing of the interface. This application does not require maintaining a control behavior tree, and the display position of the interface is identified in real time. Therefore, even if the control position changes, there is no need to modify the control click behavior in the behavior tree, which improves testing efficiency.

[0058] Furthermore, this application does not require updating the execution logic after the control position changes, thus ensuring the monitoring of control click behavior. Since the control association between different versions is fixed, the testing method of this application has low maintenance cost.

[0059] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0062] Figure 2 This application provides a schematic diagram illustrating the relationship between a user terminal and a server terminal in an embodiment of the present application.

[0063] Figure 3 This application provides a schematic diagram illustrating the relationship between a server and a user in an embodiment of the present application.

[0064] Figure 4 A schematic diagram of a server and testing equipment provided in an embodiment of this application;

[0065] Figure 5 A structural diagram of a control terminal provided in an embodiment of this application;

[0066] Figure 6 A schematic diagram of a test process for an operation interface provided in an embodiment of this application;

[0067] Figure 7A schematic diagram illustrating the connection relationship of interface identifiers provided in an embodiment of this application;

[0068] Figure 8 A schematic diagram illustrating an interface association relationship provided in an embodiment of this application;

[0069] Figure 9 A schematic diagram illustrating a feature extraction method provided in an embodiment of this application;

[0070] Figure 10 A schematic diagram illustrating the association of a grid with a detection box, provided in an embodiment of this application;

[0071] Figure 11 A schematic diagram illustrating the association of a grid with multiple detection boxes, provided as an embodiment of this application;

[0072] Figure 12 A schematic diagram illustrating the determination of control categories based on a detection box, as provided in an embodiment of this application;

[0073] Figure 13 This is a schematic diagram of model training and image processing provided in an embodiment of this application;

[0074] Figure 14 A flowchart for testing the operation interface after a click operation is provided in an embodiment of this application;

[0075] Figure 15 A flowchart for testing the operation interface after a drag-and-drop operation, provided as an embodiment of this application;

[0076] Figure 16 A schematic diagram of an interface for testing an embodiment of this application;

[0077] Figure 17 A structural diagram of a test device with an operating interface provided in an embodiment of this application;

[0078] Figure 18 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0080] To facilitate a better understanding of the technical solutions of this application by those skilled in the art, some of the terms involved in this application are introduced below.

[0081] Controls are visual elements in software interfaces or applications used for user interaction. They provide an intuitive way to operate the software, input data, or obtain information. Controls are a core component of graphical user interfaces (GUIs) and are commonly found in desktop applications, web pages, and mobile applications.

[0082] An operational interface (UI) is a visual platform through which an object interacts with a device, software, or system. It consists of a series of controls (such as buttons, menus, and input boxes), layouts, icons, text, graphics, and other elements, designed to help the user intuitively operate functions, complete tasks, or obtain information.

[0083] A Region of Interest (ROI) is a specific area within an image, video, dataset, or scene that is of particular interest to the user or computational team. By focusing on the ROI, key information can be extracted efficiently, computational load can be reduced, or resource allocation can be optimized. This approach is widely used in fields such as image processing, medical imaging, computer vision, and video analytics.

[0084] UI Behavior Tree is a tree-structured model for managing and controlling the interaction logic of a user interface. It decomposes complex UI interactions into hierarchical and modular nodes, achieving dynamic responses through conditional statements, state transitions, and action execution. UI Behavior Trees are particularly suitable for scenarios with complex interaction logic, improving code maintainability and logical clarity.

[0085] The word “exemplary” as used below means “serving as an example, embodiment, or illustration.” Any embodiment illustrated as an “exemplary” need not be construed as superior to or better than other embodiments.

[0086] The terms "first" and "second" used in this document are for descriptive purposes only and should not be construed as indicating relative importance or implying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0087] The design concept of the embodiments of this application is briefly introduced below:

[0088] Currently, some applications that focus on the user interface need to be repeatedly verified on different devices and environments during development and maintenance to ensure that different versions of the application can display the UI interface normally during use.

[0089] For example, a large-scale massively multiplayer online game (MMORPG) boasts numerous uniquely styled maps, elaborate and diverse cosmetics, and realistic, anthropomorphic character animations. The game's visuals are a core part of the gaming experience. The scenes, characters, and outfits not only require exquisite design but also repeated testing under different devices and environments to ensure players receive the best visual experience. Regarding scenes, to ensure their performance meets expectations under varying lighting conditions, weather changes, and time of day, frequent viewing and evaluation of various locations within the scene are necessary, focusing on lighting effects, detail rendering, and dynamic effects. As for characters, frequent review and comparison of the details, animations, and outfits of the in-game character models are required to confirm whether modifications meet expectations and whether there are any unexpected impacts. Although these tasks are relatively simple, considering their quantity, frequency, and timing, the impact on the testing team's daily work is still considerable.

[0090] To reduce the workload of testers, automated testing methods can be used to test the UI interface, given the relevant technology. Specifically, the following two methods can be used, but are limited to:

[0091] Method 1: Automated testing of UI interfaces through behavior trees.

[0092] A behavior tree is a representation of the execution logic of a test through the connections between nodes. Each node represents a single click action on a UI control, and each edge represents the execution order of the click actions between two connected nodes.

[0093] For example, in the behavior tree, the control click behavior represented by node 1 is: click on control 1 at position (400, 400) on the screen. Then, when the position of control 1 is updated to (238, 320), the control click behavior represented by node 1 in the behavior tree needs to be modified accordingly.

[0094] Therefore, whenever the position of a control in the UI changes, the click behavior of the corresponding node in the behavior tree needs to be corrected and modified. As a result, when testing different versions of the application, the behavior tree needs to be updated according to different UI interfaces, making the testing process too cumbersome and affecting testing efficiency.

[0095] Method 2: Implement automated testing of the UI interface through an intrusive automated testing system.

[0096] Intrusive automated testing systems need to utilize interfaces provided by the operating system or framework to obtain the control hierarchy tree and build locators based on control attributes to achieve control identification and location. Furthermore, intrusive automated testing systems can implement control clicks by calling specific functions. Thus, during testing, the intrusive automated testing system can determine whether the control click behavior was completed correctly by monitoring click events (i.e., determining whether a specific function was called).

[0097] However, invasive automated testing systems require maintaining a control hierarchy tree. The display position and size of the same control may differ on different versions of the application's UI. Therefore, invasive automated testing systems also need to update the execution logic according to the specific position of the control on different versions of the UI to ensure accurate monitoring of control click behavior. This significantly increases the maintenance cost of invasive automated testing systems and limits testing efficiency to some extent.

[0098] Therefore, there is an urgent need to provide an automated testing method to overcome the above-mentioned shortcomings.

[0099] To address the aforementioned issues, embodiments of this application provide a method, apparatus, device, and medium for testing user interfaces. After determining the interface to be tested for the application, the connection path between the interface to be tested and the main user interface can be determined based on the hierarchical connection relationship between the interface identifiers preset for the application under test. Further, starting from the acquired main user interface, control identification is performed on the acquired current user interface to obtain the control category and display position of at least one control in the current user interface. Then, based on the connection path, the interface identifier of the next user interface is obtained, and from the at least one obtained control category, the control category matching the interface identifier is determined. Based on the display position associated with the matching control category, a click command is sent to the target device to obtain the next user interface, which is then used as the new current user interface, until the interface to be tested is obtained, thus completing the automated testing of the user interface.

[0100] In this application, only the connection paths between interfaces at each level need to be maintained. There is no need to maintain a control behavior tree, and the display position of the interface is identified in real time. Therefore, even if the control position changes, there is no need to modify the control click behavior in the behavior tree, improving testing efficiency. Furthermore, since this application does not require monitoring control click behavior after a control position changes, there is no need to update the execution logic to ensure monitoring of control click behavior. Moreover, the control relationships between different versions are fixed, thus the testing method of this application has low maintenance costs.

[0101] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0102] like Figure 1 The diagram illustrates an application scenario in this embodiment. This scenario includes a user terminal 100, a server terminal 200, a control terminal 300, and an application terminal 400. The user terminal 100, server terminal 200, and control terminal 300 can be deployed on terminal devices or on servers. Terminal devices can be personal computers, tablets, laptops, or other computer devices with certain computing capabilities. Servers can be independent physical servers or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, cloud functions, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The application terminal 400 can be deployed on the terminal device.

[0103] In some scenarios, the user terminal 100 includes an operation input module, through which the user can initiate an interface test request. This request is relayed through the server terminal 200 to the control terminal 300, and then sent to the test device where the application terminal 400 is located, thereby enabling remote control of the test device. In some embodiments, such as... Figure 2 As shown, the user can send a test request to the server 200 through the operation input module of the user terminal 100.

[0104] In some scenarios, server 200 runs a requirement distribution module, used to distribute test requirements initiated by user 100 to qualified test devices. Server 200 enables remote control of the target object, allowing the test device and the target object to be located in different locations, and also allows the test device to be used by multiple test objects. For example... Figure 3 As shown, the user terminals can include device 1, device 2, and device 3. Devices 1 through 3 need to register with server 200. For example, device 1 can be an Android phone, which needs to register with the demand distribution module of server 200. Device 2 can be an Apple phone, which also needs to register with the demand distribution module of server 200. Device 3 can be a PC, which also needs to register with the demand distribution module of server 200.

[0105] In some scenarios, the device under test (i.e., the test device) needs to register with the requirement distribution module. The requirement distribution module can then distribute requirements to different test devices. For example, the requirement distribution module can distribute requirements to test device 1 and test device 3, where test device 1 is an Android phone and test device 3 is a PC. Figure 4 As shown.

[0106] In some embodiments, the control terminal 300 includes three modules: an image acquisition module 301, an AI module 302, and an operation decision module 303, such as... Figure 5 As shown. The image acquisition module 301 obtains image information of the current operation interface of the application terminal 400. In some embodiments, for Android systems, screenshots of the operation interface can be obtained via ADB commands. The AI ​​module 302 analyzes the controls in the current operation interface to find UI buttons or other areas of interest. The operation decision module 303 receives a test request from the server 200 and searches for corresponding content in the image information of the current operation interface. If the content matches successfully, the corresponding operation instruction (such as single click, double click, drag, etc.) is sent to the application terminal 400.

[0107] In some embodiments, the application terminal 400 comprises two parts: an App runtime 401 and a simulated UI input module 402. The App runtime 401 carries the application to be tested, which can be an application on an Android or iOS device, or a program on a PC. The simulated UI input module 402 is used to perform operations on a specified location on the screen, such as clicking the position (400, 400) with the top left corner of the screen as the origin. This module can be implemented in two ways: one way is to integrate it as an independent module within the application. This module is independent of the UI module and does not affect the processing of UI transactions. This module implements the simulated input function by directly calling the input interface of the UI module. The other way is to use external hardware or software devices to implement the simulated input function, such as a PC-based macro program or a stepper motor-based mobile phone click tool.

[0108] It should be noted that, Figure 1-5 The example shown is for illustrative purposes only; the actual number of user terminals 100 and application terminals 400 is not limited. Figure 1-5 The limitations are not specifically limited in the embodiments of this application.

[0109] The following describes a test method for the user interface provided by an exemplary embodiment of this application, in conjunction with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect.

[0110] See Figure 6The diagram shown is a flowchart of an implementation method for testing an interface according to an embodiment of this application. The specific implementation process of this method is as follows:

[0111] Step S601: Based on the preset interface testing requirements, determine the interface to be tested for the application to be tested on the target device.

[0112] In some embodiments, the interface testing requirements include a device identifier, an application identifier, and an interface identifier. Further, the target device for which interface testing needs to be performed can be determined based on the device identifier, and the application to be tested can be determined within the target device based on the application identifier. Further, the interface to be tested within the application to be tested can be determined from the interface configuration data of the application to be tested based on the interface identifier.

[0113] In some scenarios, device identifiers are used to indicate the type of target device being tested. For example, the identifiers for test devices running Android systems are different from those for test devices running Apple systems.

[0114] As an example, if Android phones, Apple phones, and PCs are identified by 1, 2, and 3 respectively, then when the device identifier included in the interface testing requirements is 2, it means that the target device to be tested is an Apple phone.

[0115] In some scenarios, application identifiers are used to represent applications, and different applications have different application identifiers. After determining the target device to be tested, the application to be tested on the target device can be determined based on the application identifier. In some embodiments, different applications can be represented by different numerical identifiers, for example, 11 represents application 1, and 22 represents application 2. Alternatively, different applications can be represented by different English words, for example, APP1 represents application 1, and APP2 represents application 2. Furthermore, different identifiers can also be used to represent different applications; this application does not specifically limit this approach.

[0116] In some embodiments, different user interfaces of a particular application correspond to different interface identifiers. Therefore, the interface to be tested can be determined by using the interface identifiers in the test interface requirements.

[0117] Step S602: Based on the hierarchical connection relationship between the interface identifiers preset by the corresponding application under test, determine the connection path between the interface under test and the main operation interface.

[0118] The connection path includes the interface identifiers of each operation interface presented during the process of switching from the main operation interface to the interface to be tested.

[0119] In some scenarios, for an application under test, there are pre-defined hierarchical connections between various interface icons. For example... Figure 7As shown, A1 to An are different interface identifiers. A1 is the identifier for the main operation interface after the application starts. There are three sub-interface identifiers below the main operation interface identifier, indicating that the main operation interface includes three controls. Clicking different controls will switch to one of the three different operation interfaces with identifiers A2 to A4. For example, when a control in the main operation interface is clicked, and that control corresponds to interface identifier A2, the target device will switch from the main operation interface to the operation interface corresponding to interface identifier A2. Figure 8 The image shows the main user interface displayed on the target device after launching an application; this interface is labeled S1. At this time, the main user interface includes three controls: a login control, a settings control, and an exit control. Figure 8 The settings are shown in a dashed box. Clicking the settings control displays the settings interface, which is labeled S2. Furthermore, the settings interface includes four controls: sound, video, account, and logout. Clicking the video control switches to the video interface, labeled S3.

[0120] Step S603: After the application to be tested is launched on the target device, start from the main operation interface obtained and perform the following operations in a loop until the interface to be tested is obtained:

[0121] Step S6031: Perform control identification on the acquired current operation interface to obtain the control category and display position of at least one control in the current operation interface.

[0122] In some embodiments, features of the current operating interface can be extracted according to multiple preset scales to obtain corresponding feature maps;

[0123] For each feature map, perform the following operations: Select at least one matching detection box according to the feature map's resolution, and perform control recognition on the feature map. Furthermore, based on the control recognition results of each detection box, determine the control category and display position of at least one control in the current user interface.

[0124] In some scenarios, when selecting at least one matching detection box according to the feature map resolution for control recognition, the target size of at least one detection box associated with the feature map resolution can be determined as follows: Based on a preset correlation between resolution and detection box size, the target size of at least one detection box associated with the feature map resolution is obtained. Furthermore, the feature map can be divided into multiple grids, where each grid represents a pixel feature at a spatial location in the feature map. Then, for each grid, using each type of detection box, the following operations can be performed: Using one grid as the center of a detection box, determining the grids covered by the target size of the detection box, and performing control recognition within a detection box based on the pixel features represented by each grid.

[0125] As an example, multiple preset scales are used to detect targets of different sizes in the current user interface. Specifically, feature extraction layers in the network model can be used to extract features from the current user interface at different levels, thereby obtaining feature maps of three different scales: 80×80, 40×40, and 20×20. Figure 9 As shown in the diagram, the 80×80 feature map has a higher resolution and is more suitable for detecting small targets, while the 20×20 feature map has the lowest resolution and can be used to detect large targets. The 40×40 feature map can be used to detect medium-sized targets. It should be noted that the terms "large target" and "small target" refer to the ratio of the pixels occupied by the target to the pixels in the user interface. The larger the ratio, the larger the target. The target can refer to controls, people, or other elements in the current user interface; this application does not specifically limit this.

[0126] In some scenarios, feature maps of different resolutions can be used for target recognition with bounding boxes of different sizes. For example, when a resolution is associated with a bounding box size, a high resolution associated with a small bounding box is more effective for detecting small targets, while a low resolution associated with a large bounding box is more effective for detecting large targets.

[0127] For example, an 80×80 resolution feature map can be associated with a detection box of size 9×12, a 40×40 resolution feature map can be associated with a detection box of size 20×20, and a 20×20 feature map can be associated with a detection box of size 30×30.

[0128] Assuming the current interface image size is 640×640, the downsampling step size of the feature map with a resolution of 80×80 is 8, and the detection box size associated with the feature map with a resolution of 80×80 is 8×8 (width×height), it actually covers an area of ​​9×8=72 pixels wide and 12×8=96 pixels high in the current interface.

[0129] Furthermore, the feature map can be divided into multiple grids, where each grid represents a pixel feature at a spatial location in the feature map. For example, an 80×80 feature map can be divided into 80×80 grids.

[0130] Next, a grid can be used as the center of a detection box to determine the grids covered by the target size of the detection box, and control recognition can be performed in a detection box based on the pixel features represented by each grid.

[0131] For example, since the downsampling stride of an 80×80 feature map is 8, each grid cell covers an area of ​​9×8 = 72 pixels wide and 12×8 = 96 pixels high in the current user interface. Furthermore, the network can be used as the center of a 9×12 detection box, and the grid cells covered by that detection box can be determined. Figure 10 As shown, the detection box centered on the G1 grid (shown by the dashed line) covers the four grids above, below, left, and right. Control recognition is then performed based on the pixel features of the four grids and the G1 grid, thus determining whether a control exists in the detection box.

[0132] In some scenarios, a single resolution can be associated with multiple bounding box sizes, where lower resolutions are associated with larger bounding boxes and higher resolutions are associated with smaller bounding boxes.

[0133] For example, an 80×80 resolution feature map can be associated with detection boxes of sizes 6×6, 9×12, and 12×16; a 40×40 resolution feature map can be associated with detection boxes of sizes 16×16, 20×20, and 25×32; and a 20×20 feature map can be associated with detection boxes of sizes 28×28, 30×30, and 35×35.

[0134] Furthermore, a grid can be used as the center of multiple associated detection boxes to determine the grids covered by the target size of each detection box, and control recognition can be performed based on the pixel features represented by each corresponding grid. For example... Figure 11 As shown, taking the G2 grid in an 80×80 resolution feature map as an example, a 6×6 detection box covers the G2 grid, while the 9×12 and 12×16 detection boxes each cover four grids (top, bottom, left, and right). Further, based on the pixel features identified by each grid covered by each detection box, control recognition is performed to determine the control recognition result for that detection box. In other words, a 6×6 detection box associated with the G2 grid corresponds to one control recognition result, a 9×12 detection box corresponds to one control recognition result, and a 12×16 detection box corresponds to one control recognition result.

[0135] In other scenarios, a single resolution can be associated with multiple detection box sizes. To improve the accuracy of control recognition results, the detection box size associated with a low resolution can be larger and / or smaller, while the detection box size associated with a high resolution can be larger and / or smaller.

[0136] For example, an 80×80 resolution feature map can be associated with detection boxes of sizes 6×6, 9×12, and 16×16; a 40×40 resolution feature map can be associated with detection boxes of sizes 10×10, 20×20, and 28×28; and a 20×20 feature map can be associated with detection boxes of sizes 25×25, 30×30, and 35×35.

[0137] Based on the above-mentioned relationship between resolution and detection box size, small targets (i.e., small controls in this application) and medium targets (i.e. medium-sized controls in this application) can be identified in high-resolution feature maps, small targets, medium targets, and large targets (i.e. large controls in this application) can be identified in medium-resolution feature maps, and medium targets and large targets can be identified in low-resolution feature maps.

[0138] Of course, in addition to this, small, medium and large targets can also be identified in high-resolution feature maps, or small, medium and large targets can be identified in low-resolution feature maps. This application does not make any specific limitations on this.

[0139] In some embodiments, each control identification result includes: the bounding box parameters of the corresponding detection box, the confidence score, and the control category probability. The confidence score represents the predicted probability that a control exists within the detection box. A higher confidence score indicates a greater likelihood that a control exists within the detection box.

[0140] Furthermore, based on the control recognition results of each detection box, the control category and display position of at least one control in the current operation interface are determined. This can be achieved in the following way:

[0141] First, based on the control category probability of at least one detection box in each feature map, the control category of at least one control included in the current operation interface is determined.

[0142] For example, suppose three feature maps at different scales correspond to 20, 40, and 80 detection boxes, respectively. Figure 12 As shown in the figure, only a portion of the detection boxes in each feature map are shown. Therefore, based on the control category probabilities in the control recognition results of 20+40+80=140 detection boxes, the control category of at least one control included in the current operation interface can be determined. For example... Figure 12 As shown, the control categories identified in the 20 detection boxes are control category 1, control category 4, and control category 6, the control categories identified in the 40 detection boxes are control category 1, control category 2, control category 5, and control category 7, and the control categories identified in the 80 detection boxes are control category 2, control category 3, and control category 8. Therefore, it can be determined that the current operation interface includes controls of 8 control categories.

[0143] Of course, the number of detection boxes used on the feature maps of different scales mentioned above is just an example. More or fewer detection boxes can also be used, and this application does not make any specific limitation on this.

[0144] Furthermore, the confidence scores of at least one checkbox with the same control category can be aggregated to obtain a confidence score set corresponding to each control category. For example, taking control category 1 as an example, it can be determined that among all checkboxes, at least one checkbox has the highest probability of control category 1. Then, the confidence scores of at least one checkbox can be aggregated to obtain the confidence score set corresponding to control category 1. Similarly, the confidence score sets corresponding to other control categories can be determined in the same way, which will not be elaborated further in this application.

[0145] Then, for each control category, the following operations can be performed: based on the confidence set corresponding to a control category, filter out a detection box that meets the confidence condition from the detection boxes corresponding to each confidence level included in the confidence set, and determine the display position of the control represented by a control category based on the bounding box parameter of a detection box.

[0146] As an example, taking control category 1 as an example, suppose the confidence set corresponding to control category 1 is the confidence scores of 10 detection boxes (1-10). Based on the confidence scores of detection boxes 1-10, a detection box that meets the confidence criteria can be selected.

[0147] Specifically, based on a confidence threshold, at least one detection box with a confidence level greater than the threshold can be selected first. For example, the confidence levels corresponding to detection boxes 1-10 are 0.25, 0.75, 0.45, 0.8, 0.1, 0.65, 0.35, 0.8, 0.6, and 0.3, respectively. Assuming a confidence threshold of 0.4, the selected detection boxes would be detection boxes 2, 3, 4, 6, 8, and 9. Further, the selected detection boxes can be sorted from highest to lowest confidence level, and the detection box with the highest confidence level can be selected as the target box. The overlap between the target box and other detection boxes, i.e., the intersection-overall reunion (IoU), can be calculated. If the IoU between a detection box and the target box is greater than the set overlap threshold, it is discarded. This process is repeated until the detection box with the lowest confidence level is found. Finally, the remaining detection box with the highest confidence level is selected as the detection box that meets the confidence condition.

[0148] Assume the confidence levels of the three bounding boxes are 0.95, 0.85, and 0.75, respectively, and the IoU between any two bounding boxes is 0.5. Set the overlap threshold to 0.4. First, ranking the confidence levels as 0.95, 0.85, and 0.75, select the box with a confidence level of 0.95 as the baseline. If the IoU between the bounding box with a confidence level of 0.95 and the bounding box with a confidence level of 0.85 exceeds the set overlap threshold, then the bounding box with a confidence level of 0.85 is discarded. Similarly, the bounding box with a confidence level of 0.75 is also discarded. Therefore, the bounding box with a confidence level of 0.95 is considered to meet the confidence level requirement.

[0149] Furthermore, the display position of controls of this control category can be determined based on the bounding box parameters of the detection box with a confidence level of 0.95.

[0150] Specifically, the bounding box parameters of the detection box can be represented as (t_x, t_y, t_w, t_h), where t_x and t_y represent the offset of the center point of the detection box relative to the top-left corner of the grid, and t_w and t_h represent scaling factors relative to the width and height of the detection box. Furthermore, the bounding box parameters of the detection box can be translated into the current user interface.

[0151] Center coordinate calculation: bx = σ(t_x) + Cx; by = σ(t_y) + Cy;

[0152] Where σ(t_x) and σ(t_y) are Sigmoid functions that constrain the offset to [0,1] to ensure that the center point is within the current grid. Cx and Cy represent the coordinates of the top-left corner of the current grid (e.g., grid (3,5) corresponds to Cx=3,Cy=5).

[0153] Width and height calculation:

[0154] bw = pw·e t_w bh = ph·e t_h ;

[0155] Where pw and ph represent the original width and height of the detection box assigned to the current grid; e t_w e t_h The coefficient is used to ensure that the width and height after scaling are positive numbers.

[0156] Finally, the bounding box parameter coordinates can be mapped to the current operation interface:

[0157] xmin = (bx - bw / 2) × stride;

[0158] ymin = (by - bh / 2) × stride;

[0159] xmax = (bx + bw / 2) × stride;

[0160] ymax = (by + bh / 2) × stride;

[0161] Where stride is the downsampling factor of the feature map relative to the current operating interface (e.g., when downsampling by 32 times, stride = 32).

[0162] Based on the above process, the display position corresponding to at least one of the identified control categories can be determined.

[0163] Step S6032: Obtain the interface identifier of the next operation interface based on the connection path, and determine the control category that matches the interface identifier from at least one obtained control category.

[0164] In some implementations, control categories are associated with interface identifiers. For example, when a control of the category "Settings" is clicked, an interface with the identifier "Settings" is displayed.

[0165] Taking the connection path as Settings—General—Notifications as an example, if the current screen is the Settings screen, then the interface identifier for the next screen will be determined to be General. When identifying controls in the Settings screen, it may be identified that the Settings screen includes three control categories: General, Device, and Account. Furthermore, since the interface identifier for the next screen is General, the control category matching the "General" interface identifier can be determined to be the General control category.

[0166] Step S6033: Based on the display position associated with the matched control category, send a click command to the target device to obtain the next operation interface and use the next operation interface as the new current operation interface.

[0167] In this application, only the connection paths between the interfaces at each level need to be maintained. After determining the testing requirements, it is only necessary to identify the controls and their display positions in the user interface. Then, based on the connection paths between the interfaces at each level and their display positions, click commands are sent to the target device, thereby achieving automatic testing of the user interface. This application does not require maintaining a control behavior tree, and the display position of the user interface is identified in real time. Therefore, even if the control position changes, there is no need to modify the control click behavior in the behavior tree, which improves testing efficiency.

[0168] Furthermore, this application does not require updating the execution logic after the control position changes, thus ensuring the monitoring of control click behavior. Since the control association between different versions is fixed, the testing method of this application has low maintenance cost.

[0169] In some embodiments, the current user interface can be obtained in the following ways:

[0170] The device type of the target device is determined, and the corresponding interface acquisition instruction is determined based on the device type. Furthermore, an interface acquisition instruction can be sent to the target device, so that the target device, upon receiving the interface acquisition instruction, sends the current operation interface.

[0171] As an example, if the target device is an Android system, you can send the command `adb exec-out screencap-p>screenshot.png` to instruct the target device to take a screenshot. If it is an iOS system, you can send the command `idevicescreenshot screenshot.png` to instruct the target iOS device to take a screenshot according to the above command.

[0172] In some scenarios, before the target device launches the application to be tested, an application launch command can be generated based on the application identifier of the application to be tested and sent to the target device.

[0173] In some embodiments, the determination of the control category and display position can be achieved through a network model, specifically through a YOLO v7 network. Of course, other versions of the YOLO network, or other networks capable of object detection, can also be used; this application does not impose any specific limitations on this.

[0174] In some scenarios, UI icon libraries can be used to fine-tune the initial network model (such as a pre-trained YOLO model). Unlike the training of other deep neural networks, this application does not require generalization ability; it only needs to pursue the highest accuracy within a limited UI icon library. Therefore, in the model training and update part, this application does not split the UI icon library into training and test sets with no overlap, but uses identical training and test sets. In this case, the network model may overfit, achieving the highest accuracy within the UI icon library. When there are other customized controls in the application (such as characters in game applications, cards in card-based applications, etc.), overfitting techniques cannot be used for this part of the data. Instead, the training and test sets need to be split to ensure a certain degree of generalization ability, that is, to ensure that the customized control can be recognized based on images from different angles.

[0175] During system operation, the App image obtained by the image acquisition module will serve as input to the neural network model (i.e., the fine-tuned YOLO model) in the AI ​​module, and will output the positions and labels of all UI icons and other regions of interest (ROIs) contained in the current image, such as... Figure 13 As shown. This data will be input into the operation decision module to determine the next interface operation and location information.

[0176] In some embodiments, the size of the detection box used in this application can be determined based on the size of the actual control box. The following are specific methods for determining the size of the detection box:

[0177] First, the width and height of the target bounding box for each control in the UI icon library can be calculated, and a distribution histogram or scatter plot can be drawn to identify the main size range and common aspect ratios. Furthermore, the width and height can be converted to a ratio relative to the input image size (e.g., divided by the image width and height) to accommodate different resolutions.

[0178] Then, K-means clustering can be used, taking the width and height of the target boxes as input, and using an improved distance metric for clustering, selecting the cluster centers as the detection box sizes. For example, the YOLO series uses this method to generate detection boxes that fit the data distribution. Furthermore, the number of clusters can be determined, and multiple target boxes can be set for each feature layer based on model complexity and computational resource balance.

[0179] Next, different sized bounding boxes can be designed based on the hierarchy of the feature maps. Specifically, in a Feature Pyramid Network (FPN), high-level feature maps (low resolution) correspond to large targets and use larger bounding boxes; low-level feature maps (high resolution) correspond to small targets and use smaller bounding boxes. For example, in a ResNet-based FPN, layer P3 (stride 8) is responsible for small targets, and layer P5 (stride 32) is responsible for large targets.

[0180] Furthermore, the aspect ratio can be selected based on the data distribution, such as 1:1, 1:2, or 2:1, to determine the detection box of different proportions for each position.

[0181] Finally, verify the average IoU between the detection bounding boxes and the ground truth bounding boxes. In one pass, an IoU higher than 0.5 is considered valid. Compare the detection accuracy of different detection box configurations and adjust the number, size, and ratio of the detection boxes accordingly.

[0182] As an example, the COCO dataset statistics show that the width and height of target objects are concentrated in the range of [10, 200] pixels, with common ratios of 1:1, 1:2, and 2:1. Further, K-means clustering is performed, resulting in five cluster centers, such as (24, 24), (48, 32), and (96, 64). Then, multi-scale allocation is performed, for example, setting the anchor box sizes in layers P3-P7 of the FPN to [24, 24], [48, 32], [96, 64], [160, 128], and [256, 192], respectively. Finally, training and validation are performed. For example, the initial average IoU is 0.62; by adjusting the number of clusters to 6, the IoU increases to 0.68, improving detection accuracy and allowing the determination of the final detection box size.

[0183] See Figure 14 As shown, taking the need to test the notification operation interface in Application 1 as an example, the testing method of the operation interface in this application is described in detail. The specific process is as follows:

[0184] S1401, based on the preset interface testing requirements, the interface to be tested for application 1 in the target device is determined to be the notification operation interface.

[0185] S1402, based on the hierarchical connection relationship between the interface identifiers preset by Application 1, determine the connection path between the notification operation interface and the main operation interface.

[0186] As an example, the connection path between the main operation interface and the notification operation interface can be represented as follows:

[0187] Main interface → Settings interface → General interface → Notification interface.

[0188] S1403, send a startup command to the target device to start application 1, and receive the main operation interface of the target device after starting application 1.

[0189] S1404, extract features from the main operation interface to determine the position and type of each control included in the main operation interface.

[0190] As an example, suppose the main interface of application 1 includes two controls: login and settings. When performing control recognition on the main interface, it can be identified that the control types of each control included in the main interface are login and settings, respectively.

[0191] S1405, determine the interface identifier of the settings interface, and determine the control category that matches the interface identifier from the obtained control categories.

[0192] Continuing with the example above, the next interface after the main interface is the settings interface, where the control category of the settings control can be determined from the obtained control categories.

[0193] S1406, based on the display position associated with the settings control, sends a click command to the target device to obtain the settings interface.

[0194] S1407, Perform feature extraction on the settings interface to determine the position and type of each control included in the settings interface.

[0195] As an example, suppose the settings interface of Application 1 includes three controls: General, Device, and Account. When performing control recognition on the settings interface, it is possible to identify the three control categories: General, Device, and Account, as well as their respective display positions.

[0196] S1408, determine the interface identifier of the general interface, and determine the control category that matches the general interface identifier from the obtained control categories.

[0197] S1409 sends a click command to the target device based on the display position associated with the general control, and obtains the general interface.

[0198] S1410, extract features from the general interface to determine the position and type of each control included in the general interface.

[0199] As an example, suppose the general interface of application 1 includes two controls: notification and information. When performing control recognition on the general interface, the two control categories of notification and information, as well as their respective display positions, can be identified.

[0200] S1411, determine the interface identifier of the notification interface, and determine the control category that matches the interface identifier of the notification from the obtained control categories.

[0201] S1412, based on the display position associated with the notification control, sends a click command to the target device and obtains the notification interface.

[0202] In some scenarios, in addition to the click operation mentioned above, interface operations may also include drag operation, long press operation, etc. The above example is based on the click operation, but it can be replaced by drag operation. This application does not make specific limitations on this.

[0203] See Figure 15 As shown, taking the application to be tested as application 2, it is necessary to drag and drop the character 1 in interface 1 for testing. The specific process is as follows:

[0204] S1501, based on the preset interface testing requirements, the interface to be tested for application 2 on the target device is determined to be the operation interface after dragging the character 1 in interface 1.

[0205] In some scenarios, UI testing requirements may include drag-and-drop operation parameters. For example, drag direction, drag distance, etc.

[0206] S1502, based on the hierarchical connection relationship between the interface identifiers preset by Application 2, determine the connection path between the interface to be tested and the main operation interface.

[0207] As an example, the connection path between the main operation interface and the notification operation interface can be represented as follows:

[0208] Main operation interface → Interface 1 → Interface to be tested.

[0209] S1503, send a startup command to the target device to start application 2, and receive the main operation interface of the target device after starting application 2.

[0210] S1504, extract features from the main operation interface to determine the position and type of each control included in the main operation interface.

[0211] As an example, suppose the main interface of application 2 includes two controls: login and settings. When performing control recognition on the main interface, it can be identified that the control types of each control included in the main interface are login and settings, respectively.

[0212] S1505, determine the interface identifier of interface 1, and determine the control category that matches the interface identifier from the obtained control categories.

[0213] Continuing with the example above, the next interface after the main interface is Interface 1. Interface 1 is associated with the login control; that is, clicking the login control will display Interface 1.

[0214] S1506, based on the display position associated with the login control, sends a click command to the target device and obtains Interface 1.

[0215] S1507, Perform feature extraction on interface 1 to determine the position and type of each control included in interface 1.

[0216] As an example, suppose that Interface 1 of Application 2 includes three characters. Then, determine the character categories of the three characters, where the character category can be the game name of the task.

[0217] S1508, determine the display position of Person 1 as the task category, and send a drag command to the target device based on the drag operation parameters and the display position to obtain the operation interface after dragging Person 1.

[0218] Assuming Application 2 is a multiplayer online role-playing game, and the interface testing requirement is to obtain the interface to be tested by dragging Control 1 in Interface 2, the connection relationship of each interface identifier is determined as follows: Main interface - Interface 1 - Interface 2 - Interface to be tested.

[0219] First, after the target device launches application 2, the main user interface obtained is as follows: Figure 16 As shown in the figure. Further, control recognition can be performed on the main operation interface, identifying two controls: "Enter Game" and "Exit," as well as the position of each control. The control recognition results are shown in the figure with dashed boxes.

[0220] Then, since interface 1 will only be accessed after clicking the game control, meaning that the game control and interface 1 are related, a click command is sent to the target device based on the control position of the login control, and the interface 1 displayed on the target device is obtained.

[0221] Furthermore, control recognition can be performed on interface 1 to determine the various controls included in interface 1 and their positions, such as... Figure 16As shown. Assuming interface 2 is a "grass" interface, it is necessary to determine the target control associated with the "grass" interface identifier from the various controls identified in interface 1.

[0222] Next, after clicking the target control, obtain interface 2 displayed on the target device, namely the "grass" interface. If the interface to be tested is an operation interface for dragging control 1, then it is necessary to identify control 1 within the "grass" interface. Further, based on the drag operation parameters in the interface testing requirements and the control position of control 1, send a drag command to the target device and obtain the operation interface after dragging control 1 as the interface to be tested. The obtained interface to be tested is as follows: Figure 16 As shown.

[0223] Based on the same technical concept, see [link / reference] Figure 17 The image shows a user interface testing device 1700 provided in an embodiment of this application. This device 1700 can implement any step of the aforementioned user interface testing method. To avoid repetition, further details are omitted here. The device 1700 includes a first processing module 1701, a second processing module 1702, and a third processing module 1703.

[0224] The first processing module 1701 is used to determine the interface to be tested of the application to be tested in the target device based on preset interface testing requirements.

[0225] The second processing module 1702 is used to determine the connection path between the interface to be tested and the main operation interface based on the hierarchical connection relationship between the interface identifiers preset for the application to be tested; the connection path includes: the interface identifiers of each operation interface presented during the process of switching from the main operation interface to the interface to be tested.

[0226] The third processing module 1703 is used to perform the following operations in a loop, starting from the main operation interface obtained, after the application to be tested is launched on the target device, until the interface to be tested is obtained:

[0227] The current operation interface is identified to obtain the control category and display position of at least one control in the current operation interface.

[0228] Based on the connection path, obtain the interface identifier of the next operation interface, and determine the control category that matches the interface identifier from at least one obtained control category;

[0229] Based on the display position associated with the matched control category, a click command is sent to the target device to obtain the next operation interface, and the next operation interface is used as the new current operation interface.

[0230] In one possible implementation, the third processing module 1703, when performing control identification on the acquired current operation interface and determining the control category and display position of at least one control in the current operation interface, is specifically used for:

[0231] According to multiple preset scales, features are extracted from the current operation interface to obtain corresponding feature maps;

[0232] For each feature map, perform the following operations: select at least one matching detection box according to the resolution of the feature map, and perform control recognition on the feature map;

[0233] Based on the control recognition results of each detection box, the control category and display position of at least one control in the current operation interface are determined.

[0234] In one possible implementation, the third processing module 1703, when selecting at least one matching detection box according to the resolution of the feature map and performing control recognition on the feature map, is specifically used for:

[0235] Based on a preset correlation between resolution and detection box size, the target size of at least one detection box associated with the resolution of the feature map is obtained;

[0236] The feature map is divided into multiple grids, each grid representing a pixel feature at a spatial location in the feature map;

[0237] For each of the grids, using each of the detection boxes, the following operations are performed respectively: taking one grid as the center of a detection box, determining each grid covered by the target size of the detection box, and performing control recognition in the detection box based on the pixel features represented by each grid.

[0238] In one possible implementation, each control identification result includes: the bounding box parameters of the corresponding detection box, the confidence score, and the control category probability; the confidence score is used to represent the predicted probability that a control exists within the detection box;

[0239] The third processing module 1703, when determining the control category and display position of at least one control in the current operation interface based on the control recognition results of each detection box, is specifically used for:

[0240] Based on the control category probability of at least one detection box in each feature map, the control category of at least one control included in the current operation interface is determined.

[0241] Aggregate the confidence scores of at least one checkbox with the same control category to obtain the confidence score set corresponding to each control category.

[0242] For each control category, perform the following operations: Based on the confidence set corresponding to a control category, select a detection box that meets the confidence condition from the detection boxes corresponding to each confidence level in the confidence set, and determine the display position of the control represented by the control category based on the bounding box parameters of the detection box.

[0243] In one possible implementation, the third processing module 1703 obtains the current operation interface in the following way:

[0244] Determine the device type of the target device, and determine the corresponding interface acquisition instruction based on the device type;

[0245] The interface acquisition instruction is sent to the target device so that the target device can send the current operation interface after receiving the interface acquisition instruction.

[0246] In one possible implementation, the interface testing requirements include a device identifier, an application identifier, and an interface identifier;

[0247] The first processing module 1701, when determining the interface to be tested of the application to be tested in the target device based on preset interface testing requirements, is specifically used for:

[0248] The target device for which interface testing needs to be performed is determined based on the device identifier;

[0249] Based on the application identifier, the application to be tested is determined in the target device;

[0250] Based on the interface identifier, the interface to be tested in the application to be tested is determined from the interface configuration data of the application to be tested.

[0251] In one possible implementation, the second processing module 1702 is further configured to:

[0252] Before the target device launches the application to be tested, an application launch command is generated based on the application identifier of the application to be tested.

[0253] Send the application launch command to the target device.

[0254] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0255] After introducing the testing method and apparatus for the user interface of the exemplary embodiments of this application, the electronic device according to another exemplary embodiment of this application will be introduced next.

[0256] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0257] Based on the same technical concept as the above-described method embodiments, this application also provides an electronic device. In this embodiment, the structure of the electronic device can be as follows: Figure 18 As shown, it includes a memory 1801, a communication module 1803, and one or more processors 1802.

[0258] The memory 1801 is used to store computer programs executed by the processor 1802. The memory 1801 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0259] Memory 1801 may be volatile memory, such as random-access memory (RAM); memory 1801 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1801 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1801 may be a combination of the above-described memories.

[0260] Processor 1802 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 1802 is used to implement the above-mentioned test method of operating interface when calling computer programs stored in memory 1801.

[0261] The communication module 1803 is used to communicate with terminal devices and other servers.

[0262] This application embodiment does not limit the specific connection medium between the memory 1801, communication module 1803, and processor 1802. This application embodiment... Figure 18 The memory 1801 and the processor 1802 are connected via a bus 1804, and the bus 1804 is in Figure 18 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 1804 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 18 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0263] The memory 1801 stores a computer storage medium, which stores computer-executable instructions. These instructions are used to implement the user interface testing method of this application embodiment. The processor 1802 is used to execute the aforementioned user interface testing method, such as... Figure 6 As shown.

[0264] In some possible implementations, various aspects of the user interface testing method provided in this application can also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program causes the electronic device to perform the steps in the user interface testing method according to the various exemplary embodiments of this application described above. For example, the electronic device can perform actions such as... Figure 6 The steps are shown in the figure.

[0265] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0266] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0267] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0268] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0269] Computer programs for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The computer program can execute entirely on the user's electronic device, partially on the user's electronic device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0270] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0271] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0272] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0273] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0274] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0275] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0276] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for testing a user interface, characterized in that, The method includes: Based on the preset interface testing requirements, determine the interface to be tested for the application to be tested on the target device; Based on the hierarchical connection relationship between the interface identifiers preset for the application under test, the connection path between the interface under test and the main operation interface is determined; the connection path includes: the interface identifiers of each operation interface presented during the process of switching from the main operation interface to the interface under test. After the application under test is launched on the target device, the following operations are performed in a loop, starting from the main operation interface obtained, until the interface under test is obtained: The current operation interface is identified to obtain the control category and display position of at least one control in the current operation interface. Based on the connection path, obtain the interface identifier of the next operation interface, and determine the control category that matches the interface identifier from at least one obtained control category; Based on the display position associated with the matched control category, a click command is sent to the target device to obtain the next operation interface, and the next operation interface is used as the new current operation interface.

2. The method as described in claim 1, characterized in that, The step of identifying controls on the acquired current operation interface and determining the control category and display position of at least one control in the current operation interface includes: According to multiple preset scales, features are extracted from the current operation interface to obtain corresponding feature maps; For each feature map, perform the following operations: select at least one matching detection box according to the resolution of the feature map, and perform control recognition on the feature map; Based on the control recognition results of each detection box, the control category and display position of at least one control in the current operation interface are determined.

3. The method as described in claim 2, characterized in that, The step of selecting at least one matching detection box according to the resolution of the feature map and performing control recognition on the feature map includes: Based on a preset correlation between resolution and detection box size, the target size of at least one detection box associated with the resolution of the feature map is obtained; The feature map is divided into multiple grids, each grid representing a pixel feature at a spatial location in the feature map; For each of the grids, using each of the detection boxes, the following operations are performed respectively: taking one grid as the center of a detection box, determining each grid covered by the target size of the detection box, and performing control recognition in the detection box based on the pixel features represented by each grid.

4. The method as described in claim 2, characterized in that, Each control identification result includes: the bounding box parameters of the corresponding detection box, the confidence score, and the control category probability; the confidence score is used to represent the predicted probability that a control exists within the detection box; The step of determining the control category and display position of at least one control in the current operation interface based on the control recognition results of each detection box includes: Based on the control category probability of at least one detection box in each feature map, the control category of at least one control included in the current operation interface is determined. Aggregate the confidence scores of at least one checkbox with the same control category to obtain the confidence score set corresponding to each control category. For each control category, perform the following operations: Based on the confidence set corresponding to a control category, select a detection box that meets the confidence condition from the detection boxes corresponding to each confidence level in the confidence set, and determine the display position of the control represented by the control category based on the bounding box parameters of the detection box.

5. The method according to any one of claims 1-4, characterized in that, The current user interface can be obtained in the following ways: Determine the device type of the target device, and determine the corresponding interface acquisition instruction based on the device type; The interface acquisition instruction is sent to the target device so that the target device can send the current operation interface after receiving the interface acquisition instruction.

6. The method according to any one of claims 1-4, characterized in that, The interface testing requirements include device identifier, application identifier, and interface identifier; The process of determining the interface to be tested for the application on the target device based on preset interface testing requirements includes: The target device for which interface testing needs to be performed is determined based on the device identifier; Based on the application identifier, the application to be tested is determined in the target device; Based on the interface identifier, the interface to be tested in the application to be tested is determined from the interface configuration data of the application to be tested.

7. The method as described in claim 6, characterized in that, The method further includes: Before the target device launches the application to be tested, an application launch command is generated based on the application identifier of the application to be tested. Send the application launch command to the target device.

8. A testing device with an operating interface, characterized in that, The device includes: The first processing module is used to determine the interface to be tested of the application to be tested on the target device based on the preset interface testing requirements. The second processing module is used to determine the connection path between the interface to be tested and the main operation interface based on the hierarchical connection relationship between the interface identifiers preset for the application to be tested; the connection path includes: the interface identifiers of each operation interface presented during the process of switching from the main operation interface to the interface to be tested. The third processing module is used to perform the following operations in a loop, starting from the main operation interface obtained, after the application to be tested is launched on the target device, until the interface to be tested is obtained: The current operation interface is identified to obtain the control category and display position of at least one control in the current operation interface. Based on the connection path, obtain the interface identifier of the next operation interface, and determine the control category that matches the interface identifier from at least one obtained control category; Based on the display position associated with the matched control category, a click command is sent to the target device to obtain the next operation interface, and the next operation interface is used as the new current operation interface.

9. An electronic device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method as described in any one of claims 1 to 7 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The method includes a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any one of claims 1 to 7.

Images