User interface test method and related device

Abstract

The invention discloses a user interface testing method and a related device, and relates to the field of software testing on the basis of obtaining a user interface interaction video recorded by a user. Decomposing the user interface interaction video frame by frame according to the frame number of the user interface interaction video to obtain a first picture group; and splitting the first picture group according to whether the adjacent pictures contain continuously played display contents or not to obtain at least one second picture group. And identifying the user operation behavior on each second picture group according to the operation behavior identification strategy to obtain the operation behavior of the user. And according to the occurrence sequence of each operation behavior and the corresponding relationship between each operation behavior and each second picture group, obtaining an automatic test script, and executing the automatic test script for testing. The automatic test script can be generated as long as the user records the user interface interaction video needing to be realized. Complex manual operation can be omitted, and the testing efficiency is improved while manpower is saved.

Description

Technical Field

[0001] The present application relates to the field of software testing technology, and in particular to a user interface testing method and related devices. Background Art

[0002] UI automation (i.e. user interface automation) is a technology that simulates the interaction between users and software interfaces by writing codes or scripts. It can automatically complete a series of tasks without human intervention, replacing manual repetitive and tedious tasks, thereby freeing up human resources and improving work efficiency. However, the current process of generating automated scripts takes a long time for users, resulting in low generation efficiency, and has gradually failed to meet testing needs. Summary of the invention

[0003] In view of the above problems, the present application provides a user interface testing method and related devices to achieve the purpose of reducing manual operations and improving testing efficiency. The specific scheme is as follows:

[0004] The first aspect of the present application provides a user interface testing method, comprising:

[0005] Get user interface interaction videos recorded by users;

[0006] Decomposing the user interface interaction video frame by frame according to the number of frames of the user interface interaction video to obtain a first picture group;

[0007] Splitting the first picture group according to whether adjacent pictures contain display content to be played continuously, to obtain at least one second picture group;

[0008] Identify the user operation behavior on each of the second picture groups according to the operation behavior identification strategy to obtain the user operation behavior;

[0009] According to the occurrence order of each of the operation behaviors and the corresponding relationship between each of the operation behaviors and each of the second picture groups, an automatic test script is obtained, and the automatic test script is executed to perform the test.

[0010] In a possible implementation, the user interface testing method further includes:

[0011] Recording the user interface for executing the automatic test script to obtain a script execution video;

[0012] Segmenting the script execution video according to the occurrence position of each operation behavior to obtain script execution sub-videos;

[0013] Decomposing the script execution sub-video frame by frame according to whether each script execution sub-video has dynamically played content, to obtain a third picture group;

[0014] According to the correspondence between each of the second picture groups and the third picture groups, the matching degree between the script execution video and the automatic test script is judged.

[0015] In a possible implementation, before segmenting the script execution video according to the occurrence position of each operation behavior to obtain the script execution sub-video, the method further includes:

[0016] According to the matching relationship between the occurrence position of each operation behavior and the target picture, it is determined whether the occurrence position of each operation behavior is accurate, and the target picture is the picture corresponding to the operation behavior in the second picture group.

[0017] In a possible implementation, the user interface testing method further includes:

[0018] The operation behavior is identified according to the page elements contained in the target image.

[0019] In a possible implementation, the user interface testing method further includes:

[0020] When executing the automatic test script, assertion verification is performed based on the second picture group, the third picture group, each of the operation behaviors, and the order in which each of the operation behaviors occurs.

[0021] In a possible implementation, the user interface interaction video is a recorded video with a gesture operation record, and the user operation behavior on each of the second picture groups is identified according to the operation behavior identification strategy to obtain the user's operation behavior, including:

[0022] Determining the user's sliding operation behavior and sliding position according to the position and duration of the picture with the gesture record in the second picture group;

[0023] The user's selection operation behavior and selection position are determined according to the positioning of the page elements in the pictures with gesture records during the playback of the second picture group.

[0024] In a possible implementation, the first picture group is split according to whether adjacent pictures contain continuously played display content to obtain at least one second picture group, including:

[0025] If there is a continuously changing area between adjacent pictures within a preset number of frames in the first picture group, it is determined that there is automatically played content;

[0026] The playback progress or speed of the automatically played content obtained according to the picture within the preset frame number is used to determine the playback progress operation behavior and the operation position.

[0027] A second aspect of the present application provides a user interface testing device, comprising:

[0028] A recorded video acquisition module is used to acquire user interface interaction videos recorded by users;

[0029] A recorded video decomposition module, used for decomposing the user interface interaction video frame by frame according to the number of frames of the user interface interaction video to obtain a first picture group;

[0030] A picture group splitting module, configured to split the first picture group according to whether adjacent pictures contain continuously played display content, to obtain at least one second picture group;

[0031] an operation behavior recognition module, configured to recognize the user operation behavior on each of the second picture groups according to the operation behavior recognition strategy, and obtain the user operation behavior; and

[0032] The automatic script generation module is used to obtain an automatic test script according to the occurrence order of each of the operation behaviors and the corresponding relationship between each of the operation behaviors and each of the second picture groups, and execute the automatic test script for testing.

[0033] A third aspect of the present application provides a computer program product, comprising computer-readable instructions, which, when executed on an electronic device, enables the electronic device to implement the user interface testing method of the first aspect or any implementation of the first aspect.

[0034] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0035] The memory is used to store computer programs;

[0036] The processor is used to execute the computer program so that the electronic device can implement the user interface testing method of the above-mentioned first aspect or any implementation manner of the first aspect.

[0037] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the user interface testing method of the above-mentioned first aspect or any implementation of the first aspect.

[0038] By means of the above technical scheme, the user interface testing method provided by the present application is based on obtaining the user interface interaction video recorded by the user. The user interface interaction video is decomposed frame by frame according to the number of frames of the user interface interaction video to obtain a first picture group. The first picture group is split according to whether the adjacent pictures contain continuously played display content to obtain at least one second picture group. The user operation behavior on each second picture group is identified according to the operation behavior recognition strategy to obtain the user's operation behavior. According to the order of occurrence of each operation behavior and the correspondence between each operation behavior and each second picture group, an automatic test script is obtained, and the automatic test script is executed for testing. The user interface testing method only requires the user to record the video of the user interface interaction to be implemented, and can identify the two-level user operation behavior and position according to the video content, and then generate an automatic test script according to the order and position of the operation behavior. Complex manual operations can be eliminated, which saves manpower and improves testing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0040] Figure 1 An architectural diagram of a user interface testing system provided for this application;

[0041] Figure 2 An architectural diagram of a terminal provided for this application;

[0042] Figure 3 An architectural diagram of a server provided for this application;

[0043] Figure 4 A flowchart of a user interface testing method provided in this application;

[0044] Figure 5 A structural diagram of a user interface testing device provided in this application;

[0045] Figure 6 A structural diagram of an electronic device provided in this application. DETAILED DESCRIPTION

[0046] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0047] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0048] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0049] See also Figure 1 , Figure 1 FIG. 1 shows a schematic diagram of the architecture of a user interface testing system. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the user interface testing method provided in the embodiment of the present application for one or more terminals.

[0050] Among them, an application for uploading user interface interaction videos can be installed on the terminal 100. The above application and web page can provide an interface. The terminal 100 can receive the user interface interaction video upload made by the user on the interface for uploading user interface interaction videos, and send the above user interface interaction video to the server 200. The server 200 can obtain the processing result based on the received parameters and return the processing result to the terminal 100.

[0051] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited to this.

[0052] Next describe Figure 1 The product form of the mid-terminal 100;

[0053] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any limitation on this.

[0054] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0055] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0056] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0057] Among them, the input device 132 can receive input user interface interaction video and the like.

[0058] The display unit 140 may be used to display information input by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file display and / or playback of any multimedia file. In an embodiment of the present application, the display unit 140 may be used to display an interface for user interface interactive video upload, processing results, etc.

[0059] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0060] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0061] Among them, the memory 120 can be used to store software codes related to the user interface testing method, the processor 170 can execute the steps of the user interface testing method, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to implement corresponding functions.

[0062] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0063] In this embodiment of the present application, the RF unit 110 can send the user interface interaction video to the server 200 and receive the processing result sent by the server 200.

[0064] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0065] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0066] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0067] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0068] Next describe Figure 1 The product form of the server 200;

[0069] Figure 3 A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0070] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0071] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0072] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0073] The memory 204 may be used to store software codes related to the user interface testing method, and the processor 202 may execute the steps of the user interface testing method of the chip, and may also schedule other units to implement corresponding functions.

[0074] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0075] The embodiment of the present application provides a user interface testing method. The user interface testing method of the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0076] Reference Figure 4 , Figure 4 A flowchart of a user interface testing method provided in an embodiment of the present application is shown as follows: Figure 4 As shown, a user interface testing method provided in an embodiment of the present application may include steps 401 to 405, and these steps are described in detail below.

[0077] 401. Obtain a user interface interaction video recorded by a user.

[0078] Specifically, the user interface interaction video may be a recorded video obtained by turning on the recording function on a terminal (such as a computer, tablet, smart phone, etc.) when the user performs actual operations in the actual interactive interface. The user can set and select the recording duration as needed, and there is no restriction here. For example, a user accesses a certain software on a smart phone, turns on the recording after opening the software, clicks on an element (such as a video) on the homepage, and watches it, and then returns to the previous page before finishing watching the video, and then stops recording, and obtains a user interface interaction video that includes the entire operation process.

[0079] 402. Decompose the user interface interaction video frame by frame according to the number of frames of the user interface interaction video to obtain a first picture group.

[0080] Specifically, on the basis of obtaining the user interface interaction video, the user interface interaction video is decomposed into pictures frame by frame according to the number of video frames contained in the user interface interaction video to obtain a first picture group.

[0081] 403. Split the first picture group according to whether adjacent pictures contain display content to be played continuously, to obtain at least one second picture group.

[0082] Specifically, according to the actual operation scenario, the user's operation process in the user interface can be understood as a nested process, such as browsing by sliding on the homepage, and then selecting by clicking to enter the corresponding selection interface, and then performing other operations in the selected interface, for example, when the opened interface is a video playback, there may be operations such as fast forwarding and pausing the video. According to the rules of this operation, the sliding and clicking selection operation behaviors of the pictures in the first picture group can be preferentially identified, and these operation behaviors and the locations where these behaviors occur in the user interface interaction video can be identified.

[0083] In order to facilitate the positioning and identification of operation behaviors, when recording user interface interaction videos, the smartphone can be turned on in developer mode to record operation gestures, such as touch start, touch move, touch end and other events. On this basis, the operation behavior and operation position are determined. When making specific determinations, the user's sliding operation behavior and sliding position can be determined based on the position and duration of the pictures with gesture records in the second picture group. The user's selection operation behavior and selection position are determined based on the positioning of page elements in the pictures with gesture records during the playback of the second picture group.

[0084] When identifying the operation behavior based on the operation gesture, the operation behavior can be identified based on the comparison of the image content before and after the operation. For example, for the user's sliding operation, starting from the first image recording the user's sliding gesture, the user's gesture is recorded at a certain position, where the user's gesture is after sliding, how long it takes, and how long it slides. By recording the image display of the start time of the image recording, the sliding distance and time, the background display during sliding, and the effect image after the sliding stops, and recording them into a group of images after the operation, by comparing the two groups of images before and after the sliding gesture, the image of the final page position of the sliding is obtained.

[0085] Then, the image where the click gesture is located is played back, and the image changes in the image group of the expansion effect of the element selected during the playback (for example, playing a video) are compared during the expansion process to determine the target element of the operation (that is, the location where the click operation occurs).

[0086] It is understandable that those skilled in the art may also identify the second picture group according to other strategies to obtain the identification result of the user's operation behavior, which is not limited here.

[0087] 404. Identify the user operation behavior on each second picture group according to the operation behavior identification strategy to obtain the user operation behavior.

[0088] Specifically, the first picture group can be divided into multiple second picture groups according to the identified nested dynamic content such as playing videos and animated pictures. For example, after identifying that the user opens the element on the selection page, there may be automatically playing videos, animated pictures and other dynamic effects in the new page. These pictures are used as the second picture group, and then the corresponding operation behavior recognition strategy can be used to recognize the user's operation behavior. The operation behavior recognition strategy can be the same as the above-mentioned operation behavior recognition strategy, or it can be different from the above-mentioned operation behavior recognition strategy. In order to realize the recognition of operations such as pause and fast forward on the playing video, when it is judged that there is a continuously changing area in the pictures within the preset number of frames in the second picture group, it is judged that there is an automatically playing video. Then, the playback progress operation behavior and the operation position can be determined based on the playback progress or speed of the automatically playing video obtained from the pictures within the preset number of frames.

[0089] For example, after the first picture group is cut, after the video loading is completed according to the pictures in the obtained second picture group, the pictures between the second frame and the tenth frame are checked, and the changes in the picture content are used to determine whether there is dynamic video playback. If the content of the pictures between the second frame and the tenth frame is consistent, it can be determined that the video is not automatically played. On the contrary, when the content of the pictures between the second frame and the tenth frame continues to change, it can be determined that there is an automatically played video. And according to the change time on the playback progress bar in each picture, it can be determined whether there are operations such as fast forward and pause.

[0090] 405. Obtain an automatic test script according to the occurrence sequence of each operation behavior and the corresponding relationship between each operation behavior and each second picture group, and execute the automatic test script to perform testing.

[0091] Specifically, after obtaining the above-mentioned user operation behaviors and the locations and sequences of the operation behaviors, an automatic test script can be generated, and the user interface interaction test can be performed based on the automatic test script. When generating the automatic test script, a currently commonly used automatic test script generation platform can be used to generate the automatic test script based on the information of the above-mentioned operation behaviors, which will not be repeated here.

[0092] From the above, it can be seen that the user interface testing method only needs the user to record the required user interface interaction video to generate the corresponding automatic test script, which can effectively reduce the amount of manual operation and improve testing efficiency.

[0093] Considering the current UI automation platform or automation code for recording or manually generating automation scripts, the generated automation scripts only have the generation part but no assertion part. In addition, because some models may not be compatible, testers are required to compare one by one when making assertions, which further reduces the test efficiency.

[0094] The user interface testing method also includes the following processing steps:

[0095] Step 11: Record the user interface that executes the automatic test script to obtain a script execution video.

[0096] Step 12: segment the script execution video according to the occurrence location of each operation behavior to obtain script execution sub-videos.

[0097] Step 13: Decompose the script execution sub-video frame by frame according to whether each script execution sub-video has dynamically played content to obtain a third picture group.

[0098] Step 14: According to the correspondence between each second picture group and the third picture group, the matching degree between the script execution video and the automatic test script is judged.

[0099] Before the script execution video is segmented according to the occurrence position of each operation behavior to obtain the script execution sub-video, the method further includes:

[0100] According to the matching relationship between the occurrence position of each operation behavior and the target picture, it is determined whether the occurrence position of each operation behavior is accurate, wherein the target picture is the picture corresponding to the operation behavior in the second picture group. Then, the operation behavior is identified according to the page elements contained in the target picture.

[0101] Specifically, when verifying the accuracy of the operation of the automatic test script, first match the pictures divided by the number of frames in the video recorded by the automatic test script according to the location of the operation behavior obtained in the above processing steps. And divide the script execution video by the pictures before and after the operation behavior location to obtain the script execution sub-video. Then, determine whether there is an automatically played video in the script execution sub-video based on the picture content. If there is a video, divide the script execution sub-video according to the number of frames contained to obtain a third picture group, and record the size of the video. When comparing according to the picture group, first compare the outermost picture group (that is, the second picture group records the user's operation behavior), and then match the video content after the comparison is passed, that is, match the third picture group.

[0102] After each target image is matched with the corresponding image in the script execution sub-video, the user operation behavior is identified according to the page elements contained in the target image.

[0103] When performing specific operations, you can wait for the pictures where the user's operation behavior appeared to be locked and matched, and record the content of the picture. In order to prevent inaccurate position locking, record the content while performing the operation. After all behaviors that match the user's operation and pictures are locked and matched, the page elements are located to the user's operation behaviors through the recorded content, making it more accurate when positioning. The picture group obtained in the process is processed for compatibility so that it can match most models. If there is a deviation, use the element position during recording for positioning, and record this model for autonomous cutting and placement for adaptation later.

[0104] In order to avoid changes in the recognition of page elements due to different smart terminal models, the order of elements can be recorded in a table when locating page elements. Therefore, when the size of the interactive interface changes, the elements can be accurately located according to their order to avoid inaccurate recognition results due to different page sizes.

[0105] In some specific embodiments, in order to reduce the process of assertion processing according to each use case and improve the efficiency of automatic test script assertion verification, the user interface testing method further includes:

[0106] When executing the automatic test script, assertion verification is performed based on the second picture group, the third picture group, each operation behavior, and the occurrence order of each operation behavior.

[0107] Specifically, based on the various picture groups obtained in the above-mentioned automatic test script generation process and the various picture groups obtained by verifying the automatic test script, a comparison and verification is performed in combination with the operation behavior. If they can all correspond, the comparison passes. If they do not match, a picture difference comparison is performed, and the comparison result is input to the user, so that the user only needs to record a video of a use case when performing UI automation to perform autonomous and automated assertions of compatibility.

[0108] As a specific application of the above user interface testing method, the specific testing and assertion verification process may include:

[0109] Content maintenance:

[0110] The interface interaction recording videos uploaded by users are parsed and decomposed into sequential image groups according to whether they contain automatically played videos.

[0111] An operation behavior group is obtained by identifying the user operation behavior and operation position in the picture group.

[0112] Verify the generation:

[0113] Match user operation behaviors and element opening addresses, record each operation step, and compare the recorded content with the corresponding image group and behavior group. Automatically record and record user operation behaviors and process the corresponding adaptation content according to the automated script sequence.

[0114] Execution verification: Execute the automation script and make assertions based on the image group and operation group in the content maintenance. If the comparison is consistent, it is considered that the UI automation has passed. If the comparison does not match, the automation script is considered to have failed and the result is output.

[0115] Finally, if the user operation changes, you can edit the recorded video and upload it again.

[0116] A user interface testing method provided in an embodiment of the present application is introduced above, and a device for executing the above user interface testing method will be introduced below.

[0117] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a user interface testing device provided in an embodiment of the present application. Figure 5 As shown, the user interface testing device comprises:

[0118] The recorded video acquisition module 501 is used to acquire the user interface interaction video recorded by the user.

[0119] The recorded video decomposition module 502 is used to decompose the user interface interaction video frame by frame according to the number of frames of the user interface interaction video to obtain a first picture group.

[0120] The picture group splitting module 503 is used to split the first picture group according to whether the adjacent pictures contain continuously played display content to obtain at least one second picture group.

[0121] The operation behavior recognition module 504 is used to recognize the user's operation behavior on each second picture group according to the operation behavior recognition strategy to obtain the user's operation behavior.

[0122] The automatic script generation module 505 is used to obtain an automatic test script according to the occurrence order of each operation behavior and the corresponding relationship between each operation behavior and each second picture group, and execute the automatic test script for testing.

[0123] In a possible implementation, it further includes: a test verification module for

[0124] Record the user interface that executes the automatic test script to obtain a script execution video;

[0125] Segment the script execution video according to the occurrence position of each operation behavior to obtain script execution sub-videos;

[0126] Decomposing the script execution sub-video frame by frame according to whether each script execution sub-video has dynamically played content, to obtain a third picture group;

[0127] According to the corresponding relationship between each second picture group and the third picture group, the matching degree between the script execution video and the automatic test script is judged.

[0128] In a possible implementation, before segmenting the script execution video according to the occurrence position of each operation behavior to obtain the script execution sub-video, the test verification module is further used to:

[0129] According to the matching relationship between the occurrence position of each operation behavior and the target picture, it is determined whether the occurrence position of each operation behavior is accurate, and the target picture is the picture corresponding to the operation behavior in the second picture group.

[0130] In a possible implementation, the test verification module is further used to:

[0131] Identify the operation behavior based on the page elements contained in the target image.

[0132] In a possible implementation, the test verification module is further used to:

[0133] When executing the automatic test script, assertion verification is performed based on the second picture group, the third picture group, each operation behavior, and the occurrence order of each operation behavior.

[0134] In a possible implementation, the user interface interaction video is a recorded video with gesture operation records, and the operation behavior recognition module 504 recognizes the user operation behavior on each second picture group according to the operation behavior recognition strategy to obtain the user's operation behavior, including:

[0135] Determining the user's sliding operation behavior and sliding position according to the position and duration of the picture with the gesture record in the second picture group;

[0136] The user's selection operation behavior and selection position are determined according to the positioning of the page elements in the pictures with gesture records during the playback of the second picture group.

[0137] In a possible implementation, the picture group splitting module 503 splits the first picture group according to whether adjacent pictures contain continuously played display content to obtain at least one second picture group, including:

[0138] If there is a continuously changing area between adjacent pictures within a preset number of frames in the first picture group, it is determined that there is automatic playback content;

[0139] The playback progress or speed of the automatically played content obtained according to the picture within the preset frame number is used to determine the playback progress operation behavior and the operation position.

[0140] The present application also provides an electronic device in an embodiment. Figure 6 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0141] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0142] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0143] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the user interface testing methods provided in the embodiments of the present application.

[0144] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any user interface testing method provided in the embodiment of the present application.

[0145] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0146] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0147] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0148] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A user interface testing method, characterized in that: include: Get user interface interaction videos recorded by users; Decomposing the user interface interaction video frame by frame according to the number of frames of the user interface interaction video to obtain a first picture group; Splitting the first picture group according to whether adjacent pictures contain display content to be played continuously, to obtain at least one second picture group; Identify the user operation behavior on each of the second picture groups according to the operation behavior identification strategy to obtain the user operation behavior; According to the occurrence order of each of the operation behaviors and the corresponding relationship between each of the operation behaviors and each of the second picture groups, an automatic test script is obtained, and the automatic test script is executed to perform the test.

2. The user interface testing method according to claim 1, characterized in that: Also includes: Recording the user interface for executing the automatic test script to obtain a script execution video; Segmenting the script execution video according to the occurrence position of each operation behavior to obtain script execution sub-videos; Decomposing the script execution sub-video frame by frame according to whether each script execution sub-video has dynamically played content, to obtain a third picture group; According to the correspondence between each of the second picture groups and the third picture groups, the matching degree between the script execution video and the automatic test script is judged.

3. The user interface testing method according to claim 2, characterized in that: Before the script execution video is segmented according to the occurrence position of each operation behavior to obtain the script execution sub-video, the method further includes: According to the matching relationship between the occurrence position of each operation behavior and the target picture, it is determined whether the occurrence position of each operation behavior is accurate, and the target picture is the picture corresponding to the operation behavior in the second picture group.

4. The user interface testing method according to claim 3, characterized in that: Also includes: The operation behavior is identified according to the page elements contained in the target image.

5. The user interface testing method according to claim 2, characterized in that: Also includes: When executing the automatic test script, assertion verification is performed based on the second picture group, the third picture group, each of the operation behaviors, and the order in which each of the operation behaviors occurs.

6. The user interface testing method according to claim 1, characterized in that: The user interface interaction video is a recorded video with gesture operation records. The user operation behavior on each of the second picture groups is identified according to the operation behavior identification strategy to obtain the user's operation behavior, including: Determining the user's sliding operation behavior and sliding position according to the position and duration of the picture with the gesture record in the second picture group; The user's selection operation behavior and selection position are determined according to the positioning of the page elements in the pictures with gesture records during the playback of the second picture group.

7. The user interface testing method according to claim 1, characterized in that: The first picture group is split according to whether adjacent pictures contain continuously played display content to obtain at least one second picture group, including: If there is a continuously changing area between adjacent pictures within a preset number of frames in the first picture group, it is determined that there is automatically played content; The playback progress or speed of the automatically played content obtained according to the picture within the preset frame number is used to determine the playback progress operation behavior and the operation position.

8. A user interface testing device, characterized in that: include: A recorded video acquisition module is used to acquire user interface interaction videos recorded by users; A recorded video decomposition module, used for decomposing the user interface interaction video frame by frame according to the number of frames of the user interface interaction video to obtain a first picture group; A picture group splitting module, configured to split the first picture group according to whether adjacent pictures contain continuously played display content, to obtain at least one second picture group; An operation behavior recognition module, used to recognize the user operation behavior on each of the second picture groups according to the operation behavior recognition strategy, and obtain the user operation behavior; as well as The automatic script generation module is used to obtain an automatic test script according to the occurrence order of each of the operation behaviors and the corresponding relationship between each of the operation behaviors and each of the second picture groups, and execute the automatic test script for testing.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the user interface testing method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the user interface testing method as described in any one of claims 1 to 7.

Images