A mobile application GUI intelligent testing method and system

Abstract

The present invention discloses a mobile application GUI intelligent testing method and system, which relates to the technical field of GUI intelligent testing. The method comprises extracting attribute information of an initial interface element to be tested of a mobile application, calculating the image similarity between the initial interface to be tested and the standard image initial interface, and selecting different initial test processing strategies; after receiving an initial element test command, clicking on the initial interface elements to be tested in sequence to obtain a second element test interface to be tested, calculating the image similarity between the second element test interface to be tested and the standard second element test interface, and calculating the initial element test difference value, and issuing a GUI test abnormality warning; calculating the mean value of the performance coefficient of the mobile application on different platforms, and issuing a GUI test standard remake warning, which can timely discover problems existing in the mobile application on different platforms, and help the development team to quickly locate the problems and repair them.

Description

Technical Field

[0001] The present invention relates to the technical field of GUI intelligent testing, and in particular to a mobile application GUI intelligent testing method and system. Background Art

[0002] With the rapid development of mobile internet technology, mobile applications (apps) have become an integral part of our daily lives. As the primary means for users to interact with mobile applications, the quality and user experience of the graphical user interface (GUI) directly impacts the popularity and market competitiveness of mobile applications. Therefore, efficient and accurate testing of mobile application GUIs is crucial. Traditional GUI testing methods rely heavily on manual labor, which is not only time-consuming and labor-intensive but also susceptible to human factors, making it difficult to ensure the accuracy and consistency of test results. To address this issue, academia and industry have begun exploring the use of automated testing technologies to improve the efficiency and quality of GUI testing. However, existing automated testing tools and methods still have numerous limitations, such as inefficient test case generation and execution, insufficient test coverage, and limited ability to handle complex interaction scenarios.

[0003] In the Chinese invention application with application publication number CN113032243A, a mobile application GUI intelligent testing method and system are disclosed, including training a target detection model based on GUI screenshots with graphic element labels; training a deep reinforcement learning model based on a GUI interaction event sequence with both graphic element labels and interaction labels; obtaining a GUI screenshot of the current state of the mobile application; obtaining a GUI screenshot of the current state with graphic element labels based on the trained target detection model; detecting whether there are GUI defects in the graphic element labels; obtaining a GUI screenshot of the current state with graphic element labels and interaction labels based on the trained deep reinforcement learning model; applying an operation action to the operated graphic element based on the GUI screenshot of the current state with graphic element labels and interaction labels to update the state of the mobile application; and determining whether to stop testing based on the reward function in the deep reinforcement learning model.

[0004] In the above invention application, an object detection model is used to identify graphical elements on the GUI, and a deep reinforcement learning model is used to update the status of the mobile application. However, cross-platform GUI testing needs to be performed on different operating systems and devices, and the hardware, software environments, and system configurations between these platforms may vary significantly. Such environmental differences can lead to inconsistent test results and difficult-to-reproduce issues. For example, certain functions may run normally on one platform but may behave abnormally or crash on another.

[0005] To this end, the present invention provides a mobile application GUI intelligent testing method and system. Summary of the Invention

[0006] (1) Technical problems solved

[0007] In response to the shortcomings of the existing technology, the present invention provides a mobile application GUI intelligent testing method and system. The present invention extracts the attribute information of the initial interface element to be tested of the mobile application, calculates the image similarity Txs between the initial interface to be tested and the standard image initial interface, and selects different initial test processing strategies; after receiving the initial element test command, clicks on the initial interface elements to be tested in turn to obtain the second element test interface to be tested, and calculates the image similarity Txs between the second element test interface to be tested and the standard second element test interface. , and calculate the initial element test difference value Cy, and issue a GUI test abnormality warning; calculate the average performance coefficient of mobile applications on different platforms , issuing warnings for GUI test standard revisions, allowing for timely identification of issues with mobile applications across different platforms, such as inconsistent interfaces, misplaced elements, and malfunctioning functions. This helps development teams quickly identify and fix issues, improving application quality and user experience, thus resolving the technical issues discussed in the background art.

[0008] (2) Technical solution

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions: a mobile application GUI intelligent testing method, comprising the following steps:

[0010] Extract the attribute information of the initial interface elements to be tested of the mobile application, based on the total number n of initial interface elements to be tested, the type of each element , element center coordinates and element area ratio , calculate the image similarity Txs between the initial interface to be tested and the initial interface of the standard image, and select different initial test processing strategies;

[0011] After receiving the initial element test command, click on the initial interface elements to be tested in sequence to obtain the second element test interface to be tested, and extract the total number of elements in the second element test interface to be tested , the type of each element , element center coordinates and element area ratio , calculate the image similarity between the second element test interface to be tested and the standard second element test interface , and calculate the initial element test difference value Cy, and issue a GUI test abnormality warning;

[0012] Obtain the image similarity Txs and initial element test difference Cy of mobile applications on different platforms, and calculate the mean performance coefficient of mobile applications on different platforms , issuing an alert for the revision of GUI test standards.

[0013] Furthermore, the total number n of initial interface elements to be tested and the type of each element are obtained. , element center coordinates and element area ratio , calculate the image similarity Txs between the initial interface to be tested and the initial interface of the standard image:

[0014]

[0015] in, i Indicates the order of the initial interface elements to be tested from left to right and from top to bottom. i =1, 2, 3, 4, ..., ，N Indicates the total number of elements in the initial interface of the standard image. Represents the element type of the standard image initial interface, Indicates the center coordinates of the elements in the initial interface of the standard image. Indicates the element area ratio of the initial interface of the standard image.

[0016] Furthermore, the image similarity Txs between the initial interface to be tested and the initial interface of the standard image is obtained, and different initial test processing strategies are selected, specifically:

[0017] When Txs=0, the feedback indicates that the current initial interface to be tested is the same as the standard image initial interface. No action is required and the conventional GUI intelligent test script is used for testing.

[0018] when When the initial interface to be tested is different from the initial interface of the standard image, the initial element test command is sent out.

[0019] Further, obtain the attribute information of the second element test interface to be tested, including the total number of elements in the second element test interface to be tested , the type of each element , element center coordinates and element area ratio , calculate the image similarity between the second element test interface to be tested and the standard second element test interface :

[0020]

[0021] in, jIndicates the order of the second element test interface elements from left to right and from top to bottom, j = 1, 2, 3, 4, ..., ， Indicates the total number of elements in the standard second element test interface after clicking the element numbered i on the initial interface to be tested. Indicates that the number of the initial interface element to be tested is clicked i After the element, the standard second element test interface element number is j The element type, Indicates that the number of the initial interface element to be tested is clicked i After the element, the standard second element test interface element number is J's element center coordinates, Indicates that the number of the initial interface element to be tested is clicked i After the element, the standard second element test interface element number is j The area ratio of the elements.

[0022] Furthermore, the image similarity between all the second element test interfaces to be tested and the standard second element test interface is obtained. , calculate the initial element test difference value Cy:

[0023]

[0024] Furthermore, the initial element test difference value Cy is obtained and different GUI test warning strategies are selected, specifically:

[0025] when =0, the feedback is that the current initial element test is normal, no action is required, and the conventional GUI intelligent test script is used for testing;

[0026] when When the current initial element test is abnormal, it will feedback that the current initial element test is abnormal and issue a GUI test abnormality warning.

[0027] Furthermore, we obtain the image similarity Txs and initial element test difference Cy of mobile applications on different platforms, and calculate the average image similarity of mobile applications on different platforms. and the mean of the initial element test difference , calculate the mean performance coefficient of mobile applications on different platforms :

[0028]

[0029] in, a Indicates the sequential number of different platforms, a =1, 2, 3, 4, ..., ，c Table shows the total number of all platforms.

[0030] Furthermore, when the performance coefficients of mobile applications on different platforms are averaged Exceed When , it indicates that the GUI test standard needs to be revised, and a GUI test standard revision warning is issued.

[0031] A mobile application GUI intelligent testing system, comprising:

[0032] Initial interface analysis module, extracts the attribute information of the initial interface elements to be tested of the mobile application, and classifies the type of each element according to the total number n of initial interface elements to be tested. , element center coordinates and element area ratio , calculate the image similarity Txs between the initial interface to be tested and the initial interface of the standard image, and select different initial test processing strategies;

[0033] GUI test exception analysis module, after receiving the initial element test command, clicks on the initial interface elements to be tested in sequence, obtains the second element test interface to be tested, and extracts the total number of elements in the second element test interface to be tested , the type of each element , element center coordinates and element area ratio , calculate the image similarity between the second element test interface to be tested and the standard second element test interface , and calculate the initial element test difference value Cy, and issue a GUI test abnormality warning;

[0034] GUI test standard analysis module, obtains the image similarity Txs and initial element test difference value Cy of mobile applications on different platforms, and calculates the average performance coefficient of mobile applications on different platforms , issuing an alert for the revision of GUI test standards.

[0035] (3) Beneficial effects

[0036] The present invention provides a mobile application GUI intelligent testing method and system, which has the following beneficial effects:

[0037] 1. Extract the attribute information of the initial interface elements to be tested of the mobile application, and the type of each element according to the total number n of initial interface elements to be tested , element center coordinates and element area ratio , calculate the image similarity Txs between the initial interface to be tested and the initial interface of the standard image, and select different initial test processing strategies. Selecting different test strategies can more specifically discover problems and improve the accuracy and effectiveness of the test.

[0038] 2. After receiving the initial element test command, click on the initial interface elements to be tested in sequence to obtain the second element test interface to be tested, and extract the total number of elements in the second element test interface to be tested , the type of each element , element center coordinates and element area ratio , calculate the image similarity between the second element test interface to be tested and the standard second element test interface , and calculate the initial element test difference value Cy, and issue a GUI test abnormality warning. Through the warning, the test team can quickly locate problems in the GUI, such as layout errors, missing elements or functional abnormalities, and repair them in time.

[0039] 3. Obtain the image similarity Txs and initial element test difference Cy of mobile applications on different platforms, and calculate the average performance coefficient of mobile applications on different platforms , issuing warnings for GUI test standard revisions, allowing timely identification of issues with mobile apps across different platforms, such as inconsistent interfaces, misplaced elements, and abnormal functionality. This helps development teams quickly identify and fix issues, thereby improving app quality and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flowchart of a mobile application GUI intelligent testing method according to the present invention;

[0041] Figure 2 This is a structural diagram of a mobile application GUI intelligent testing system of the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] See also Figure 1 The present invention provides a mobile application GUI intelligent testing method, comprising the following steps:

[0044] Step 1: Extract the attribute information of the initial interface elements to be tested in the mobile application. According to the total number n of initial interface elements to be tested, the type of each element , element center coordinates and element area ratio , calculate the image similarity Txs between the initial interface to be tested and the initial interface of the standard image, and select different initial test processing strategies.

[0045] The step 1 includes the following:

[0046] Step 101: Enter the initial interface of the mobile application to be tested, scale and align the initial interface to be tested with the standard image initial interface, and then extract the attribute information of the elements of the initial interface to be tested of the mobile application through image recognition technology and UI hierarchy analysis, including the total number n of elements in the initial interface to be tested, the type of each element , element center coordinates and element area ratio .

[0047] The interface elements include buttons, text boxes, drop-down menus, etc.

[0048] Step 102: Obtain the total number n of initial interface elements to be tested and the type of each element , element center coordinates and element area ratio , calculate the image similarity Txs between the initial interface to be tested and the initial interface of the standard image:

[0049]

[0050] in, i Indicates the order of the initial interface elements to be tested from left to right and from top to bottom. i =1, 2, 3, 4, ..., ，N Indicates the total number of elements in the initial interface of the standard image. Represents the element type of the standard image initial interface, Indicates the center coordinates of the elements in the initial interface of the standard image. Indicates the element area ratio of the initial interface of the standard image.

[0051] Step 103: Obtain the image similarity Txs between the initial interface to be tested and the initial interface of the standard image, and select different initial test processing strategies, specifically:

[0052] When Txs=0, the feedback shows that the current initial interface to be tested is the same as the standard image initial interface. No action is required. Use the regular GUI intelligent test script for testing and continue monitoring to prevent abnormal situations.

[0053] when When the initial interface to be tested is different from the initial interface of the standard image, the initial element test command is sent out.

[0054] When using, combine the contents in steps 101 to 103:

[0055] Extract the attribute information of the initial interface elements to be tested of the mobile application, based on the total number n of initial interface elements to be tested, the type of each element , element center coordinates and element area ratio , calculate the image similarity Txs between the initial interface to be tested and the initial interface of the standard image, and select different initial test processing strategies. Selecting different test strategies can more specifically discover problems and improve the accuracy and effectiveness of the test.

[0056] Step 2: After receiving the initial element test command, click on the initial interface elements to be tested in sequence to obtain the second element test interface to be tested, and extract the total number of elements in the second element test interface to be tested , the type of each element , element center coordinates and element area ratio , calculate the image similarity between the second element test interface to be tested and the standard second element test interface , and calculate the initial element test difference value Cy, and issue a GUI test abnormality warning to the outside.

[0057] The second step includes the following:

[0058] Step 201: After receiving the initial element test command, click on the initial interface elements to be tested in sequence to obtain the second element test interface to be tested. Based on image recognition technology and UI hierarchy analysis, extract the attribute information of the second element test interface to be tested, including the total number of elements in the second element test interface to be tested. , the type of each element , element center coordinates and element area ratio .

[0059] Step 202: Obtain the attribute information of the second element test interface to be tested, including the total number of elements in the second element test interface to be tested. , the type of each element , element center coordinates and element area ratio , calculate the image similarity between the second element test interface to be tested and the standard second element test interface :

[0060]

[0061] in, j Indicates the order of the second element test interface elements from left to right and from top to bottom, j = 1, 2, 3, 4, ..., ， Indicates the total number of elements in the standard second element test interface after clicking the element numbered i on the initial interface to be tested. Indicates that the number of the initial interface element to be tested is clicked i After the element, the standard second element test interface element number is j The element type, Indicates that the number of the initial interface element to be tested is clicked i After the element, the standard second element test interface element number is J's element center coordinates, Indicates that the number of the initial interface element to be tested is clicked i After the element, the standard second element test interface element number is j The area ratio of the elements.

[0062] Step 203: Obtain the image similarity between all the second element test interfaces to be tested and the standard second element test interface. , calculate the initial element test difference value Cy:

[0063]

[0064] Step 204: Obtain the initial element test difference value Cy and select different GUI test warning strategies, specifically:

[0065] when =0, the feedback indicates that the current initial element test is normal and no action is required. Use the regular GUI smart test script for testing and continue monitoring to prevent any abnormalities.

[0066] when When the current initial element test is abnormal, it will feedback that the current initial element test is abnormal and issue a GUI test abnormality warning.

[0067] When using, combine the contents in steps 201 to 204:

[0068] After receiving the initial element test command, click on the initial interface elements to be tested in sequence to obtain the second element test interface to be tested, and extract the total number of elements in the second element test interface to be tested , the type of each element , element center coordinates and element area ratio , calculate the image similarity between the second element test interface to be tested and the standard second element test interface , and calculate the initial element test difference value Cy, and issue a GUI test abnormality warning. Through the warning, the test team can quickly locate problems in the GUI, such as layout errors, missing elements or functional abnormalities, and repair them in time.

[0069] Step 3: Obtain the image similarity Txs and initial element test difference Cy of mobile applications on different platforms, and calculate the average performance coefficient of mobile applications on different platforms , issuing an alert for the revision of GUI test standards.

[0070] The step three includes the following:

[0071] Step 301: Obtain the image similarity Txs and initial element test difference Cy of mobile applications on different platforms, and calculate the average image similarity of mobile applications on different platforms. and the mean of the initial element test difference , calculate the mean performance coefficient of mobile applications on different platforms :

[0072]

[0073] in, a Indicates the sequential number of different platforms, a =1, 2, 3, 4, ..., ，c Table shows the total number of all platforms.

[0074] Step 302: When the performance coefficients of mobile applications on different platforms are averaged Exceed When , it indicates that the GUI test standard needs to be revised, and a GUI test standard revision warning is issued.

[0075] When using, combine the contents in steps 301 and 302:

[0076] Obtain the image similarity Txs and initial element test difference Cy of mobile applications on different platforms, and calculate the mean performance coefficient of mobile applications on different platforms , issuing warnings for GUI test standard revisions, allowing timely identification of issues with mobile apps across different platforms, such as inconsistent interfaces, misplaced elements, and abnormal functionality. This helps development teams quickly identify and fix issues, thereby improving app quality and user experience.

[0077] See also Figure 2The present invention provides a mobile application GUI intelligent testing system, comprising:

[0078] Initial interface analysis module, extracts the attribute information of the initial interface elements to be tested of the mobile application, and classifies the type of each element according to the total number n of initial interface elements to be tested. , element center coordinates and element area ratio , calculate the image similarity Txs between the initial interface to be tested and the initial interface of the standard image, and select different initial test processing strategies.

[0079] GUI test exception analysis module, after receiving the initial element test command, clicks on the initial interface elements to be tested in sequence, obtains the second element test interface to be tested, and extracts the total number of elements in the second element test interface to be tested , the type of each element , element center coordinates and element area ratio , calculate the image similarity between the second element test interface to be tested and the standard second element test interface , and calculate the initial element test difference value Cy, and issue a GUI test abnormality warning to the outside.

[0080] GUI test standard analysis module, obtains the image similarity Txs and initial element test difference value Cy of mobile applications on different platforms, and calculates the average performance coefficient of mobile applications on different platforms , issuing an alert for the revision of GUI test standards.

[0081] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. Those skilled in the art will appreciate that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution.

[0082] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0083] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A mobile application GUI intelligent testing method, characterized by: The steps include: Extract the attribute information of the initial interface elements to be tested of the mobile application, based on the total number n of initial interface elements to be tested, the type of each element , element center coordinates and element area ratio , calculate the image similarity Txs between the initial interface to be tested and the initial interface of the standard image, and select different initial test processing strategies; Get the total number n of initial interface elements to be tested and the type of each element , element center coordinates and element area ratio , calculate the image similarity Txs between the initial interface to be tested and the initial interface of the standard image: in, i Indicates the order of the initial interface elements to be tested from left to right and from top to bottom. i =1, 2, 3, 4, ..., ，N Indicates the total number of elements in the initial interface of the standard image. Represents the element type of the standard image initial interface, Indicates the center coordinates of the elements in the initial interface of the standard image. Indicates the element area ratio of the initial interface of the standard image; After receiving the initial element test command, click on the initial interface elements to be tested in sequence to obtain the second element test interface to be tested, and extract the total number of elements in the second element test interface to be tested , the type of each element , element center coordinates and element area ratio , calculate the image similarity between the second element test interface to be tested and the standard second element test interface , and calculate the initial element test difference value Cy, and issue a GUI test abnormality warning; Obtain the image similarity Txs and initial element test difference Cy of mobile applications on different platforms, and calculate the mean performance coefficient of mobile applications on different platforms , issued an early warning for the revision of GUI test standards; Obtain the image similarity Txs and initial element test difference Cy of mobile applications on different platforms, and calculate the average image similarity of mobile applications on different platforms and the mean of the initial element test difference , calculate the mean performance coefficient of mobile applications on different platforms : in, a Indicates the sequential number of different platforms, a =1, 2, 3, 4, ..., ，c Table shows the total number of all platforms.

2. A mobile application GUI intelligent testing method according to claim 1, characterized in that: Obtain the image similarity Txs between the initial interface to be tested and the initial interface of the standard image, and select different initial test processing strategies, specifically: When Txs=0, the feedback indicates that the current initial interface to be tested is the same as the standard image initial interface. No action is required and the conventional GUI intelligent test script is used for testing. when When the initial interface to be tested is different from the initial interface of the standard image, the initial element test command is sent out.

3. The mobile application GUI intelligent testing method according to claim 1, characterized in that: Get the attribute information of the second element test interface to be tested, including the total number of elements in the second element test interface to be tested , the type of each element , element center coordinates and element area ratio , calculate the image similarity between the second element test interface to be tested and the standard second element test interface : in, j Indicates the order of the second element test interface elements from left to right and from top to bottom, j = 1, 2, 3, 4, ..., ， Indicates the total number of elements in the standard second element test interface after clicking the element numbered i on the initial interface to be tested. Indicates that the number of the initial interface element to be tested is clicked i After the element, the standard second element test interface element number is j The element type, Indicates that the number of the initial interface element to be tested is clicked i After the element, the standard second element test interface element number is J's element center coordinates, Indicates that the number of the initial interface element to be tested is clicked i After the element, the standard second element test interface element number is j The area ratio of the elements.

4. The mobile application GUI intelligent testing method according to claim 1, characterized in that: Get the image similarity of all the second element test interfaces to be tested and the standard second element test interface , calculate the initial element test difference value Cy:

5. The mobile application GUI intelligent testing method according to claim 1, characterized in that: Get the initial element test difference value Cy and select different GUI test warning strategies, specifically: when =0, the feedback is that the current initial element test is normal, no action is required, and the conventional GUI intelligent test script is used for testing; when When the current initial element test is abnormal, it will feedback that the current initial element test is abnormal and issue a GUI test abnormality warning.

6. A mobile application GUI intelligent testing method according to claim 1, characterized in that: When the performance coefficient of mobile applications on different platforms is averaged Exceed When , it indicates that the GUI test standard needs to be revised, and a GUI test standard revision warning is issued.

7. A mobile application GUI intelligent testing system, configured to implement the method according to any one of claims 1 to 6, characterized in that: include: Initial interface analysis module, extracts the attribute information of the initial interface elements to be tested of the mobile application, and classifies the type of each element according to the total number n of initial interface elements to be tested. , element center coordinates and element area ratio , calculate the image similarity Txs between the initial interface to be tested and the initial interface of the standard image, and select different initial test processing strategies; GUI test exception analysis module, after receiving the initial element test command, clicks on the initial interface elements to be tested in sequence, obtains the second element test interface to be tested, and extracts the total number of elements in the second element test interface to be tested , the type of each element , element center coordinates and element area ratio , calculate the image similarity between the second element test interface to be tested and the standard second element test interface , and calculate the initial element test difference value Cy, and issue a GUI test abnormality warning; GUI test standard analysis module, obtains the image similarity Txs and initial element test difference value Cy of mobile applications on different platforms, and calculates the average performance coefficient of mobile applications on different platforms , issuing an alert for the revision of GUI test standards.

Images