A method and device for merging interface jump path graphs

Abstract

The present invention discloses a method and device for merging an interface jump path graph. The method analyzes the similarity of interface scenes for each interface jump path generated by an automated testing tool for UI application interface testing, merges similar interface scene nodes, and thus merges each interface jump path to generate a merged interface jump path graph, thereby reducing the complexity of subsequent analysis. In the present invention, when analyzing the similarity of interface scenes, a method combining interface screenshot and control layout similarity comparison is adopted. Among them, when calculating the similarity of interface screenshots, feature vectors are extracted through a convolutional neural network, and then the distance between feature vectors is used as the interface screenshot similarity; when calculating the similarity of control layouts, the control layout is converted into a simplified layout text, and then the text similarity between the layout texts is calculated.

Description

Technical Field

[0001] The present invention relates to automated test model merging for UI applications. Background Art

[0002] UI applications are software with interface interactions, including desktop software in various desktop computers and various software or web applications on mobile terminals. Each UI application needs to undergo a complete interface function test before release to ensure that the UI application meets the design expectations. This kind of interface function test of UI applications usually requires testers to complete the work manually. However, with the development of software technology, some automatic testing tools have also been produced to perform automated simulation of manual testing. The test of UI applications by automated testing tools is based on random events, which results in a large number of redundant tests. At the same time, due to the lack of guiding test strategies and test directions, the test scenarios are unevenly distributed, some scenarios are over-tested, and some scenarios are under-tested. In order to enable automated testing tools to perform UI application testing more effectively, reduce test redundancy, and enhance the logic and comprehensiveness of the test, it is necessary to analyze the interface paths tested by automated testing tools. The premise of interface path analysis is that the various paths generated by the UI application test of automated testing tools need to be merged to reduce redundancy. Summary of the invention

[0003] The problem to be solved by the present invention is to merge various paths generated by UI application testing of an automated testing tool.

[0004] To solve the above problems, the solution adopted by the present invention is as follows:

[0005] A method for merging interface jump path graphs according to the present invention comprises the following steps:

[0006] S1: Get the interface scene atlas;

[0007] The interface scene atlas is a collection of interface scene graphs;

[0008] The interface scene graph includes an interface scene information set and interface jump information;

[0009] The interface scene information set is a collection of interface scene information;

[0010] The interface scene information is used to represent the interface when the program executes a certain functional scene, including interface screenshots and control layout information;

[0011] The interface jump information is used to mark the jump relationship between each interface scene information in the interface scene information set;

[0012] S2: According to the specific colors corresponding to the specific controls, the interface screenshots of each of the interface scene information in the interface scene graph are colored according to the corresponding control layout information, so that the interface screenshots are converted into interface coloring graphs, and then each of the obtained interface coloring graphs is input into a trained convolutional neural network for feature recognition to obtain an interface feature vector corresponding to the interface screenshot;

[0013] S3: calculating the interface similarity between the two interface scene information by comparing the interface feature vectors corresponding to the interface screenshots of the two interface scene information and comparing the control layout information of the two interface scene information, including the following steps:

[0014] S31: Calculating the distance between the interface feature vectors corresponding to the two interface screenshots as the screenshot similarity;

[0015] S32: Calculating the layout similarity between the two control layout information;

[0016] S33: Calculating the interface similarity between the two interface scene information by using a weighted average method for the screenshot similarity and the layout similarity;

[0017] S4: taking the interface scene atlas as a whole, merging two interface scene information in the interface scene graph whose interface similarity is higher than a certain threshold, and merging the interface jump information, so as to obtain an overall interface jump relationship graph;

[0018] The interface jump relationship graph uses the interface scene information as nodes and the interface jump information as edges.

[0019] Further, according to the interface jump path map merging method of the present invention, the control layout information is in text format; the step S32 includes the following steps:

[0020] S321: extracting information about control types from the control layout information in text format in sequence to form a simplified control layout text;

[0021] S322: performing text similarity comparison on the control layout texts extracted from the two control layout information, and using the text similarity as the layout similarity.

[0022] A device for merging interface jump path graphs according to the present invention includes the following modules:

[0023] M1, used to: obtain the interface scene atlas;

[0024] The interface scene atlas is a collection of interface scene graphs;

[0025] The interface scene graph includes an interface scene information set and interface jump information;

[0026] The interface scene information set is a collection of interface scene information;

[0027] The interface scene information is used to represent the interface when the program executes a certain functional scene, including interface screenshots and control layout information;

[0028] The interface jump information is used to mark the jump relationship between each interface scene information in the interface scene information set;

[0029] M2, used for: coloring the interface screenshots of each interface scene information in the interface scene graph according to the specific color corresponding to the specific control and the corresponding control layout information, so that the interface screenshots are converted into interface coloring graphs, and then inputting each obtained interface coloring graph into a trained convolutional neural network for feature recognition to obtain an interface feature vector corresponding to the interface screenshot;

[0030] M3, used to calculate the interface similarity between the two interface scene information by comparing the interface feature vectors corresponding to the interface screenshots of the two interface scene information and comparing the control layout information of the two interface scene information, including the following modules:

[0031] M31, used to: calculate the distance between the interface feature vectors corresponding to the two interface screenshots as the screenshot similarity;

[0032] M32, used to: calculate the layout similarity between the two control layout information;

[0033] M33, used to: calculate the interface similarity between two interface scene information by using a weighted average device for the screenshot similarity and layout similarity;

[0034] M4 is used to: treat the interface scene atlas as a whole, merge two interface scene information in the interface scene graph whose interface similarity is higher than a certain threshold, and merge the interface jump information, so as to obtain an overall interface jump relationship graph;

[0035] The interface jump relationship graph uses the interface scene information as nodes and the interface jump information as edges.

[0036] Further, according to the interface jump path map merging device of the present invention, the control layout information is in text format; the module M32 includes the following modules:

[0037] M321, used for: extracting information about control types from control layout information in text format in sequence to form a simplified control layout text;

[0038] M322 is used to compare text similarity of the control layout texts extracted from the two control layout information, and use the text similarity as the layout similarity.

[0039] The technical effects of the present invention are as follows: the present invention merges similar interface scene nodes through similarity analysis of interface scenes, thereby reducing the complexity of subsequent analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is an example of an interface scene atlas of an embodiment of the present invention, wherein dotted lines represent similar nodes.

[0041] Figure 2 It is a combined interface jump path diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention is further described in detail below with reference to the accompanying drawings.

[0043] The method for merging the interface jump path graph of the present invention mainly includes two steps: similarity calculation and node merging. The aforementioned step S1 obtains the interface scene atlas, which means that the input of the present invention is the interface scene atlas. The interface scene atlas is a collection of interface scene graphs. The interface scene graph includes an interface scene information set and an interface jump information. The interface scene information set is a collection of interface scene information. The interface scene information is used to represent the interface when the program executes a certain functional scenario, including interface screenshots and control layout information. It can be seen that the interface scene graph is an interface path jump graph with interface scene information as nodes and interface jump information as edges, and the interface scene atlas is an atlas composed of several interface path jump graphs. Each interface scene graph represents a test path of a UI interface. Obviously, the interface path jump graph is a directed graph.

[0044] In this embodiment, the input of the present invention is automatically generated by an automated testing tool. Of course, those skilled in the art understand that it is not excluded that the input is edited after manual testing. How the interface scene atlas is generated is not the scope of the present invention and does not need to be elaborated.

[0045] The output of the present invention is an overall interface path jump graph formed by merging the various path jump graphs in the atlas. The merging process relies on the similarity comparison of the interface scene information nodes. The similarity calculation step is the aforementioned steps S2 and S3. When the nodes are merged, similar interface scene information nodes are merged, which is the aforementioned step S4. Similar interface scene information is usually represented by a similarity exceeding a certain threshold.

[0046] The objects of similarity calculation are two interface scene information. Since the interface scene information includes interface screenshots and control layout information. Therefore, in the present invention, it is necessary to perform similarity calculation on the interface screenshots in the two interface scene information, and at the same time, it is necessary to perform similarity calculation on the control layout information in the two interface scene information. The similarity values ​​obtained by the two similarity calculations are then calculated in a weighted average manner to obtain the final similarity between the two interface scene information. In the present invention, the similarity between the two interface scene information is referred to as interface similarity. The similarity between the two interface screenshots is referred to as screenshot similarity. The similarity between the two control layout information is referred to as layout similarity. In addition, the control layout information here refers to the layout information of the interface control. The control in the present invention is the abbreviation of the interface control.

[0047] The similarity of two interface screenshots is calculated, and the screenshot similarity is calculated. The present invention adopts the following method:

[0048] S21: First, according to the specific colors corresponding to the specific controls, the interface screenshot is colored according to the corresponding control layout information, so that the interface screenshot is converted into an interface coloring image.

[0049] S22: Input the obtained interface coloring image into the trained convolutional neural network for feature recognition to obtain an interface feature vector corresponding to the interface screenshot.

[0050] S31: Calculate the distance between the interface feature vectors corresponding to the two interface screenshots as the screenshot similarity.

[0051] In step S31, the distance between two interface feature vectors may be the Euclidean distance or the Mahalanobis distance.

[0052] In step S22, the trained convolutional neural network uses the trained VGG16 model. The VGG16 model is familiar to those skilled in the art and will not be described in detail. It should be emphasized that in this embodiment, the VGG16 model uses a ready-made trained VGG16 model, so no additional model training is required. The final output of step S22 is an interface feature vector containing 1000 elements.

[0053] Since the similarity calculation between the interface scene information is a pairwise calculation, this means that if there are a total of N interface scene information in the interface scene graph, each interface scene information needs to perform the similarity calculation of the above steps S21, S22, and S23 N-1 times to calculate the similarity with the other N-1 interface scene information. Obviously, when the similarity is calculated N-1 times, step S21 and step S22 are repeated. To this end, the present invention extracts the above steps S21 and step S22 to form step S2, and according to the specific color corresponding to the specific control, the interface screenshots of each interface scene information in the interface scene graph are colored according to the corresponding control layout information, so that the interface screenshots are converted into interface coloring graphs, and then the obtained interface coloring graphs are input into the trained convolutional neural network for feature recognition to obtain the interface feature vector corresponding to the interface screenshot. Therefore, after step S22, the interface screenshot is converted into an interface feature vector, and thus, the interface scene information is composed of the interface feature vector and the control layout information. When the interface screenshots are subsequently performed for similarity calculation, the similarity between the interface feature vectors can be directly calculated.

[0054] The similarity calculation of the two control layout information is performed to calculate the layout similarity, which is the aforementioned step S32. In this embodiment, the control layout information is the layout of the relevant interface controls extracted by the automated testing tool during the test process, and is expressed in the form of XML text format. It is understood by those skilled in the art that the control layout information can also be expressed in other ways, such as using json text format or HTML text format. However, no matter which format is used, for ease of understanding and manual editing, the control layout information is usually in this text format. The control layout information in text format includes information such as which interface controls there are, the inclusion relationship and position relationship between the interface controls, the type of interface controls, and the style presented by the interface controls. Such control layout information applied to different screens will present different interface effects. In addition, when the control layout information is actually presented, it is related to the content presented by the control. The content presented by the control may be included in the control layout information, or it may not exist in the control layout information. For example, for a button, the text content or icon on the button is usually directly described in the control layout information, while for a text input box, the content in the text input box is dynamically controlled by the UI application and will not be described in the control layout information. Therefore, the control layout information itself cannot be equated with the interface screenshot. The control layout information similarity comparison compares the containment relationship and position relationship between the interface controls and the type of interface controls. Other information, such as the style presented by the interface controls, is not compared. Therefore, the control layout information similarity calculation calculates the similarity presented by the containment relationship and position relationship between the interface controls and the type of interface controls. There are many methods for calculating similarity, such as first converting the control layout information into a control layout tree, and then calculating the similarity by matching the tree structure. In this embodiment, the similarity calculation between the control layout information adopts the following steps:

[0055] S321: extracting information about control types from the control layout information in text format in sequence to form a simplified control layout text;

[0056] S322: performing text similarity comparison on the control layout texts extracted from the two control layout information, and using the text similarity as the layout similarity.

[0057] For example, a simplified control layout information is expressed in XML text format as follows:

[0058]

[0059] <bar>

[0060] <button>< / button> <button>< / button> <button>< / button>

[0061] < / bar>

[0062] <bar>

[0063] <stringedit>< / stringedit> <button>< / button>

[0064] < / bar>

[0065] <view>

[0066] <list>

[0067] <string>< / String> <string>< / String>

[0068] < / string> < / string> < / list>

[0069] <textedit>

[0070] < / textedit>

[0071] < / view>

[0072] <view>

[0073] <string>< / String> <text>< / text>

[0074] < / string> < / view>

[0075]

[0076] In the above control layout information, Frame, Bar, Button, String, Image, StringEdit, View, TextEdit, List, and Text all represent certain control types. In this embodiment, in order to simplify the processing, each control type is represented by a single letter. For example, in the above control layout information, the control types Frame, Bar, Button, String, Image, StringEdit, View, TextEdit, List, and Text are represented by letters F, R, B, S, I, E, V, M, L, and T, respectively. Therefore, the information about the control type is extracted through step S321 to form a simplified control layout text as follows:

[0077] FRBBBIREBVLSSMVSTI

[0078] Then the text similarity between the simplified control layout texts is calculated as layout similarity. There are many methods for calculating text similarity comparison, such as the method for calculating text similarity disclosed in patent document CN 105224518A. This specification will not be repeated.

[0079] After the screenshot similarity and layout similarity are calculated respectively by steps S31 and S32, the interface similarity can be calculated by using the weighted average method. The calculation formula of the interface similarity can be expressed as: S_u=S_g*w_1+S_r*w_2. Where S_u represents the interface similarity, S_g is the screenshot similarity, S_r is the layout similarity, w_1 and w_2 are the weighted weights of the screenshot similarity and layout similarity respectively. In this embodiment, w_1 and w_2 are both 0.5.

[0080] The node merging step, that is, the aforementioned step S4, that is, taking the interface scene atlas as a whole, merging the two interface scene information in the interface scene graph whose interface similarity is higher than a certain threshold, and merging the interface jump information, thereby obtaining an overall interface jump relationship graph. Two interface scene information whose interface similarity is higher than a certain threshold are indicated as two interface scene information being similar, otherwise, the two interface scene information are not similar. Step S4 is to merge two similar interface scene information. In this embodiment, the similarity threshold used to determine whether they are similar is 0.95.

[0081] Figure 1 and Figure 2 This example illustrates the process of node merging. Figure 1An example of an interface scene graph set with three interface scene graphs is given. Among them, the interface scene information set of the first interface scene graph includes three interface scene information marked as A1, A2, and A3, and through its corresponding interface jump information, its corresponding interface jump path is: A1->A2->A3; the interface scene information set of the second interface scene graph includes three interface scene information marked as B1, B2, and B3, and through its corresponding interface jump information, its corresponding interface jump path is: B1->B2->B3; the interface scene information set of the third interface scene graph includes four interface scene information marked as C1, C2, C3, and C4, and through its corresponding interface jump information, its corresponding interface jump path is: C1->C2->C3->C4. Through the calculations in the aforementioned steps S2 and S3, the interface scene information A1 is similar to the interface scene information B1; the interface scene information B2 is similar to the interface scene information C1; the interface scene information A2, the interface scene information B3 and the interface scene information C2 are similar in pairs; the interface scene information A3 is similar to the interface scene information C4. Through step S4, refer to Figure 2 , similar interface scene information A1 and interface scene information B1 are merged into interface scene information A1B1; similar interface scene information B2 and interface scene information C1 are merged into interface scene information B2C1; similar interface scene information A2, interface scene information B3 and interface scene information C2 are merged into interface scene information A2B3C2; similar interface scene information A3 and interface scene information C4 are merged into interface scene information A3C4. When the interface scene information is merged, the corresponding jump information is also merged, so we have Figure 2 The example of a whole interface jump relationship diagram: interface scene information A1B1 may jump to interface scene information B2C1 or interface scene information A2B3C2; interface scene information B2C1 may jump to interface scene information A2B3C2; interface scene information A2B3C2 may jump to interface scene information C3 or interface scene information A3C4; interface scene information C3 may jump to interface scene information A3C4. It is obtained after merging.

[0082] In addition, in this embodiment, after step S4 is merged, the step of calculating the degree centrality, eigenvector centrality, betweenness centrality, and connection centrality of the node is also included. The degree centrality, eigenvector centrality, betweenness centrality, and connection centrality of the node are parameters used to indicate the importance of the node, which belong to the scope of intellectual activity rules and methods, and have nothing to do with the interface path merging process of the present invention. Therefore, it is not the scope discussed in the present invention, and this specification does not need to be elaborated.

Claims

1. A method for merging an interface jump path graph, characterized in that: The steps include: S1: Get the interface scene atlas; The interface scene atlas is a collection of interface scene graphs; The interface scene graph includes an interface scene information set and interface jump information; The interface scene information set is a collection of interface scene information; The interface scene information is used to represent the interface when the program executes a certain functional scene, including interface screenshots and control layout information; The interface jump information is used to mark the jump relationship between each interface scene information in the interface scene information set; S2: According to the specific colors corresponding to the specific controls, the interface screenshots of each of the interface scene information in the interface scene graph are colored according to the corresponding control layout information, so that the interface screenshots are converted into interface coloring graphs, and then each of the obtained interface coloring graphs is input into a trained convolutional neural network for feature recognition to obtain an interface feature vector corresponding to the interface screenshot; S3: calculating the interface similarity between the two interface scene information by comparing the interface feature vectors corresponding to the interface screenshots of the two interface scene information and comparing the control layout information of the two interface scene information, including the following steps: S31: Calculating the distance between the interface feature vectors corresponding to the two interface screenshots as the screenshot similarity; S32: Calculating the layout similarity between the two control layout information; S33: Calculating the interface similarity between the two interface scene information by using a weighted average method for the screenshot similarity and the layout similarity; S4: taking the interface scene atlas as a whole, merging two interface scene information in the interface scene graph whose interface similarity is higher than a certain threshold, and merging the interface jump information, so as to obtain an overall interface jump relationship graph; The interface jump relationship graph uses the interface scene information as nodes and the interface jump information as edges.

2. The method for merging interface jump path graphs according to claim 1, characterized in that: The control layout information is in text format; the step S32 includes the following steps: S321: extracting information about control types from the text-formatted control layout information in sequence to form a simplified control layout text; S322: performing text similarity comparison on the control layout texts extracted from the two control layout information, and using the text similarity as the layout similarity.

3. A device for merging interface jump path graphs, characterized in that: Includes the following modules: M1, used to: obtain the interface scene atlas; The interface scene atlas is a collection of interface scene graphs; The interface scene graph includes an interface scene information set and interface jump information; The interface scene information set is a collection of interface scene information; The interface scene information is used to represent the interface when the program executes a certain functional scene, including interface screenshots and control layout information; The interface jump information is used to mark the jump relationship between each interface scene information in the interface scene information set; M2, used for: coloring the interface screenshots of each interface scene information in the interface scene graph according to the specific color corresponding to the specific control and the corresponding control layout information, so that the interface screenshots are converted into interface coloring graphs, and then inputting each obtained interface coloring graph into a trained convolutional neural network for feature recognition to obtain an interface feature vector corresponding to the interface screenshot; M3, used to calculate the interface similarity between the two interface scene information by comparing the interface feature vectors corresponding to the interface screenshots of the two interface scene information and comparing the control layout information of the two interface scene information, including the following modules: M31, used to: calculate the distance between the interface feature vectors corresponding to the two interface screenshots as the screenshot similarity; M32, used to: calculate the layout similarity between the two control layout information; M33, used to: calculate the interface similarity between two interface scene information by using a weighted average device for the screenshot similarity and layout similarity; M4 is used to: treat the interface scene atlas as a whole, merge two interface scene information in the interface scene graph whose interface similarity is higher than a certain threshold, and merge the interface jump information, so as to obtain an overall interface jump relationship graph; The interface jump relationship graph uses the interface scene information as nodes and the interface jump information as edges.

4. The interface jump path graph merging device according to claim 3, characterized in that: The control layout information is in text format; the module M32 includes the following modules: M321, used for: extracting information about control types from control layout information in text format in sequence to form a simplified control layout text; M322 is used to compare text similarity of the control layout texts extracted from the two control layout information, and use the text similarity as the layout similarity.

Images