Interface generation method based on multiplexing

Abstract

The invention discloses an interface generation method based on multiplexing. The method comprises the following steps: 1) inputting an interface screenshot of a mobile application and a layout information file corresponding to the screenshot as input of the method; 2) extracting interface element features, and designing an abstract data structure representing mobile application interface layout features; 3) constructing a mobile application interface structure model represented by the screenshot and the layout information file; 4) designing an interface element similarity comparison rule, including comparison of component elements and comparison of container elements; 5) designing a plurality of strategies for reconstructing the data structure model of the mobile application interface, and providing a random number-based model conversion method; and step 6) realizing a generation method from the interface model to the picture. According to the method, the user interface of the mobile application can be modeled, one interface model can be converted into more interface models by adjusting the sequence of similar interface elements on the interface, and pictures are reversely generated from the models.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic data generation, and in particular to a multiplexing-based interface generation method. Background Art

[0002] In today's era, mobile applications are developing rapidly, and we have also witnessed the diversity and richness of mobile applications and functions. Mobile applications must not only consider whether they can meet user needs and the perfection of functions, but also the interactivity and friendliness of their user interfaces. Good user experience is also becoming a key factor that developers must consider. How to provide a better user experience has become an important topic in mobile application development, and many mobile application manufacturers are currently paying attention to related technologies.

[0003] User interface (UI) is the key element that mobile applications bring to users. It is one of the most important aspects when developing mobile applications and one of the most important factors that determine application usability and user experience.

[0004] One method of detecting UI is to use machine learning technology for identification and analysis, and training the corresponding machine learning model requires a large amount of training data, that is, a large number of mobile application user interface images. Then obtaining so many mobile application user interface images is an important stage in this work and also a challenge. Therefore, we hope to model the mobile application user interface, and can automatically generate mobile application user interface images through the constructed model to meet the needs of training machine learning models, realize automated UI style inspection, and improve the efficiency of UI defect detection work, thereby playing a positive role in improving the usability and user experience of mobile applications. However, there are relatively few technologies for automatically generating mobile application interface images, and the method proposed in the present invention is an attempt. Unlike most of the current modeling methods, this work only focuses on the UI features of the user interface, and does not consider the interaction between interface elements and users and page jumps. Behaviors such as abstraction, while simplifying the model data of elements unrelated to UI features. Elements related to UI features require more detailed abstraction and processing; in the link of generating images, our expectation is to generate a large number of user interface images, so it is necessary to implement some strategies to increase the diversity of generated images, such as similar component lists can adjust the order on the interface, so that we can construct more similar page models based on a page model and then generate the same amount of images. Summary of the invention

[0005] In view of the technical problem that there are not enough task data sets related to mobile application interface images in the prior art, the present invention provides a reuse-based interface generation method, which can convert an interface at the data model level to obtain more interfaces. The method analyzes interface screenshots and layout information to model the interface, provides ideas and strategies for comparing the similarity of interface elements based on element attributes, and proposes a random number-guided model conversion method to solve the problem of multiple data models. The present invention is implemented through the following technical solutions.

[0006] A method for generating an interface based on reuse, characterized in that it comprises the following steps:

[0007] Step 1) inputting a screenshot of the interface of a mobile application and a layout information file corresponding to the screenshot as input of the method;

[0008] Step 2) extracting interface element features and designing an abstract data structure representing the layout features of the mobile application interface;

[0009] Step 3) constructing a mobile application interface structure model represented by the screenshots and layout information files;

[0010] Step 4) Designing rules for comparing the similarity of interface elements, including comparison of component elements and comparison of container elements;

[0011] Step 5) design multiple strategies for reconstructing the data structure model of the mobile application interface, and propose a model conversion method based on random numbers;

[0012] Step 6) Implement the method of generating images from the interface model.

[0013] The above-mentioned interface generation method based on reuse is characterized in that the layout information file in step 1) should include the control tree structure of the mobile application interface.

[0014] The above-mentioned interface generation method based on reuse is characterized in that the step 2) extracting interface element features and designing an abstract data structure representing the layout features of the mobile application interface specifically includes the following steps:

[0015] Step 21) specifies an abstract data structure representing the layout features of the mobile application interface, that is, the interface model includes the following properties: interface description, denoted as description; the root node of the interface element structure tree, denoted as root; a screenshot of the interface, denoted as screenshot;

[0016] Step 22) specifies the abstract data structure representing the interface element, that is, each node n of the interface element structure tree has the following properties: number, denoted as id(n); type, denoted as type(n); depth in the structure tree, denoted as depth(n); position coordinates, denoted as position(n); size, denoted as size(n); image, denoted as image(n); contains 0 or more child nodes, denoted as children(n).

[0017] The above-mentioned interface generation method based on reuse is characterized in that the step 3) constructing the mobile application interface structure model represented by the screenshot and layout information file specifically includes the following steps:

[0018] Step 31) parsing the XML file through preprocessing, mainly using regular expression matching to obtain the control tree information in the layout information file and storing it in the memory using a tree structure;

[0019] Step 32) Depth-first traverse the control tree information and execute step 33) for each interface element;

[0020] Step 33) constructs the interface element object and writes the number, type, depth, position and size information, wherein the number is the number of rows in the layout information file; then determines the type of the element, if the element type is a component, only executes step 34), if the element type is a container, executes steps 35) to 37);

[0021] Step 34) intercepting the corresponding part of the screenshot according to the position and size information of the element on the interface and storing it in the component element as an image member;

[0022] Step 35) remove all the child images contained in the container, leaving only the border image and storing it in the image member, and initialize the child list;

[0023] Step 36) If the element has only one child and the position and size information of the element and the child are the same, execute step 35) using the child of the element, otherwise execute step 37);

[0024] Step 37) executes step 33) for each child of the element and stores it in children.

[0025] The above-mentioned interface generation method based on reuse is characterized in that the specific rules of the interface element similarity comparison rules designed in step 4) include:

[0026] 41) Component similarity comparison rule 1: whether the two are in the same container, have similar sizes, and are of the same category;

[0027] 42) Component similarity comparison rule 2: whether the two are in the same container and have similar sizes;

[0028] 43) Container similarity comparison rule 1: whether the two are in the same container, have the same size and type, and have similar child distributions, where similar child distributions refer to the same type and similar size in order;

[0029] 44) Container similarity comparison rule 2: whether the two are in the same container, whether the two are of similar size, and whether the child set has the same number of containers and components;

[0030] 45)Container similarity comparison rule three: whether the two objects are in the same container and whether they are similar in size.

[0031] The above-mentioned interface generation method based on reuse is characterized in that the step 5) designs multiple strategies for reconstructing the mobile application interface data structure model, and proposes a model conversion method based on random numbers, which specifically includes the following steps:

[0032] Step 51) searching for a set S of exchangeable element pairs by deeply traversing the element structure tree and using similarity comparison rules;

[0033] Step 52) If the converted number of requirements is greater than 2^len(S)-1, execute step 53), otherwise execute step 54);

[0034] Step 53) If the similarity comparison rule has the largest granularity, execute step 54), otherwise obtain S' through a similarity comparison rule with a larger granularity and expand S with S' and execute step 52);

[0035] Step 54) Generate a random number set R of the required size and in the range of [1, 2^len(S)-1];

[0036] Step 55) Use each random number r in the set R to transform the PM model to generate the interface model pm' by converting the random number r into a binary expression, where the number of bits of r is the same as the number of exchangeable element pairs. If the number of bits of r corresponding to a certain exchangeable element pair is 1, the exchange is performed, otherwise it is not performed to achieve the effect of random exchange. Finally, the required number of converted models can be generated.

[0037] The above-mentioned reuse-based interface generation method is characterized in that the step 6) generates a picture from the interface model by deeply traversing the element structure tree of the interface model, and splicing out the corresponding mobile application interface according to the position information and image information of the element structure tree.

[0038] The present invention adopts the above technical solution and has the following beneficial effects:

[0039] This method uses parsed screenshots and layout information files to build an abstract data structure of the mobile application interface, constructs an interface model by extracting the features of the interface elements, and reconstructs the model based on different element similarity comparison rules. It can realize the generation from one interface to multiple interfaces, solves the problem of lack of existing data sets and generates more realistic pictures. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the framework of the interface generation method based on reuse according to an embodiment of the present invention.

[0041] Figure 2 This is a file schematic diagram of an application interface screenshot and a layout information file according to an embodiment of the present invention.

[0042] Figure 3 It is an abstract data structure representing the layout characteristics of the mobile application interface according to an embodiment of the present invention.

[0043] Figure 4 This is a schematic diagram of the data structure before and after interface modeling in an embodiment of the present invention.

[0044] Figure 5 This is a schematic diagram of the comparison rules of different particle sizes of containers according to an embodiment of the present invention.

[0045] Figure 6 This is a schematic diagram of a container before and after random exchange according to an embodiment of the present invention.

[0046] Figure 7 The following is a comparison diagram of the interface before and after conversion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] The present invention provides a method for generating an interface based on reuse, which converts an interface at the data model level to obtain more interfaces. The method analyzes interface screenshots and layout information to model the interface, provides ideas and strategies for comparing the similarity of interface elements based on element attributes, and proposes a random number-guided model conversion method to solve the problem of multiple data models. The technical problem to be solved is the insufficient number of task data sets related to mobile application interface images. Figure 1 As shown, Figure 1 The schematic diagram of the framework of the interface generation method based on reuse in an embodiment of the present invention is as follows:

[0049] Step 1) inputting a screenshot of the interface of a mobile application and a layout information file corresponding to the screenshot as input of the method;

[0050] Step 2) extracting interface element features and designing an abstract data structure representing the layout features of the mobile application interface;

[0051] Step 3) constructing a mobile application interface structure model represented by the screenshots and layout information files;

[0052] Step 4) Designing rules for comparing the similarity of interface elements, including comparison of component elements and comparison of container elements;

[0053] Step 5) design multiple strategies for reconstructing the data structure model of the mobile application interface, and propose a model conversion method based on random numbers;

[0054] Step 6) Implement the generation method from model to image.

[0055] like Figure 2 As shown, Figure 2 This is a file schematic diagram of the application interface screenshot and layout file of the embodiment of the present invention. The layout information file in step 1) should include the control tree structure of the interface.

[0056] like Figure 3 As shown, Figure 3 For the abstract data structure representing the layout features of the mobile application interface of the embodiment of the present invention, the step 2) extracting the interface element features and designing the abstract data structure representing the layout features of the mobile application interface specifically includes the following steps:

[0057] Step 21) specifies an abstract data structure representing the layout features of the mobile application interface, that is, the interface model includes the following properties: interface description, denoted as description; the root node of the interface element structure tree, denoted as root; a screenshot of the interface, denoted as screenshot;

[0058] Step 22) specifies the abstract data structure representing the interface element, that is, each node n of the interface element structure tree has the following properties: number, denoted as id(n); type, denoted as type(n); depth in the structure tree, denoted as depth(n); position coordinates, denoted as position(n); size, denoted as size(n); image, denoted as image(n); contains 0 or more child nodes, denoted as children(n).

[0059] like Figure 4 As shown, Figure 4 This is a schematic diagram of the data structure before and after the interface modeling of the embodiment of the present invention. The step 3) constructing the mobile application interface structure model specifically includes the following steps:

[0060] Step 31) parsing the XML file through preprocessing, mainly using regular expression matching to obtain the control tree information in the layout information file and storing it in the memory using a tree structure;

[0061] Step 32) Depth-first traverse the control tree information and execute step 33) for each interface element;

[0062] Step 33) constructs the interface element object and writes the number, type, depth, position and size information, wherein the number is the number of rows in the layout information file; then determines the type of the element, if the element type is a component, only executes step 34), if the element type is a container, executes steps 35) to 37);

[0063] Step 34) intercepting the corresponding part of the screenshot according to the position and size information of the element on the interface and storing it in the component element as an image member;

[0064] Step 35) remove all the child images contained in the container, leaving only the border image and storing it in the image member, and initialize the child list;

[0065] Step 36) If the element has only one child and the position and size information of the element and the child are the same, execute step 35) using the child of the element, otherwise execute step 37);

[0066] Step 37) executes step 33) for each child of the element and stores it in children.

[0067] like Figure 5 As shown, Figure 5 This is a schematic diagram of the comparison rules of different granularities of containers in an embodiment of the present invention. The interface element similarity comparison rules in step 4) specifically include:

[0068] 41) Component similarity comparison rule 1: whether the two are in the same container, have similar size and the same category;

[0069] 42) Component similarity comparison rule 2: whether the two are in the same container and have similar sizes;

[0070] 43) Container similarity comparison rule 1: whether the two are in the same container, have the same size and type, and have similar distribution of children (same type and similar size in order);

[0071] 44) Container similarity comparison rule 2: whether the two are in the same container, whether the two are of similar size, and whether the child set has the same number of containers and components;

[0072] 45) Container similarity comparison rule three: whether the two objects are in the same container and whether they are similar in size.

[0073] like Figure 6 As shown, Figure 6 The schematic diagram of the embodiment of the present invention before and after a container is randomly exchanged, the step 5) designs multiple strategies for reconstructing the mobile application interface data structure model and proposes a model conversion method based on random numbers, which specifically includes the following steps:

[0074] Step 51) searching for a set S of exchangeable element pairs by deeply traversing the element structure tree and using similarity comparison rules;

[0075] Step 52) If the converted number of requirements is greater than 2^len(S)-1, execute step 53), otherwise execute step 54);

[0076] Step 53) If the similarity comparison rule has the largest granularity, execute step 54), otherwise obtain S' through a similarity comparison rule with a larger granularity and expand S with S' and execute step 52);

[0077] Step 54) Generate a random number set R of the required size and in the range of [1, 2^len(S)-1];

[0078] Step 55) Use each random number r in R to transform the PM model to generate the interface model pm' by converting the random number r into a binary expression, where the number of bits of r is the same as the number of exchangeable element pairs. If the number of bits of r corresponding to a certain exchangeable element pair is 1, the exchange is performed, otherwise it is not performed to achieve the effect of random exchange; finally, the required number of converted models can be generated.

[0079] like Figure 7 As shown, Figure 7 This is a comparison diagram of the interface before and after the conversion of the embodiment of the present invention. The step 6) deeply traverses the element structure tree of the interface model, and pieces together the corresponding mobile application interface according to the position information and image information of the element structure tree.

[0080] The above is only a preferred embodiment of the present invention, but the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Any simple modification, equivalent changes and modifications made to the above embodiments by any technician familiar with the profession according to the technical essence of the present invention without departing from the principle and spirit of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating an interface based on reuse, characterized in that: The method comprises the following steps: Step 1) inputting a screenshot of the interface of a mobile application and a layout information file corresponding to the screenshot as input of the method; Step 2) extracting interface element features and designing an abstract data structure representing the layout features of the mobile application interface; Step 3) constructing a mobile application interface structure model represented by the screenshots and layout information files; Step 4) Designing rules for comparing the similarity of interface elements, including comparison of component elements and comparison of container elements; Step 5) design multiple strategies for reconstructing the data structure model of the mobile application interface, and propose a model conversion method based on random numbers; Step 6) Implement the method of generating images from the interface model.

2. The interface generation method based on reuse according to claim 1, characterized in that: The layout information file in step 1) should include the control tree structure of the mobile application interface.

3. The interface generation method based on reuse according to claim 1, characterized in that: The step 2) extracting interface element features and designing an abstract data structure representing the mobile application interface layout features specifically includes the following steps: Step 21) specifies an abstract data structure representing the layout features of the mobile application interface, that is, the interface model includes the following properties: interface description, denoted as description; the root node of the interface element structure tree, denoted as root; a screenshot of the interface, denoted as screenshot; Step 22) specifies the abstract data structure representing the interface element, that is, each node n of the interface element structure tree has the following properties: number, denoted as id(n); type, denoted as type(n); depth in the structure tree, denoted as depth(n); position coordinates, denoted as position(n); size, denoted as size(n); image, denoted as image(n); contains 0 or more child nodes, denoted as children(n).

4. The interface generation method based on reuse according to claim 1, characterized in that: The step 3) constructing the mobile application interface structure model represented by the screenshot and layout information file specifically includes the following steps: Step 31) parsing the XML file through preprocessing, mainly using regular expression matching to obtain the control tree information in the layout information file and storing it in the memory using a tree structure; Step 32) Depth-first traverse the control tree information and execute step 33) for each interface element; Step 33) constructs the interface element object and writes the number, type, depth, position and size information, wherein the number is the number of rows in the layout information file; then determines the type of the element, if the element type is a component, only executes step 34), if the element type is a container, executes steps 35) to 37); Step 34) intercepting the corresponding part of the screenshot according to the position and size information of the element on the interface and storing it in the component element as an image member; Step 35) remove all the child images contained in the container, leaving only the border image and storing it in the image member, and initialize the child list; Step 36) If the element has only one child and the position and size information of the element and the child are the same, execute step 35) using the child of the element, otherwise execute step 37); Step 37) executes step 33) for each child of the element and stores it in children.

5. The interface generation method based on reuse according to claim 1, characterized in that: The specific rules of the interface element similarity comparison rules designed in step 4) include: 41) Component similarity comparison rule 1: whether the two are in the same container, have similar sizes, and are of the same category; 42) Component similarity comparison rule 2: whether the two are in the same container and have similar sizes; 43) Container similarity comparison rule 1: whether the two are in the same container, have the same size and type, and have similar child distributions, where similar child distributions refer to the same type and similar size in order; 44) Container similarity comparison rule 2: whether the two are in the same container, whether the two are of similar size, and whether the child set has the same number of containers and components; 45)Container similarity comparison rule three: whether the two objects are in the same container and whether they are similar in size.

6. The interface generation method based on reuse according to claim 1, characterized in that: The step 5) designs a variety of strategies for reconstructing the mobile application interface data structure model and proposes a model conversion method based on random numbers, which specifically includes the following steps: Step 51) searching for a set S of exchangeable element pairs by deeply traversing the element structure tree and using similarity comparison rules; Step 52) If the converted number of requirements is greater than 2^len(S)-1, execute step 53), otherwise execute step 54); Step 53) If the similarity comparison rule has the largest granularity, execute step 54), otherwise obtain S' through a similarity comparison rule with a larger granularity and expand S with S' and execute step 52); Step 54) Generate a random number set R of the required size and in the range of [1, 2^len(S)-1]; Step 55) Use each random number r in the set R to transform the PM model to generate the interface model pm' by converting the random number r into a binary expression, where the number of bits of r is the same as the number of exchangeable element pairs. If the number of bits of r corresponding to a certain exchangeable element pair is 1, the exchange is performed, otherwise it is not performed to achieve the effect of random exchange. Finally, the required number of converted models can be generated.

7. The interface generation method based on reuse according to claim 1, characterized in that: The step 6) generates an image from the interface model by deeply traversing the element structure tree of the interface model, and stitching out the corresponding mobile application interface according to the position information and image information of the element structure tree.

Images