An interface generation method based on multiplexing

Abstract

The application discloses a kind of based on multiplexing interface generation method, comprising: step 1) input the interface screenshot of a mobile application and the layout information file corresponding to the screenshot as the input of method;Step 2) extract interface element features, design abstract data structure representing mobile application interface layout features;Step 3) build the mobile application interface structure model represented by screenshot and layout information file;Step 4) design interface element similarity comparison rule, including comparison to component element and comparison to container element;Step 5) design a variety of reconstruction mobile application interface data structure model strategy, and propose a kind of model conversion method based on random number;Step 6) realize the generation method from interface model to picture.This method can model mobile application user interface, by adjusting the order of similar interface elements on the interface, can convert an interface model into more interface models, and generate pictures from model reversely.

Description

Technical Field

[0001] This invention relates to the field of automated data generation technology, and in particular to a method for generating interfaces based on reuse. Background Technology

[0002] In today's era, mobile applications are developing rapidly, and we have witnessed the diversity and richness of mobile applications and functions. Mobile applications must not only consider whether they can meet user needs and have complete functionality, but also the interactivity and user-friendliness of their user interface. A good user experience is becoming a key factor that developers urgently need to consider. How to provide a better user experience has become an important topic in mobile application development, and many mobile application vendors are currently focusing on related technologies.

[0003] The user interface (UI) is a key element that provides the most intuitive experience to users of mobile applications. It is one of the most important aspects of developing mobile applications and one of the most important factors that determine application usability and user experience.

[0004] One method for UI inspection is to use machine learning techniques for identification and analysis. Training the corresponding machine learning model requires a large amount of training data, namely, a large number of mobile application user interface images. Obtaining such a large number of mobile application user interface images is a crucial and challenging stage in this work. Therefore, we aim to model mobile application user interfaces and automatically generate mobile application user interface images from the constructed model to meet the needs of training machine learning models, achieving automated UI style checks, improving the efficiency of UI defect detection, and thus playing a positive role in improving the usability and user experience of mobile applications. However, there are currently few technologies for automatically generating mobile application interface images, and the method proposed in this invention is one such attempt. Unlike most current modeling methods, this work only focuses on the UI features of the user interface, without considering the abstraction of user interactions and page transitions. While simplifying the model data of elements unrelated to UI features, elements related to UI features require more detailed abstraction and processing. In the image generation stage, our expectation is to generate a large number of user interface images. Therefore, we need to implement strategies to increase the diversity of generated images. For example, the order of similar component lists on the interface can be adjusted, allowing us to construct more similar page models based on a single page model and then generate an equal number of images. Summary of the Invention

[0005] To address the common technical problem of insufficient datasets for mobile application interface image-related tasks in existing technologies, this invention provides a reuse-based interface generation method. This method can transform an interface at the data model level to obtain more interfaces. It analyzes interface screenshots and layout information to model the interface, provides a strategy for comparing interface element similarity based on element attributes, and proposes a random number-guided model transformation method to solve the problem of excessive data model generation. This invention is achieved through the following technical solutions.

[0006] A reuse-based interface generation method, characterized by the following steps:

[0007] Step 1) Input a screenshot of a mobile application interface and the layout information file corresponding to the screenshot as input to the method;

[0008] Step 2) Extract the features of interface elements and design an abstract data structure to represent the layout features of the mobile application interface;

[0009] Step 3) Construct a mobile application interface structure model represented by the screenshots and layout information files;

[0010] Step 4) Design similarity comparison rules for interface elements, including comparison of component elements and comparison of container elements;

[0011] Step 5) Design various strategies for reconstructing the data structure model of mobile application interfaces, and propose a model transformation method based on random numbers;

[0012] Step 6) Implement the method for generating images from the interface model.

[0013] The above-described interface generation method based on reuse is characterized in that the layout information file in step 1) should include the control tree structure of the mobile application interface.

[0014] The above-described interface generation method based on reuse is characterized in that step 2) extracting interface element features and designing an abstract data structure representing the layout features of a mobile application interface specifically includes the following steps:

[0015] Step 21) Defines the abstract data structure representing the layout features of the mobile application interface, i.e., the interface model includes the following properties: interface description, denoted as description; root node of the interface element structure tree, denoted as root; screenshot of the interface, denoted as screenshot.

[0016] Step 22) Defines the abstract data structure representing the interface elements, that is, each node n in the interface element structure tree has the following properties: number, denoted as id(n); type, denoted as type(n); depth in the structure tree, denoted as depth(n); position coordinates, denoted as position(n); size, denoted as size(n); image, denoted as image(n); contains 0 or more child nodes, denoted as children(n).

[0017] The above-described interface generation method based on reuse is characterized in that step 3) of constructing the mobile application interface structure model represented by the screenshot and layout information files specifically includes the following steps:

[0018] Step 31) The XML file is parsed through preprocessing, mainly using regular expression matching to obtain the control tree information in the layout information file and storing it in memory in a tree structure;

[0019] Step 32) Perform a depth-first traversal of the control tree information and execute step 33) for each UI element;

[0020] Step 33) Construct the interface element object and write its number, type, depth, position and size information, where the number is the line number in the layout information file; then determine the type of the element. If the element type is a component, only step 34) is executed. If the element type is a container, steps 35)-37) are executed.

[0021] Step 34) Extract the corresponding part of the screenshot based on the element's position and size information on the interface and store it as an image member in the component element;

[0022] Step 35) Remove the images of all children contained in the container, leaving only the border images and storing them in the image member; initialize the child list.

[0023] Step 36) If the element has only one child and the position and size information of the element and the child are the same, perform step 35) using the child of the element; otherwise, perform step 37).

[0024] Step 37) Perform step 33) on each child of the element and store it in children.

[0025] The above-described interface generation method based on reuse is characterized in that the specific rules for comparing the similarity of interface elements designed in step 4) include:

[0026] 41) Component similarity comparison rule one: Whether the two are in the same container and are similar in size and category;

[0027] 42) Component similarity comparison rule two: Whether the two are in the same container and are similar in size;

[0028] 43) Container similarity comparison rule one: whether the two are in the same container, and have the same size and type, and similar child distribution, wherein the similar child distribution means that they are of the same type and similar size in sequence;

[0029] 44) Container similarity comparison rule two: whether the two are in the same container, whether the two are similar in size, and whether the child set has the same number of containers and components;

[0030] 45) Container similarity comparison rule three: whether the two are in the same container and whether they are similar in size.

[0031] The above-mentioned reuse-based interface generation method is characterized in that step 5) designs various strategies for reconstructing the data structure model of the mobile application interface, and proposes a model transformation method based on random numbers, specifically including the following steps:

[0032] Step 51) Find the set S of commutative element pairs by depth-first traversing the element structure tree and using similarity comparison rules;

[0033] Step 52) If the number of transformed requirements is greater than 2^len(S)-1, then execute step 53); otherwise execute step 54.

[0034] Step 53) If the similarity comparison rule is already at the largest granularity, proceed to step 54); otherwise, obtain S' using a similarity comparison rule with a larger granularity, expand S with S', and proceed to step 52).

[0035] Step 54) Generate a set R of random numbers of size equal to the required number, in the range [1, 2^len(S)-1].

[0036] Step 55) Transform the PM model using each random number r in the set R to generate the interface model pm'. The method is to convert the random number r into binary representation, where the number of bits in r is the same as the number of commutative element pairs. If the number of bits in r corresponding to a commutative element pair is 1, then a commutation is performed; otherwise, no commutation is performed to achieve the effect of random commutation. Finally, the required number of transformed models can be generated.

[0037] The above-described interface generation method based on reuse is characterized in that step 6) generates an image from the interface model by depth-traversing the element structure tree of the interface model and piecing together the corresponding mobile application interface based on the position information and image information of the element structure tree.

[0038] The present invention adopts the above technical solution and has the following beneficial effects:

[0039] This method utilizes parsed screenshots and layout information files to construct an abstract data structure for mobile application interfaces. It extracts features from interface elements to build an interface model, and then reconstructs the model based on this model using different element similarity comparison rules. This enables the generation of multiple interfaces from a single interface, solving the problem of insufficient existing datasets and producing more realistic images. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the framework of the interface generation method based on reuse in an embodiment of the present invention.

[0041] Figure 2 This is a screenshot of the application interface and a schematic diagram of the layout information file in an embodiment of the present invention.

[0042] Figure 3 This is an abstract data structure representing the layout features of a mobile application interface in an embodiment of the present invention.

[0043] Figure 4 This is a schematic diagram of the data structure before and after interface modeling in an embodiment of the present invention.

[0044] Figure 5 This is a schematic diagram illustrating the comparison rules for different particle sizes of the container in an embodiment of the present invention.

[0045] Figure 6 This is a schematic diagram showing the process of randomly swapping a container according to an embodiment of the present invention.

[0046] Figure 7 This is a comparison diagram of the interface before and after the conversion in an embodiment of the present invention. Detailed Implementation

[0047] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0048] This invention provides a reuse-based interface generation method that transforms an interface at the data model level to obtain more interfaces. This method analyzes interface screenshots and layout information to model the interface, provides an approach and strategy for comparing the similarity of interface elements based on element attributes, and proposes a random number-guided model transformation method to solve the problem of multiple data models. The aim is to address the technical problem of insufficient datasets for mobile application interface image-related tasks. Figure 1 As shown, Figure 1 This is a schematic diagram of the framework of the interface generation method based on reuse according to an embodiment of the present invention. The method of the present invention includes the following steps:

[0049] Step 1) Input a screenshot of a mobile application interface and the layout information file corresponding to the screenshot as input to the method;

[0050] Step 2) Extract the features of interface elements and design an abstract data structure to represent the layout features of the mobile application interface;

[0051] Step 3) Construct a mobile application interface structure model represented by the screenshots and layout information files;

[0052] Step 4) Design similarity comparison rules for interface elements, including comparison of component elements and comparison of container elements;

[0053] Step 5) Design various strategies for reconstructing the data structure model of mobile application interfaces, and propose a model transformation method based on random numbers;

[0054] Step 6) Implement the method for generating images from models.

[0055] like Figure 2 As shown, Figure 2 The following is a screenshot of the application interface and a schematic diagram of the layout file in an embodiment of the present invention. The layout information file in step 1) should include the control tree structure of the interface.

[0056] like Figure 3 As shown, Figure 3 As an embodiment of the present invention, the abstract data structure representing the layout features of a mobile application interface, step 2) of extracting interface element features and designing the abstract data structure representing the layout features of a mobile application interface specifically includes the following steps:

[0057] Step 21) Defines the abstract data structure representing the layout features of the mobile application interface, i.e., the interface model includes the following properties: interface description, denoted as description; root node of the interface element structure tree, denoted as root; screenshot of the interface, denoted as screenshot.

[0058] Step 22) Defines the abstract data structure representing the interface elements, that is, each node n in the interface element structure tree has the following properties: number, denoted as id(n); type, denoted as type(n); depth in the structure tree, denoted as depth(n); position coordinates, denoted as position(n); size, denoted as size(n); image, denoted as image(n); contains 0 or more child nodes, denoted as children(n).

[0059] like Figure 4 As shown, Figure 4 This is a schematic diagram of the data structure before and after interface modeling in an embodiment of the present invention. Step 3) of constructing the mobile application interface structure model specifically includes the following steps:

[0060] Step 31) The XML file is parsed through preprocessing, mainly using regular expression matching to obtain the control tree information in the layout information file and storing it in memory in a tree structure;

[0061] Step 32) Perform a depth-first traversal of the control tree information and execute step 33) for each UI element;

[0062] Step 33) Construct the interface element object and write its number, type, depth, position and size information, where the number is the line number in the layout information file; then determine the type of the element. If the element type is a component, only step 34) is executed. If the element type is a container, then steps 35)-37) are executed.

[0063] Step 34) Extract the corresponding part of the screenshot based on the element's position and size information on the interface and store it as an image member in the component element;

[0064] Step 35) Remove the images of all children contained in the container, leaving only the border images and storing them in the image member; initialize the child list.

[0065] Step 36) If an element has only one child and the position and size information of the element and its child are the same, perform step 35) using the child of the element; otherwise, perform step 37).

[0066] Step 37) Perform step 33) on each child of the element and store it in children.

[0067] like Figure 5 As shown, Figure 5 This is a schematic diagram illustrating the comparison rules for different granularities of containers in an embodiment of the present invention. The interface element similarity comparison rules in step 4) specifically include:

[0068] 41) Rule 1 for component similarity comparison: whether the two components are in the same container and are similar in size and category;

[0069] 42) Rule 2 for component similarity comparison: whether the two components are in the same container and are similar in size;

[0070] 43) Container similarity comparison rule one: whether the two are in the same container, and whether they are the same size and type, and whether their children are similarly distributed (in order of same type and similar size);

[0071] 44) Rule 2 for container similarity comparison: whether the two are in the same container, whether the two are similar in size, and whether the child set has the same number of containers and components;

[0072] 45) Container similarity comparison rule three: whether the two are in the same container and whether they are similar in size.

[0073] like Figure 6 As shown, Figure 6 This is a schematic diagram illustrating the process of randomly swapping a container according to an embodiment of the present invention. Step 5) involves designing various strategies for reconstructing the data structure model of the mobile application interface and proposing a model transformation method based on random numbers, specifically including the following steps:

[0074] Step 51) Find the set S of commutative element pairs by depth-first traversing the element structure tree and using similarity comparison rules;

[0075] Step 52) If the number of requirements after transformation is greater than 2^len(S)-1, proceed to step 53); otherwise, proceed to step 54.

[0076] Step 53) If the similarity comparison rule is already at the largest granularity, then proceed to step 54); otherwise, obtain S' using a similarity comparison rule with a larger granularity and expand S with S' and proceed to step 52).

[0077] Step 54) Generate a set R of random numbers of size equal to the required number, in the range [1, 2^len(S)-1].

[0078] Step 55) Transform the PM model using each random number r in R to generate the interface model pm'. The method is to convert the random number r into a binary expression, where the number of bits in r is the same as the number of commutative element pairs. If the number of bits in r corresponding to a commutative element pair is 1, then a commutation is performed; otherwise, no commutation is performed to achieve the effect of random commutation. Finally, the required number of transformed models can be generated.

[0079] like Figure 7 As shown, Figure 7 The following is a comparison diagram of the interface before and after the conversion in an embodiment of the present invention. In step 6), the corresponding mobile application interface is pieced together by traversing the element structure tree of the interface model through depth and according to the position information and image information of the element structure tree.

[0080] The above description is merely a preferred embodiment of the present invention, but the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention. Any simple modifications, equivalent changes, and alterations made by those skilled in the art to the above embodiments based on the technical essence of the present invention without departing from the principles and spirit of the present invention should be included within the protection scope of the present invention.

Claims

1. A reuse-based interface generation method, characterized in that, The method includes the following steps: Step 1) Input a screenshot of a mobile application interface and a layout information file corresponding to the screenshot as input to the method; the layout information file should include the control tree structure of the mobile application interface; Step 2) Extract the features of the interface elements and design an abstract data structure to represent the layout features of the mobile application interface; the abstract data structure includes: interface description, root node of the interface element structure tree, screenshot of the interface, and also includes: number, type, depth in the structure tree, position coordinates, size, image, and contains 0 or more child nodes. Step 3) Construct a mobile application interface structure model represented by the screenshots and layout information files; Step 4) Design the interface element similarity comparison rules, including the comparison of component elements and the comparison of container elements; the interface element similarity comparison rules specifically include: 41) Component similarity comparison rule one: Whether the two are in the same container and are similar in size and category; 42) Component similarity comparison rule two: Whether the two are in the same container and are similar in size; 43) Container similarity comparison rule one: whether the two are in the same container, and have the same size and type, and similar child distribution, wherein the similar child distribution means that they are of the same type and similar size in sequence; 44) Container similarity comparison rule two: whether the two are in the same container, whether the two are similar in size, and whether the child set has the same number of containers and components; 45) Rule 3 for comparing container similarity: Are the two containers in the same container and are they similar in size? Step 5) Design various strategies for reconstructing the data structure model of mobile application interfaces, and propose a model transformation method based on random numbers; Step 5) specifically includes: Step 51) Find the set S of commutative element pairs by depth-first traversing the element structure tree and using similarity comparison rules; Step 52) If the number of transformed requirements is greater than 2^len(S)-1, then proceed to step 53); otherwise, proceed to step 54. Step 53) If the similarity comparison rule is already at the largest granularity, proceed to step 54); otherwise, obtain S' using a similarity comparison rule with a larger granularity, expand S with S', and proceed to step 52). Step 54) Generate a set R of random numbers of size equal to the required number, in the range [1, 2^len(S)-1]. Step 55) Transform the PM model using each random number r in the set R to generate the interface model pm'. The method is to convert the random number r into binary representation, where the number of bits in r is the same as the number of commutative element pairs. If the number of bits in r corresponding to a commutative element pair is 1, then a commutation is performed; otherwise, no commutation is performed to achieve the effect of random commutation. Finally, the required number of transformed models can be generated. Step 6) Implement the method for generating images from the interface model.

2. The interface generation method based on reuse according to claim 1, characterized in that, Step 2) extracting interface element features and designing an abstract data structure representing the layout features of the mobile application interface specifically includes the following steps: Step 21) Define the abstract data structure representing the layout features of the mobile application interface, that is, the interface model includes the following properties: interface description, denoted as description; root node of the interface element structure tree, denoted as root; screenshot of the interface, denoted as screenshot. Step 22) Define the abstract data structure representing the interface elements, that is, each node n in the interface element structure tree has the following properties: number, denoted as id(n); type, denoted as type(n); depth in the structure tree, denoted as depth(n); position coordinates, denoted as position(n); size, denoted as size(n); image, denoted as image(n); contains 0 or more child nodes, denoted as children(n).

3. The interface generation method based on reuse according to claim 1, characterized in that, Step 3) constructing the mobile application interface structure model represented by the screenshot and layout information file specifically includes the following steps: Step 31) The XML file is parsed through preprocessing, mainly by using regular expression matching to obtain the control tree information in the layout information file and storing it in memory in a tree structure; Step 32) Perform a depth-first traversal of the control tree information, and execute step 33 for each UI element. Step 33) Construct the interface element object and write its number, type, depth, position, and size information, where the number is the line number in the layout information file; then determine the element type. If the element type is a component, only step 34) is executed; if the element type is a container, steps 35) to 37) are executed. Step 34) Extract the corresponding part of the screenshot based on the element's position and size information on the interface and store it as an image member in the component element; Step 35) Remove the images of all children contained in the container, leaving only the border images and storing them in the image member; initialize the child list. Step 36) If the element has only one child and the position and size information of the element and its child are the same, then execute step 35) using the child of the element; otherwise, execute step 37). Step 37) Perform step 33) on each child of the element and store it in children.

4. The interface generation method based on reuse according to claim 1, characterized in that, Step 6) generates an image from the interface model by performing a depth-first traversal of the element structure tree of the interface model and piecing together the corresponding mobile application interface based on the position information of the element structure tree and the image information.

Images