UI interface generation method and device and storage medium

Abstract

The invention discloses a UI interface generation method and device and a storage medium. The method comprises the steps of obtaining a layout file; wherein the layout file comprises a global attribute, a UI element container and a filling rule used for controlling UI element filling; and based on the global attribute in the layout file, the UI element container and a preset UI resource library, generating a UI interface according to a filling rule. According to the method, the UI interfaces with various styles can be automatically generated, the data acquisition cost is greatly reduced, the original data and the corresponding annotation data required in the semantic understanding task of the UI interface can be relatively quickly provided, and the recognition accuracy of the corresponding model is improved.

Description

Technical Field

[0001] The present invention relates to a UI interface generation method, device and storage medium, and belongs to the technical field of UI interface. Background Art

[0002] Semantic understanding of UI interface refers to the process of analyzing and understanding user interface (UI) elements to extract meaningful information and semantics from them in order to better understand and process the user's intentions and needs. In the existing technology, UI interfaces are often recognized through deep learning models, which often require a large amount of raw data and annotated data for training. Raw data is difficult to collect, and if human annotation is used, it also requires a lot of manpower costs. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide a UI interface generation method, device and storage medium to solve the technical problems of difficulty in obtaining original data and high cost of annotating data in the task of semantic understanding of UI interface.

[0004] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0005] Obtain a layout file; wherein the layout file includes global attributes and UI element containers, and filling rules for controlling filling of UI elements;

[0006] Based on the global attributes, UI element containers, and pre-set UI resource libraries in the layout file, a UI interface is generated according to the filling rules.

[0007] Furthermore, the global attributes include: the size of the UI interface and the number of generated UI interfaces.

[0008] Furthermore, the UI element container obtains UI elements from a UI resource library and carries the obtained UI elements to generate a container of the area; wherein the UI element container includes size parameters of the UI elements, and the size parameters are used to adjust the original size of the obtained UI elements to conform to the size parameters.

[0009] Furthermore, the filling rules include:

[0010] When filling a UI element container, the relative position parameters of the UI element container and the existing area;

[0011] When multiple UI element containers are filled at the same time, the relative position parameters of the multiple UI element containers and the existing area, as well as the arrangement method, spacing parameters and quantity parameters between the multiple UI element containers.

[0012] Furthermore, the step of generating the UI interface includes:

[0013] Under the restriction of the global attributes in the layout file, traverse the UI element containers in the layout file;

[0014] Determining a corresponding UI element from the UI resource library according to the UI element container;

[0015] Based on the filling rule and the determined UI elements, corresponding areas are generated in sequence;

[0016] When the traversal is completed, the corresponding UI interface is obtained.

[0017] Furthermore, the UI resource library is provided with atomic UI elements and combined UI elements; wherein the atomic UI elements include text elements, icon elements and picture elements, and the combined UI elements are a combination of any two or more atomic UI elements.

[0018] In a second aspect, the present invention provides a UI interface generation device, comprising:

[0019] An acquisition module, which is used to obtain a layout file; wherein the layout file includes global attributes and UI element containers, and filling rules for controlling the filling of UI elements;

[0020] A generation module generates a UI interface based on the global attributes, UI element container, and a preset UI resource library in the layout file and in accordance with filling rules.

[0021] In a third aspect, the present invention provides an electronic device, including a processor and a storage medium;

[0022] The storage medium is used to store instructions;

[0023] The processor is configured to operate according to the instructions to execute the steps of any of the aforementioned methods.

[0024] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the aforementioned methods.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] The present invention provides a UI interface generation method, device and storage medium, which can automatically generate UI interfaces with diverse styles, greatly reduce the cost of data acquisition, and can relatively quickly provide the original data and labeled data missing in the UI interface semantic understanding task, thereby improving the recognition accuracy of the corresponding model. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of a UI interface generation method provided by an embodiment of the present invention;

[0028] Figure 2 It is a block diagram of a UI interface generating device provided by an embodiment of the present invention;

[0029] Figure 3 It is a block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0031] In the related art, training data is a data set used to train a machine learning model or a deep learning model. These data sets can be a collection of data of types such as images, texts, or videos. These data sets often contain a variety of features and attributes, and the purpose of obtaining training data is to enable the machine learning model or deep learning model to learn the patterns and laws in the data so that it can be more accurately predicted or classified in future prediction or classification tasks. Labeled data refers to the data obtained after the original data in the training data is labeled, often in the form of manual labeling. Labeled data can help the machine learning model or deep learning model better understand the features and attributes in the data, thereby improving the accuracy and reliability of the model.

[0032] Correspondingly, in the semantic understanding of UI interfaces, it is difficult to obtain high-quality original data, because UI images often involve privacy information, so it is difficult to obtain a large amount of original data. Secondly, manual labeling of UI elements is relatively inefficient, requires a lot of human resources, and is prone to errors.

[0033] Example 1

[0034] This embodiment provides a method for generating a UI interface. Figure 1 As shown, the method includes:

[0035] Step S100: obtaining a layout file; wherein the layout file includes global attributes and UI element containers, and filling rules for controlling filling of UI elements.

[0036] In this embodiment, the above-mentioned layout file can be a layout file in various formats, for example, it can be a file in JSON format, or it can be a file in XML format or TXT format. Specifically, in this embodiment, the layout file in JSON format is preferred.

[0037] The above-mentioned global attributes are expressed as relevant attribute parameters for limiting the generated area. For example, it can include the size of the UI interface and the number of UI interfaces. Specifically, when the size of the UI interface is 800*600, it means that for the area obtained when filling the UI elements, whether it is one area or the total area obtained by adding up multiple areas, it cannot exceed the size limit of 800*600. And it needs to be equal to the size limit. The number of UI interfaces is expressed as the number of UI interfaces to be generated. Accordingly, each UI interface can correspond to one size, or multiple interfaces can use the same size.

[0038] In this embodiment, the above-mentioned UI element container is represented as a container for obtaining corresponding UI elements from the UI resource library and carrying the obtained UI elements to generate an area. The above-mentioned UI element library is provided with atomic UI elements and combined UI elements; wherein, the atomic UI elements include text elements, icon elements and picture elements, and the combined UI elements are the combination of any two or more atomic UI elements.

[0039] Specifically, the text element can be a text element in Chinese characters or in other languages, such as English. The text element can be a single character or multiple characters. The icon element represents a graphic whose size is smaller than a certain size limit, such as 64*64. The icon element can be used to represent a certain function, state or concept. The icon element can be used to form more complex UI elements such as buttons, menus and toolbars in UI images.

[0040] Image elements represent images whose size is larger than the above size limits. For combined UI elements, various combinations can be used, including nested combinations, such as text elements nested in icon elements to fit button-shaped UI elements, or multiple icon elements arranged vertically or horizontally to fit toolbar-shaped UI elements.

[0041] It should be noted that, in addition to the text elements that can be directly filled in the above-mentioned UI element container, the icon elements and picture elements fill the specific paths of the corresponding resources in the above-mentioned UI element container. When the specific area is generated, the corresponding icon or picture can be generated according to the filled path. Correspondingly, in the UI resource library, the icon elements and picture elements also correspond to their respective path information. In the above-mentioned UI resource library, the text element "I" with a length of 1 is defined as a single text element with a length of 7. For example, the text element "I am an IT engineer" is also defined as a single text element. Correspondingly, the combination type of UI elements are also defined as single text elements.

[0042] The above UI element container is pre-configured, and can be a placeholder for filling a single UI element in the above UI element container, or a placeholder for filling multiple UI elements, and the placeholder for filling multiple UI elements can be in the form of an array. Specifically, a regular expression can be used to set the placeholder, or a normal string can be used to hold the placeholder. For example, <t>: Indicates that this is used to fill in text elements with unlimited text length; <t5>: Indicates that only text elements with a maximum text length of 5 can be filled here; <e>: Indicates that only English text elements can be filled in here; : Indicates the path used to fill the image element. In specific implementations, other symbols can also be configured to define the placeholder name range.

[0043] Specifically, the above-mentioned UI element container also includes the size parameter of the UI element, and the size parameter is used to adjust the original size of the obtained UI element to meet the size parameter. It should be noted that the generated UI image often has a certain size limit, for example, the size limit can be implemented through the global attribute in the layout file. Therefore, the original size of the text element, icon element and image element obtained from the UI resource library can be adjusted through the size parameter configured in the UI element container to meet the size parameter. For example, the global attribute in a certain layout file limits the size of the UI interface to 800*600. The UI element container in the layout file obtains a 1600*800 image element from the UI resource library. The 1600*800 image element exceeds the size limit specified by the global attribute. Therefore, in this case, the 1600*800 image element can be adjusted based on the size parameter configured in the UI element container to meet the size limit of 800*600.

[0044] Step S200: Generate a UI interface based on the global attributes, UI element container, and pre-set UI resource library in the layout file and in accordance with the filling rules.

[0045] In this embodiment, the step of generating a UI interface may include:

[0046] Step S201: Under the restriction of the global attributes in the above layout file, traverse the UI element containers in the layout file.

[0047] Specifically, under the restriction of the above global properties, for example, the size of the UI interface in the global properties is 800*600, and the number of UI interfaces is 1000, that is, 1000 UI interfaces with a size of exactly 800*600 are generated.

[0048] Based on the above restrictions, each UI element container in the layout file is searched cyclically, and then the corresponding UI element is obtained from the UI resource library according to the UI element container. Specifically, as mentioned above, multiple types of UI elements have been pre-configured in the UI resource library, and each type of UI element can also be configured with multiple UI elements. For example, atomic UI elements, such as various text elements, icon elements and picture elements, correspondingly, text elements can also be further subdivided into types, such as single characters, two characters, more than ten characters, etc. Icon elements and image elements can also be further subdivided into types. For example, they can be further subdivided according to icons and the semantic information expressed by the icons. In addition to atomic UI elements, the UI resource library can also configure composite UI elements, and composite UI elements can also be further subdivided into types.

[0049] As described above, the UI element container may also be configured with multiple types, and the types of the UI element container correspond to the types of the UI elements in the UI resource library.

[0050] Step S202: Determine a corresponding UI element from the UI resource library according to the UI element container.

[0051] During the traversal process, the target UI element set can be determined from the UI resource library according to the type of UI element container. The set can be understood as the subdivided types of the above-mentioned text elements, icon elements, picture elements, and combined UI elements. Then, the UI element corresponding to the UI element container is obtained from the set in a random manner.

[0052] Step S203: Generate corresponding areas in sequence based on the filling rule and the determined UI elements.

[0053] Once the UI element is determined, the corresponding area can be generated. Here, the location information of the area needs to be determined. In the layout file, the corresponding area can be directly generated for the UI element of the first UI element container, because there is no existing area for reference at this time, so it can be generated directly. Then the UI elements of subsequent UI element containers need to be generated sequentially based on the existing areas. Therefore, it is necessary to introduce filling rules to control the sequential generation of subsequent areas. In this embodiment, the rule of directly generating the corresponding area for the UI element of the first UI element container also falls within the scope of the above-mentioned filling rules.

[0054] Step S204: When the traversal is completed, the corresponding UI interface is obtained.

[0055] The existing image generation program can be used to generate the image. For example, in this embodiment, the Slint toolkit can be preferably used to generate the corresponding image. It should be noted that the Slint toolkit is a generation tool that can effectively develop a smooth graphical user interface for any display (embedded device and desktop application).

[0056] In this embodiment, the above-mentioned filling rule may include: when filling a UI element container, the relative position parameter of the UI element container and the existing area.

[0057] Specifically, when a UI element container obtains a corresponding UI element from the above UI resource library, it is necessary to generate a corresponding area at a relative position of an existing area according to the filling rule. The relative position can be any orientation and any distance of the existing area.

[0058] The above-mentioned filling rules may also include: when filling multiple UI element containers at the same time, relative position parameters of the multiple UI element containers and the existing area, and the arrangement mode, interval parameters and quantity parameters between the multiple UI element containers.

[0059] Specifically, when filling multiple UI element containers at the same time, it is necessary not only to consider the relative position relationship between the multiple UI element containers and the existing area, that is, the orientation and distance, but also to consider the arrangement of the multiple elements, for example, horizontal or vertical arrangement, or the form of a nine-square grid, and the corresponding spacing distances between the multiple UI elements. The above number parameter represents the number of UI element containers filled at the same time.

[0060] The above process is a process of creating a UI interface with various styles. In the process of continuously generating areas, corresponding annotation data can also be continuously generated to obtain an annotation file corresponding to the UI interface. The annotation data includes the position information of each UI element in the UI interface, such as the X, Y of the upper left corner, and category information. For text elements, it can also include corresponding text information. Specifically, the position information of the UI element can be determined based on the size parameters and filling rules in the UI element container.

[0061] Therefore, after obtaining the generated UI interface and annotated data, the deep learning model can be trained.

[0062] The method of this embodiment can obtain UI interfaces with various styles, and secondly, can automatically generate corresponding annotation data, so compared with manual annotation, it is more efficient and can also generate diversified UI interfaces. It can solve the technical problems in the prior art that it is difficult to obtain diversified original data in the semantic understanding of UI interfaces and the manual annotation cost is high.

[0063] Furthermore, in the method of this embodiment, the use of UI element containers for filling is more flexible than some text filling methods in the prior art. Specifically, the text filling methods in the prior art generally use a placeholder corresponding to a text filling, such as Python's format, template and other methods, which require knowing the number of each placeholder and the name of the placeholder, and then replacing the placeholder name with the text one by one. For this method, first, you must know the number of placeholders to be filled in the text; secondly, you must know the name of each placeholder; then fill the same content for the same placeholder name, and in order to fill different contents, the number of placeholders, names and filling contents must correspond completely one by one. However, in the case of filling large texts with more placeholders or the text to be filled is often modified, the inconvenience of the format and template methods is reflected. Every time a new text to be filled appears or the number of placeholders / names of the text to be filled is modified, the filling code needs to be modified accordingly. Accordingly, in this implementation method, multiple types of UI element containers can be set, and different types of UI element containers are used to fill different types of UI elements. Compared with the traditional text filling method, the placeholders do not need to correspond one to one, which greatly reduces the placeholders. Secondly, compared with the traditional text filling method, the filling content of this embodiment method can be randomly extracted from the UI resource library, supporting the generation of diversified UI interfaces. Finally, when the filling quantity of a certain type of element needs to be modified, compared with the traditional text filling method, this implementation method only needs the parameters in the layout file, and there is no need to modify the type and filling rules of the already defined UI element container.

[0064] Example 2

[0065] This embodiment provides a UI interface generation device, such as Figure 2 As shown, including:

[0066] An acquisition module, which is used to obtain a layout file; wherein the layout file includes global attributes and UI element containers, and filling rules for controlling the filling of UI elements;

[0067] A generation module generates a UI interface based on the global attributes, UI element container, and a preset UI resource library in the layout file and in accordance with filling rules.

[0068] Example 3

[0069] This embodiment provides an electronic device, such as Figure 3 As shown, it includes a processor and a storage medium;

[0070] The storage medium is used to store instructions;

[0071] The processor is used to operate according to the instructions to execute the steps of the method according to any one of Embodiment 1.

[0072] Example 4

[0073] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of any method described in Embodiment 1 are implemented.

[0074] Embodiment 5:

[0075] The present invention also proposes a hardware system that applies a UI interface generation method in embodiment 1. The method can be applied to the terminal side or the server side.

[0076] The terminal may be an electronic device with network access capability, for example, a desktop computer, a tablet computer, a laptop computer, a smart phone, a digital assistant, a shopping guide terminal, a television, etc.

[0077] The server can be an electronic device with certain computing and processing capabilities. For example, the server can be a server of a distributed system, or a system with multiple processors, memories, network communication modules, etc. operating in coordination. The server can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The server can also be a server cluster formed by several servers. Alternatively, with the development of science and technology, the server can also be a new technical means that can realize the corresponding functions of the implementation method of the specification. For example, it can be a new form of "server" based on quantum computing.

[0078] Embodiment 6:

[0079] The present invention also provides a computer program product comprising instructions, which, when executed by a computer, enables the computer to execute a UI interface generating method in embodiment 1.

[0080] It should be understood that the specific examples herein are only intended to help those skilled in the art to better understand the embodiments of the present specification, rather than to limit the scope of the present invention.

[0081] It can be understood that in the various implementations of this specification, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the implementation methods of this specification.

[0082] It can be understood that the various embodiments described in this specification can be implemented individually or in combination, and the embodiments of this specification are not limited to this.

[0083] Unless otherwise specified, all technical and scientific terms used in the embodiments of this specification have the same meaning as those commonly understood by those skilled in the art of the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the scope of this specification. The term "and / or" used in this specification includes any and all combinations of one or more related listed items. The singular forms of "a", "above", and "the" used in the embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0084] It can be understood that the processor of the embodiment of this specification can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method implementation can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (DigitalSignalProcessor, DSP), an application-specific integrated circuit (ApplicationSpecificIntegratedCircuit, ASIC), a field programmable gate array (FieldProgrammableGateArray, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiment of this specification can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of this specification can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0085] It is understood that the memory in the embodiments of this specification may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (programmableROM, PROM), an erasable programmable read-only memory (erasablePROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0086] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0087] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method implementation methods and will not be repeated here.

[0088] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0089] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0090] In addition, each functional unit in each embodiment of the present specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0091] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0092] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention. < / e> < / t>

Claims

1. A UI interface generation method, characterized in that: include: Obtain a layout file; wherein the layout file includes global attributes and UI element containers, and filling rules for controlling filling of UI elements; Based on the global attributes, UI element containers, and pre-set UI resource libraries in the layout file, a UI interface is generated according to the filling rules.

2. The UI interface generation method according to claim 1, characterized in that: The global attributes include: the size of the UI interface and the number of generated UI interfaces.

3. The UI interface generation method according to claim 2, characterized in that: The UI element container is a container that obtains UI elements from a UI resource library and carries the obtained UI elements to generate an area; wherein the UI element container includes size parameters of the UI elements, and the size parameters are used to adjust the original size of the obtained UI elements to conform to the size parameters.

4. The UI interface generation method according to claim 3, characterized in that: The filling rules include: When filling a UI element container, the relative position parameters of the UI element container and the existing area; When multiple UI element containers are filled at the same time, the relative position parameters of the multiple UI element containers and the existing area, as well as the arrangement method, spacing parameters and quantity parameters between the multiple UI element containers.

5. The UI interface generation method according to claim 4, characterized in that: The step of generating the UI interface includes: Under the restriction of the global attributes in the layout file, traverse the UI element containers in the layout file; Determining a corresponding UI element from the UI resource library according to the UI element container; Based on the filling rule and the determined UI elements, corresponding areas are generated in sequence; When the traversal is completed, the corresponding UI interface is obtained.

6. The UI interface generation method according to claim 1, characterized in that: The UI resource library is provided with atomic UI elements and combined UI elements; wherein the atomic UI elements include text elements, icon elements and picture elements, and the combined UI elements are the combination of any two or more atomic UI elements.

7. A UI interface generation device, characterized in that: include: An acquisition module, which is used to obtain a layout file; wherein the layout file includes global attributes and UI element containers, and filling rules for controlling the filling of UI elements; A generation module generates a UI interface based on the global attributes, UI element container, and a preset UI resource library in the layout file and in accordance with filling rules.

8. An electronic device, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images