User interface code generation method and apparatus, electronic device, and storage medium

Abstract

The application provides a user interface code generation method and device, electronic equipment and a storage medium, a plurality of target user interface pictures are acquired; for each target user interface picture, a preset training-to-convergence detection and recognition model is used to detect and recognize target elements and interface boundaries in the target user interface picture to obtain target element information and interface boundary information; interface jump relationship information between each target user interface picture is determined according to the target element information and the interface boundary information; and corresponding target code is generated according to the target element information and the interface jump relationship information. The application can automatically generate target element code and jump relationship user interface code according to target user interface pictures, shortens the development cycle of user interface code, and improves the development and generation efficiency of user interface code.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, and particularly relates to a user interface code generation method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the development of the Internet, a large number of Internet products with user interfaces have been generated, such as various electronic device application programs. In order to meet market demand, the iteration development speed of Internet products is getting faster and faster, which brings great challenges to Internet practitioners.

[0003] At present, an algorithm is generally used to identify interface element information in an Internet product user interface diagram, and interface element codes are generated according to the interface element information. Product designers and developers communicate to determine interface jump relationships.

[0004] However, the existing method cannot automatically generate interface jump relationship codes, and the interface jump relationship needs to be determined through communication and explanation, and the interface jump relationship codes need to be manually written, which seriously affects the generation efficiency of user interface codes. SUMMARY

[0005] The present application provides a user interface code generation method, device, electronic equipment and storage medium, which solves the problem that user interface codes cannot be automatically generated, and improves the generation efficiency of user interface codes.

[0006] In a first aspect, the present application provides a user interface code generation method, comprising:

[0007] Obtaining a plurality of target user interface diagrams;

[0008] For each target user interface diagram, a preset detection and recognition model trained to convergence is used to detect and recognize target elements and interface boundaries in the target user interface diagram to obtain target element information and interface boundary information;

[0009] According to the target element information and the interface boundary information, interface jump relationship information between the target user interface diagrams is determined;

[0010] According to the target element information and the interface jump relationship information, corresponding target codes are generated.

[0011] Optionally, the target element information includes target element color information, and the interface boundary information includes interface boundary color information.

[0012] The interface jump relationship information between the target user interface diagrams is determined according to the target element information and the interface boundary information, comprising:

[0013] Determine interface jump relationship information between each target user interface graph according to the target element color information and the interface boundary color information.

[0014] Further, the determining of the interface jump relationship information between each target user interface graph according to the target element color information and the interface boundary color information comprises:

[0015] Matching the target element color information in any one target user interface graph with the interface boundary color information of the interface boundary existing in at least one other target user interface graph;

[0016] If it is determined that the target element color information in a certain target user interface graph matches the interface boundary color information in another certain target user interface graph, it is determined that there is jump relationship information between the two target user interface graphs;

[0017] Obtaining identification information of the two target user interface graphs and identification information of the target element;

[0018] Constructing interface jump relationship information according to the identification information of the two target user interface graphs and the identification information of the target element.

[0019] Optionally, for each target user interface graph, a preset training-to-convergence detection and recognition model is used to detect and recognize the target element and the interface boundary in the target user interface graph to obtain target element information and interface boundary information, comprising:

[0020] Inputting each target interface graph into a preset first training-to-convergence element detection and recognition model, detecting and recognizing the target element in the target user interface graph by the first training-to-convergence element detection and recognition model, and outputting target element information;

[0021] Inputting each target interface graph into a preset second training-to-convergence boundary detection and recognition model, detecting and recognizing the interface boundary in the target user interface graph by the second training-to-convergence boundary detection and recognition model, and outputting interface boundary information.

[0022] Further, the first training-to-convergence element detection and recognition model comprises a first training-to-convergence element detection and recognition sub-model and a second training-to-convergence element detection and recognition sub-model.

[0023] The inputting of each target interface graph into a preset first training-to-convergence element detection and recognition model, the detecting and recognizing of the target element in the target user interface graph by the first training-to-convergence element detection and recognition model, and the outputting of target element information, comprises:

[0024] input each target interface graph into the first trained-to-converge element detection and recognition sub-model, detect and recognize the type and position of the target element in the target user interface graph through the first trained-to-converge element detection and recognition sub-model, and output the type information of the target element and the position information of the target element;

[0025] input the type information of the target element and the position information of the target element into the second trained-to-converge element detection and recognition sub-model, recognize the color of the target element through the second trained-to-converge element detection and recognition sub-model, and output the color information of the target element.

[0026] Optionally, the first trained-to-converge element detection and recognition sub-model is a YOLOv3 model.

[0027] Further, the inputting each target interface graph into the preset second trained-to-converge boundary detection and recognition model, detecting and recognizing the interface boundary in the target user interface graph through the second trained-to-converge boundary detection and recognition model, and outputting the interface boundary information, comprises:

[0028] input each target interface graph into the preset second trained-to-converge boundary detection and recognition model, and detect the interface boundary in the target user interface graph through the second trained-to-converge boundary detection and recognition model;

[0029] If it is determined that the target user interface graph has an interface boundary, the color of the interface boundary is recognized, and the color of the interface boundary is output.

[0030] Further, the target code comprises: target element code and jump relationship code.

[0031] The generating of the corresponding target code according to the target element information and the interface jump relationship information comprises:

[0032] generating target element code according to the type information of the target element and the position information of the target element;

[0033] generating jump relationship code of the corresponding target element and the target user interface jumped to according to the interface jump relationship information.

[0034] Optionally, the obtaining of the plurality of target user interface graphs comprises:

[0035] obtaining a plurality of user-drawn user interface graphs to be processed;

[0036] scaling the user interface graphs to be processed according to the scaling ratio between the user interface graphs to be processed and the target user interface graphs, to obtain a plurality of target user interface graphs.

[0037] In a second aspect, the present application provides a user interface code generation device, comprising:

[0038] an acquisition unit configured to acquire a plurality of target user interface images;

[0039] a detection and recognition unit configured to, for each target user interface image, use a preset detection and recognition model trained to convergence to detect and recognize target elements and interface boundaries in the target user interface image, to obtain target element information and interface boundary information;

[0040] a determination unit configured to determine interface jump relationship information between the target user interface images according to the target element information and the interface boundary information;

[0041] a generation unit configured to generate corresponding target code according to the target element information and the interface jump relationship information.

[0042] Optionally, the determination unit is specifically configured to:

[0043] determine interface jump relationship information between the target user interface images according to the target element color information and the interface boundary color information.

[0044] Further, the determination unit is specifically further configured to:

[0045] match target element color information in any one target user interface image with interface boundary color information of other target user interface images that have interface boundaries;

[0046] if it is determined that the target element color information in a certain target user interface image matches interface boundary color information in another target user interface image, determine that there is jump relationship information between the two target user interface images;

[0047] acquire identification information of the two target user interface images and identification information of the target elements;

[0048] construct interface jump relationship information according to the identification information of the two target user interface images and the identification information of the target elements.

[0049] Optionally, the detection and recognition unit is specifically configured to:

[0050] input each target interface image into a preset first trained-to-convergence element detection and recognition model, detect and recognize target elements in the target user interface image through the first trained-to-convergence element detection and recognition model, and output target element information;

[0051] input each target interface graph into a preset second trained-to-converged boundary detection and recognition model, detect and recognize the interface boundary in the target user interface graph through the second trained-to-converged boundary detection and recognition model, and output interface boundary information.

[0052] Further, the detection and recognition unit is specifically further configured to:

[0053] input each target interface graph into a first trained-to-converged element detection and recognition submodel, detect and recognize the type and position of the target element in the target user interface graph through the first trained-to-converged element detection and recognition submodel, and output type information of the target element and position information of the target element;

[0054] input the type information of the target element and the position information of the target element into a second trained-to-converged element detection and recognition submodel, recognize the color of the target element through the second trained-to-converged element detection and recognition submodel, and output color information of the target element.

[0055] Optionally, the first trained-to-converged element detection and recognition submodel is a YOLOv3 model.

[0056] Further, the detection and recognition unit is specifically further configured to:

[0057] input each target interface graph into a preset second trained-to-converged boundary detection and recognition model, detect and recognize the interface boundary in the target user interface graph through the second trained-to-converged boundary detection and recognition model;

[0058] If it is determined that the target user interface graph contains an interface boundary, the color of the interface boundary is recognized, and the color of the interface boundary is output.

[0059] Optionally, the generation unit is specifically configured to:

[0060] generate a target element code according to the type information of the target element and the position information of the target element;

[0061] generate a jump relationship code of the corresponding target element and the target user interface jumped to according to the interface jump relationship information.

[0062] Optionally, the acquisition unit is specifically configured to:

[0063] acquire a plurality of user-drawn user interface graphs to be processed;

[0064] perform scaling processing on the user interface graphs to be processed according to a scaling ratio between the user interface graphs to be processed and the target user interface graph, to obtain a plurality of target user interface graphs.

[0065] In a third aspect, the present application provides an electronic device, comprising: at least one processor and a memory;

[0066] The memory stores computer-executable instructions.

[0067] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the user interface code generation method according to any one of the first aspect.

[0068] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the user interface code generation method according to the first aspect and various possible designs of the first aspect is implemented.

[0069] In a fifth aspect, the present application provides a computer program product, comprising a computer program, and when a processor executes the computer program, the user interface code generation method according to any one of the first aspect is implemented.

[0070] The user interface code generation method, device, electronic device and storage medium provided by the present application obtain multiple target user interface pictures; for each target user interface picture, a preset detection and recognition model trained to convergence is used to detect and recognize target elements and interface boundaries in the target user interface picture, so as to obtain target element information and interface boundary information; the interface jump relationship information between each target user interface picture is determined according to the target element information and the interface boundary information; and the corresponding target code is generated according to the target element information and the interface jump relationship information. Compared with the prior art, the user interface code generation method provided by the present application first obtains multiple target user interface pictures, detects and recognizes the obtained multiple target user interface pictures to obtain target element information and interface boundary information, then determines interface jump relationship information according to the target element information and the interface boundary information, and finally generates target code by using the obtained target element information and interface jump relationship information, so as to solve the problem that user interface code cannot be automatically generated in whole, shorten the development cycle of user interface code, and improve the generation efficiency of user interface code. BRIEF DESCRIPTION OF DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0072] Figure 1A schematic diagram of a network architecture based on the present disclosure;

[0073] Figure 2 A flowchart of a user interface code generation method provided by an embodiment of the present disclosure;

[0074] Figure 3 A schematic diagram for matching target element color information with interface boundary color information;

[0075] Figure 4 An effect example diagram of a target code;

[0076] Figure 5 A structural schematic diagram of a user interface code generation apparatus provided by an embodiment of the present disclosure;

[0077] Figure 6 A structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0078] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0079] Currently, various Internet products are changing people's way of life, especially Internet products with user interfaces, which bring users intuitive visual experience, and have become an indispensable part of people's life, such as various electronic device application programs. In order to meet the needs of users, the Internet product iteration update speed is getting faster and faster, and how to develop Internet products more efficiently has become a problem that major Internet companies need to solve.

[0080] In the prior art, first, a user interface diagram is drawn according to the product demand of the user, then the element feature information in the user interface diagram is recognized by using an algorithm, and the elements in the interface diagram are described in language based on computer program language to generate language description information, and finally the interface element code is generated according to the element feature information and the corresponding language description information.

[0081] However, the prior art can only generate basic interface element code, and cannot obtain the logical jump relationship between interfaces. After the product manager and the developer communicate and explain to determine the interface jump relationship, the developer develops the interface jump relationship code, and finally generates the Internet product user interface code. This leads to an increase in the development cycle of the user interface code, and seriously affects the generation efficiency of the user interface code.

[0082] To solve the problems, the inventors have found that the user interface jump relationship is mainly realized by triggering a target element on one user interface to jump to another user interface, indicating that the target element and the user interface have a matching relationship. Therefore, the target element information in a user interface diagram can be identified, and the interface boundary information of other user interface diagrams can be identified. The target element information and the interface boundary information can be matched to determine the interface jump relationship information between the user interface diagrams, so as to generate the user interface code according to all the target element information and the interface jump relationship information. Therefore, the target element information and the interface boundary information in the user interface diagram can be identified to determine the interface jump relationship information between the user interface diagrams, so as to generate the corresponding target code according to the target element information and the interface jump relationship information. The target code is the user interface code, which shortens the development cycle of the user interface code and improves the generation efficiency of the user interface code.

[0083] Figure 1 A schematic diagram of a network architecture based on the present disclosure is shown in FIG. 1. The network architecture based on the present disclosure can include a user interface code generation device 1, a shooting device 2, and a display device 3. Figure 1

[0084] The user interface code generation device 1 can be hardware, software, or a combination of software and hardware, which can be used to execute the user interface code generation method described in each of the following embodiments.

[0085] When the user interface code generation device 1 is hardware, it can be an electronic device with computing function. When the user interface code generation device 1 is software, it can be installed in an electronic device with computing function. The electronic device includes but is not limited to servers, notebook computers, desktop computers, and the like.

[0086] The shooting device 2 can be specifically a hardware device such as a smartphone or a smart camera with shooting function, and the display device 3 can be specifically a hardware device such as a display or a projector with display function.

[0087] Specifically, the user interface code generation device 1 can transmit data with the shooting device 2 through a network to obtain a target user interface diagram shot by the shooting device 2. In addition, the user interface code generation device 1 is connected in communication with the display device 3, and displays the generated target code on the display device 3 for viewing and editing by a developer.

[0088] ​For example, a designer uses a smart phone to take 50 target user interface pictures, the user interface code generation device acquires the 50 target user interface pictures, and identifies target element information and interface boundary information in each target user interface picture, generates corresponding target code according to the target element information and the interface boundary information in each target user interface picture, and finally displays the generated target code on a display device for a developer to view and edit.

[0089] The user interface code generation method, device, electronic device and storage medium provided in the present application will be further described below:

[0090] Embodiment one, Figure 2 A flowchart of a user interface code generation method provided by the present embodiment is shown in FIG. 1. Figure 2 As shown in FIG. 1, the execution subject of the present embodiment is a user interface code generation device, and the user interface code generation method provided by the present embodiment includes:

[0091] S21, acquiring a plurality of target user interface pictures.

[0092] The target user interface picture is a user interface picture that needs to be subjected to user interface code generation.

[0093] In the present embodiment, when the plurality of target user interface pictures are acquired, the plurality of target user interface pictures can be acquired from a shooting device, and after the target user interface pictures are preprocessed, the corresponding target user interface pictures are obtained, or the pre-stored target user interface pictures are directly acquired from a pre-set storage area, or the target user interface pictures are acquired by accessing a gallery in which the target user interface pictures are stored, and the present embodiment does not limit this.

[0094] S22, for each target user interface picture, using a pre-set detection and recognition model trained to convergence to detect and recognize target elements and interface boundaries in the target user interface picture to obtain target element information and interface boundary information.

[0095] Since each target user interface picture contains target elements, for a user interface picture that is not a homepage, the interface boundary is also included, therefore, the target elements and the interface boundary in each target user interface picture are detected and recognized to obtain the target element information and the interface boundary information, and then the corresponding target code is generated according to the target element information and the interface boundary information.

[0096] The target element information can include type information of the target element, position information of the target element, and color information of the target element.

[0097] For example, the type of the target element can be a title, a button, an input box, and text, etc. The position information of the target element can be represented by the coordinates of the area occupied by the target element. The color information of the target element can be represented by the border color of the target element or the pixel color of the image area occupied by the target element.

[0098] The interface boundary information can include color information of the interface boundary and identification information of the interface boundary.

[0099] In this embodiment, a detection and recognition model trained to convergence can be preset for the target element and the interface boundary, respectively. The target element in the target user interface image is detected and recognized by using the detection and recognition model trained to convergence corresponding to the target element, so as to obtain target element information. The interface boundary in the target user interface image is detected and recognized by using the detection and recognition model trained to convergence corresponding to the interface boundary, so as to obtain interface boundary information.

[0100] The detection and recognition model trained to convergence corresponding to the target element can be a machine learning model or a deep learning model, which is not limited in this embodiment. Similarly, the detection and recognition model trained to convergence corresponding to the interface boundary can also be a machine learning model or a deep learning model, which is not limited in this embodiment.

[0101] S23, determining interface jump relationship information between the target user interface images according to the target element information and the interface boundary information.

[0102] Since there is a jump relationship between different target user interface images, triggering a target element of a target user interface can jump to another target user interface, and therefore, it is necessary to determine interface jump relationship information between the target user interface images according to the target element information and the interface boundary information.

[0103] The interface jump relationship information is the jump relationship between a target element in an interface and other interfaces.

[0104] In this embodiment, the interface jump relationship information between the target user interface images can be determined according to the target element color information in the target element information and the interface boundary color information in the interface boundary information. Alternatively, the interface jump relationship information between the target user interface images can be determined according to the target element jump mark information in the target element information and the jump mark information in the interface boundary information, and the way of determining the interface jump relationship information between the target user interface images according to the target element information and the interface boundary information is not limited in this embodiment.

[0105] S24, generating corresponding target code according to the target element information and the interface jump relationship information.

[0106] The target code is user interface code, which can include target element code and jump relationship code, and can also include other related code, which is not limited in the embodiment.

[0107] In the embodiment, the target element code corresponding to the target element information can be generated, and the jump relationship code corresponding to the interface jump relationship information can be generated, and then the target element code and the jump relationship code can be spliced according to a preset splicing strategy to generate the final target code.

[0108] The embodiment provides a user interface code generation method, obtains multiple target user interface pictures; for each target user interface picture, a preset detection and recognition model trained to convergence is used to detect and recognize target elements and interface boundaries in the target user interface picture to obtain target element information and interface boundary information; interface jump relationship information between the target user interface pictures is determined according to the target element information and the interface boundary information; and corresponding target code is generated according to the target element information and the interface jump relationship information. The user interface code generation method provided in the embodiment first obtains multiple target user interface pictures, detects and recognizes the obtained multiple target user interface pictures to obtain target element information and interface boundary information, then determines interface jump relationship information according to the target element information and the interface boundary information, and finally generates target code by using the obtained target element information and the interface jump relationship information, thereby solving the problem that user interface code cannot be automatically generated in whole, shortening the development cycle of the user interface code, and improving the generation efficiency of the user interface code.

[0109] Embodiment two, the execution subject of the embodiment is a user interface code generation device, and the user interface code generation method provided in the embodiment is a further description of the step S21 in embodiment one, and the S21 includes:

[0110] S211, obtaining multiple user-drawn user interface pictures to be processed.

[0111] The multiple user-drawn user interface pictures to be processed can be user-drawn user interface pictures or user interface pictures drawn by the user through drawing software, which is not limited in the embodiment.

[0112] S212, scaling the user interface picture to be processed according to a scaling ratio between the user interface picture to be processed and the target user interface picture to obtain multiple target user interface pictures.

[0113] The scaling ratio is set by the user according to the size of the user interface picture to be processed and the target user interface picture, and the purpose is to scale the user interface picture to be processed to the same size as the actual user interface.

[0114] In this embodiment, after obtaining the plurality of user-drawn user interface graphs to be processed, since the size of the user-drawn user interface graph to be processed and the target user interface graph can be different, it is necessary to perform scaling processing on the user-drawn user interface graph to be processed to process the user-drawn user interface graph to be processed into a preset size, and obtain a plurality of target user interface graphs.

[0115] For example, the size of the 50 user-drawn user interface graphs to be processed is 210mm*297mm, and the preset size is 70mm*99mm. The plurality of user-drawn user interface graphs to be processed are reduced by 3 times to obtain 50 target user interface graphs with a size of 70mm*99mm.

[0116] Embodiment three, the execution subject of this embodiment is a user interface code generation device, and the user interface code generation method provided by the present disclosure is a further description of step S22 in embodiment one. S22 includes:

[0117] S221, input each target interface graph into a preset first trained and converged element detection and recognition model, detect and recognize the target elements in the target user interface graph through the first trained and converged element detection and recognition model, and output target element information.

[0118] Among them, the first trained and converged element detection and recognition model further includes a first trained and converged element detection and recognition sub-model and a second trained and converged element detection and recognition sub-model.

[0119] In this embodiment, each target interface graph is input into the first trained and converged element detection and recognition sub-model, the type and position of the target elements in the target user interface graph are detected and recognized through the first trained and converged element detection and recognition sub-model, and the type information of the target elements and the position information of the target elements are output. The type information of the target elements and the position information of the target elements are input into the second trained and converged element detection and recognition sub-model, the color of the target elements is recognized through the second trained and converged element detection and recognition sub-model, and the color information of the target elements is output.

[0120] Specifically, the first training-to-convergence element detection and recognition sub-model is trained, a large number of user interface images are collected, and the types and positions of target elements in the user interface images are labeled to obtain a sample data set. The model is trained using the sample data set until convergence, and a trained model is obtained. The first training-to-convergence element detection and recognition sub-model can be applied to the detection and recognition of the types and positions of target elements in a user interface to obtain type information of the target elements and position information of the target elements. The second training-to-convergence element detection and recognition sub-model is an element color recognition model. When the element color recognition model is trained, a large number of user interface images with different colored target elements are collected, and the colors of the target elements in the user interface images are labeled to obtain a color sample data set. The color recognition model is trained using the color sample data set until convergence, and a trained element color recognition model is obtained. The element color recognition model can be applied to the recognition of the colors of target elements in a user interface to obtain color information of the target elements.

[0121] In example one, the first training-to-convergence element detection and recognition sub-model is a YOLOv3 model. When the YOLOv3 model is used to recognize target elements in 50 target user interface images, each target user interface image has a size of 192 mm x 108 mm. The target elements in the first target user interface image are detected and recognized, and it is determined that the first target user interface image is a login interface containing three types of target elements. The type information of the target elements includes one title (a certain telecommunications company), two input boxes (username and password), and two buttons (login and registration). The position information of the target elements includes title position information (0, 0, 30, 20), username input box position information (76, 50, 116, 58), password input box position information (76, 60, 116, 68), login button position information (90, 70, 112, 78), and registration button position information (90, 80, 112, 88). The first two numbers in the position information represent the upper left corner coordinates, and the last two numbers represent the lower right corner coordinates. The unit of the coordinates is mm. The second training-to-convergence element detection and recognition sub-model is used to recognize the colors of the target elements according to the type information of the target elements and the position information of the target elements, and it is determined that the login button is red and the registration button is yellow, and the other target elements have no color.

[0122] S222, input each target interface image into a preset second training-to-convergence boundary detection and recognition model, detect and recognize the interface boundary in the target user interface image through the second training-to-convergence boundary detection and recognition model, and output interface boundary information.

[0123] The interface boundary information is the color of the interface boundary, and is used for matching the target element and the target user interface diagram.

[0124] In this embodiment, each target interface diagram is input into a preset second trained and converged boundary detection and recognition model, and the interface boundary in the target user interface diagram is detected by the second trained and converged boundary detection and recognition model; if it is determined that the interface boundary exists in the target user interface diagram, the color of the interface boundary is recognized, and the color of the interface boundary is output.

[0125] Specifically, the second trained and converged boundary detection and recognition sub-model is a constructed boundary detection and recognition model. A large number of user interface diagrams are collected, and the positions and colors of the boundaries in the user interface diagrams are labeled to obtain a boundary sample data set. The constructed boundary detection and recognition model is trained by using the boundary sample data set until convergence is achieved, and a trained boundary detection and recognition model is obtained. The boundary detection and recognition model can be applied to the detection and recognition of the boundary positions and colors in the user interface to obtain the color of the interface boundary.

[0126] Example two, based on the foregoing example one, the interface boundaries in 50 target user interface diagrams are detected by the second trained and converged boundary detection and recognition model. After detection, the first target user interface diagram does not have an interface boundary, the second target user interface diagram is a registration interface and has an interface boundary, and the third target user interface diagram is a logged-in interface and has an interface boundary. After the interface boundaries of the second target user interface diagram and the third target user interface diagram are recognized, it is determined that the interface boundaries of the second target user interface diagram and the third target user interface diagram are yellow and red.

[0127] Example four, the execution subject of this embodiment is a user interface code generation apparatus, and the user interface code generation method provided by the present disclosure is a further description of the S23 step in example one. In this embodiment, the target element information includes target element color information, the interface boundary information includes interface boundary color information, and correspondingly, S23 includes:

[0128] S231, determining the interface jump relationship information between the target user interface diagrams according to the target element color information and the interface boundary color information.

[0129] Since the interface jump relationship information between the target user interface diagrams is determined according to the target element color information and the interface boundary color information in the target element information, the interface jump relationship code can be more accurately generated, and therefore the method of matching the target element color information and the interface boundary color information is used to determine the interface jump relationship information between the target user interface diagrams.

[0130] In this embodiment, the color information of the target element in any target user interface diagram is matched with the interface boundary color information of the interface boundary in at least one other target user interface diagram; if it is determined that the color information of the target element in a certain target user interface diagram matches the interface boundary color information in another target user interface diagram, it is determined that there is a jump relationship information between the two target user interface diagrams; the identification information of the two target user interface diagrams and the identification information of the target element are obtained; and the interface jump relationship information is constructed according to the identification information of the two target user interface diagrams and the identification information of the target element.

[0131] Example three, based on the foregoing example one and example two, Figure 3 The matching diagram of the color information of the target element and the interface boundary color information is shown in FIG. 1. Figure 3 As shown, it is determined that the first target user interface diagram is a login interface, and the interface contains a red login button and a yellow registration button, the second target user interface diagram is a registration interface and the interface boundary is yellow, and the third target user interface diagram is a login interface and the interface boundary is red. After matching, it is determined that the red login button in the first target user interface diagram matches the red interface boundary color of the third target user interface diagram, and the yellow registration button in the first target user interface diagram matches the yellow interface boundary color of the second target user interface diagram. Therefore, the interface jump relationship information constructed according to the identification information of the first target user interface diagram, the identification information of the third target user interface diagram and the identification information of the red login button is that the login button in the first target user interface diagram can jump to the third target user interface diagram, and the interface jump relationship information constructed according to the identification information of the first target user interface diagram, the identification information of the second target user interface diagram and the identification information of the yellow registration button is that the registration button in the first target user interface diagram can jump to the second target user interface diagram.

[0132] Example five, the execution subject of this embodiment is a user interface code generation device, and the user interface code generation method provided by the present disclosure is a further description of step S24 in example one. In this embodiment, the target code includes target element code and jump relationship code, and correspondingly, S24 includes:

[0133] S241, generating target element code according to the type information and position information of the target element.

[0134] In this embodiment, the target element code determines the style and position of the target element in the user interface.

[0135] In this embodiment, the target element code can be java code, python code, etc. This embodiment does not limit it.

[0136] S242, generating, according to the interface jump relationship information, jump relationship code of the corresponding target element and a target user interface jumped to.

[0137] The jump relationship code determines the logical relationship between the user interfaces.

[0138] In this embodiment, the jump relationship code can be java code, python code, etc., which is not limited in this embodiment.

[0139] Example four, based on the aforementioned example one, example two and example three, according to the type information of the target element and the position information of the target element of the first target user interface picture, the type information of the target element includes: 1 title (a certain telecom company), 2 input boxes (username and password), 2 buttons (login and registration), the position information of the target element includes: the title position information is (0, 0, 30, 20), the username input box position information is (76, 50, 116, 58), the password input box position information is (76, 60, 116, 68), the login button position information is (90, 70, 112, 78), and the registration button position information is (90, 80, 112, 88). The target element code is generated; then according to the trigger of the login button in the first target user interface picture, the third target user interface picture can be jumped to, and the trigger of the registration button in the first target user interface picture can be jumped to the second target user interface picture, the trigger of the login button jump to the third target user interface picture code and the trigger of the registration button jump to the second target user interface picture code are generated.

[0140] After generating the target code, running the target code can display the user interface of the Internet product, and the user triggers the target element in the user interface to jump between different interfaces, Figure 4 For an effect example of a target code, as Figure 4 shown, the interface homepage is a login interface of a certain telecom company, the user inputs the username and password and clicks the login button, the interface jumps to the main interface of the certain telecom company; the user can also click the registration button, the interface jumps to the user registration interface, and the user can perform user registration operation on the registration interface.

[0141] According to the user interface code generation method of the above embodiment, Figure 5 A structural schematic diagram of a user interface code generation device provided by the embodiment of the present disclosure. For ease of illustration, only the part related to the embodiment of the present disclosure is shown. Referring to Figure 5 , the user interface code generation device 50 includes:

[0142] The acquisition unit 51 is configured to acquire a plurality of target user interface pictures.

[0143] The detection and recognition unit 52 is configured to, for each target user interface image, adopt a preset detection and recognition model trained to convergence, detect and recognize target elements and interface boundaries in the target user interface image, and obtain target element information and interface boundary information.

[0144] The determination unit 53 is configured to determine interface jump relationship information between target user interface images according to the target element information and the interface boundary information.

[0145] The generation unit 54 is configured to generate corresponding target code according to the target element information and the interface jump relationship information.

[0146] Optionally, the determination unit 53 is specifically configured to:

[0147] determine interface jump relationship information between target user interface images according to the target element color information and the interface boundary color information.

[0148] Further, the determination unit 53 is specifically further configured to:

[0149] match target element color information in any target user interface image with interface boundary color information of other target user interface images that have interface boundaries;

[0150] if it is determined that the target element color information in a target user interface image matches interface boundary color information in another target user interface image, it is determined that there is jump relationship information between the two target user interface images;

[0151] obtain identification information of the two target user interface images and identification information of the target element;

[0152] construct interface jump relationship information according to the identification information of the two target user interface images and the identification information of the target element.

[0153] Optionally, the detection and recognition unit 52 is specifically configured to:

[0154] input each target interface image into a preset first trained-to-converged element detection and recognition model, detect and recognize target elements in the target user interface image through the first trained-to-converged element detection and recognition model, and output target element information;

[0155] input each target interface image into a preset second trained-to-converged boundary detection and recognition model, detect and recognize interface boundaries in the target user interface image through the second trained-to-converged boundary detection and recognition model, and output interface boundary information.

[0156] Further, the detection and recognition unit 52 is specifically further configured to:

[0157] input each target interface image into the first trained and converged element detection and recognition sub-model, detect and recognize the type and position of the target element in the target user interface image through the first trained and converged element detection and recognition sub-model, and output the type information of the target element and the position information of the target element;

[0158] input the type information of the target element and the position information of the target element into the second trained and converged element detection and recognition sub-model, recognize the color of the target element through the second trained and converged element detection and recognition sub-model, and output the color information of the target element.

[0159] Optionally, the first trained and converged element detection and recognition sub-model is a YOLOv3 model.

[0160] Further, the detection and recognition unit 52 is specifically further configured to:

[0161] input each target interface image into the preset second trained and converged boundary detection and recognition model, and detect the interface boundary in the target user interface image through the second trained and converged boundary detection and recognition model;

[0162] if it is determined that the target user interface image contains an interface boundary, recognize the color of the interface boundary, and output the color of the interface boundary.

[0163] Optionally, the generation unit 54 is specifically configured to:

[0164] generate a target element code according to the type information of the target element and the position information of the target element;

[0165] generate a jump relationship code of the corresponding target element and the target user interface jumped to according to the interface jump relationship information.

[0166] Optionally, the acquisition unit 51 is specifically configured to:

[0167] acquire a plurality of user-drawn user interface images to be processed;

[0168] perform scaling processing on the user interface images to be processed according to a scaling ratio between the user interface images to be processed and the target user interface images, to obtain a plurality of target user interface images.

[0169] This embodiment provides a user interface code generation device. An acquisition unit acquires multiple target user interface images; a detection and recognition unit, for each target user interface image, uses a preset, trained-to-convergence detection and recognition model to detect and recognize target elements and interface boundaries in the target user interface images to obtain target element information and interface boundary information; a determination unit determines interface transition relationships between the target user interface images based on the target element information and interface boundary information; and a generation unit generates corresponding target code based on the target element information and the interface transition relationship information. The user interface code generation device provided in this embodiment first acquires multiple target user interface images and detects and recognizes them to obtain target element information and interface boundary information. Then, it determines interface transition relationship information based on the target element information and interface boundary information. Finally, it generates target code using the obtained target element information and interface transition relationship information, thereby solving the problem that user interface code cannot be automatically generated in its entirety, shortening the development cycle of user interface code, and improving the generation efficiency of user interface code. Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, such as... Figure 6 As shown, the electronic device 60 in this embodiment may include: a memory 61 and a processor 62.

[0170] The memory 61 is used to store computer programs (such as applications that implement the above-mentioned user interface code generation method, functional modules, etc.), computer instructions, etc.

[0171] The aforementioned computer programs, computer instructions, etc., can be stored in one or more partitions of memory 61. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by processor 62.

[0172] The processor 62 is configured to execute the computer program stored in the memory 61 to implement the various steps in the methods described in the above embodiments.

[0173] For details, please refer to the relevant descriptions in the preceding method embodiments.

[0174] The memory 61 and the processor 62 can be independent structures or integrated structures. When the memory 61 and the processor 62 are independent structures, the memory 61 and the processor 62 can be coupled together via bus 63.

[0175] An electronic device in this embodiment can perform... Figure 2 For the technical solution of the method shown, please refer to the specific implementation process and technical principles. Figure 2 The relevant descriptions of the methods shown will not be repeated here.

[0176] In addition, the embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores computer execution instructions. When at least one processor of a user equipment executes the computer execution instructions, the user equipment executes the various possible methods.

[0177] In one embodiment, the computer readable storage medium includes a computer readable storage medium having program instructions stored in or on the computer readable storage medium. The computer readable storage medium can include the same distribution of the same type of media that is used for program instructions. The computer readable storage medium can be a computer readable storage medium that is non-transitory. In one embodiment, the computer readable storage medium includes a computer readable storage medium that stores programming for use by or in connection with the processor. In the context of this document, a "non-transitory computer readable storage medium" includes any medium that is sufficient to tangibly store programming for use by or in connection with the processor, such that the programming can result in an apparatus that performs at least one operation in response to the programming, where the programming can be present in a computer readable storage medium and / or a memory of the processor at the time of executing the programming. The above types of medium include, for example, portable or fixed storage devices, optical storage devices, magnetic storage devices, memory devices, and any variation or combination thereof that is non-transitory. Examples of non-transitory computer readable storage media that store data include all tangible computer readable storage media. Such media include, but are not limited to, portable or fixed storage devices, optical storage devices, magnetic storage devices, memory devices, and any variation or combination thereof that is non-transitory. Examples of portable mass storage devices include compact discs, digital video discs, Blu-ray discs, flash drives, portable hard drives, and any variation or combination thereof that is non-transitory. Examples of fixed mass storage devices include hard disk drives, solid state drives, and any variation or combination thereof that is non-transitory. Examples of optical storage devices include compact discs, digital video discs, Blu-ray discs, and any variation or combination thereof that is non-transitory. Examples of memory devices include random access memory, dynamic random access memory, static random access memory, flash memory, and any variation or combination thereof that is non-transitory. In one embodiment, the computer readable storage medium is tangible and non-transitory.

[0178] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the foregoing storage medium includes: ROM, RAM, magnetic disc or optical disc and various storage medium that can store program codes.

[0179] In addition, the present application further provides a computer program product, including a computer program, which is executed by a processor to realize the scheme of the human virtual model display method as above.

[0180] The above description is merely preferred embodiments of the present disclosure and a description of the principles of the technology applied. Those skilled in the art should understand that the disclosed range of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the disclosed concept. For example, the technical solutions formed by replacing the above features with the technical features disclosed in the present disclosure (but not limited to) having similar functions.

[0181] Moreover, while operations are depicted in a particular order, this should not be understood as requiring such an order nor imposing a sequential order of executed operations. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, while several specific implementation details are contained in the above discussion, these should not be taken as limiting the scope of the present disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0182] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method for generating user interface code, characterized in that, include: Obtain multiple target user interface images; For each target user interface image, a pre-set detection and recognition model trained to convergence is used to detect and recognize the target elements and interface boundaries in the target user interface image to obtain target element information and interface boundary information. Based on the target element information and interface boundary information, determine the interface transition relationship information between each target user interface diagram; Generate corresponding target code based on the target element information and the interface navigation relationship information; The target element information includes: target element color information; the interface boundary information includes: interface boundary color information; determining the interface transition relationship information between each target user interface diagram based on the target element information and the interface boundary information includes: Match the color information of the target element in any target user interface image with the color information of the interface boundary in at least one other target user interface image that has an interface boundary. If the color information of a target element in a target user interface image is determined to match the color information of the interface boundary in another target user interface image, then it is determined that there is a jump relationship between the two target user interface images. Obtain the identification information of the two target user interface images, as well as the identification information of the target elements; Based on the identification information of the two target user interface diagrams and the identification information of the target elements, construct the interface jump relationship information.

2. The method according to claim 1, characterized in that, For each target user interface image, a pre-set detection and recognition model trained to convergence is used to detect and recognize target elements and interface boundaries in the target user interface image to obtain target element information and interface boundary information, including: Each target interface image is input into a preset first training-to-convergence element detection and recognition model. The target elements in the target user interface image are detected and recognized by the first training-to-convergence element detection and recognition model, and the target element information is output. Each target interface image is input into a preset second training-to-convergence boundary detection and recognition model. The second training-to-convergence boundary detection and recognition model detects and recognizes the interface boundaries in the target user interface image and outputs the interface boundary information.

3. The method according to claim 2, characterized in that, The first element detection and recognition model trained to convergence includes a first element detection and recognition sub-model trained to convergence and a second element detection and recognition sub-model trained to convergence. The step involves inputting each target interface image into a preset first trained and converged element detection and recognition model. The first trained and converged element detection and recognition model then detects and recognizes target elements in the target user interface image and outputs target element information, including: Each target interface image is input into the first element detection and recognition sub-model that has been trained to convergence. The first element detection and recognition sub-model that has been trained to convergence detects and recognizes the type and position of the target elements in the target user interface image and outputs the type information and position information of the target elements. The type information and position information of the target element are input into the second element detection and recognition sub-model that has been trained to convergence. The color of the target element is identified by the second element detection and recognition sub-model that has been trained to convergence, and the color information of the target element is output.

4. The method according to claim 3, characterized in that, The first element detection and recognition sub-model trained to convergence is the YOLOv3 model.

5. The method according to claim 3, characterized in that, The step involves inputting each target interface image into a preset, second, converged boundary detection and recognition model. This model detects and recognizes the interface boundaries in the target user interface images and outputs interface boundary information, including: Each target interface image is input into a preset second training-to-convergence boundary detection and recognition model, and the interface boundaries in the target user interface image are detected by the second training-to-convergence boundary detection and recognition model. If it is determined that there is an interface boundary in the target user interface diagram, the color of the interface boundary is identified and the color of the interface boundary is output.

6. The method according to claim 5, characterized in that, The target code includes: target element code and jump relationship code; The step of generating corresponding target code based on the target element information and the interface navigation relationship information includes: Generate a target element code based on the type information and position information of the target element; Generate the corresponding jump relationship code between the target element and the target user interface based on the interface jump relationship information.

7. The method according to any one of claims 1-6, characterized in that, The acquisition of multiple target user interface images includes: Obtain multiple user interface diagrams drawn by the user and ready for processing; Based on the scaling ratio between the user interface image to be processed and the target user interface image, the user interface image to be processed is scaled to obtain multiple target user interface images.

8. A user interface code generation device, characterized in that, include: The acquisition unit is used to acquire multiple target user interface images; The detection and recognition unit is used to detect and recognize target elements and interface boundaries in each target user interface image using a preset detection and recognition model that has been trained to convergence, so as to obtain target element information and interface boundary information. The target element information includes: target element color information; the interface boundary information includes: interface boundary color information. The determining unit is used to determine the interface transition relationship information between each target user interface diagram based on the target element information and the interface boundary information. The generation unit is used to generate corresponding target code based on the target element information and the interface jump relationship information; The determining unit is further configured to match the color information of the target element in any target user interface image with the color information of the interface boundary in at least one other target user interface image that has an interface boundary. If the color information of a target element in a target user interface image is determined to match the color information of the interface boundary in another target user interface image, then it is determined that there is a jump relationship between the two target user interface images. Obtain the identification information of the two target user interface images, as well as the identification information of the target elements; Based on the identification information of the two target user interface diagrams and the identification information of the target elements, construct the interface jump relationship information.

9. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the user interface code generation method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the user interface code generation method as described in any one of claims 1-7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the user interface code generation method according to any one of claims 1-7.

Images