Method and device for automatically generating front-end code by image

Abstract

The embodiment of the invention discloses a method and a device for automatically generating a front-end code by an image. In the embodiment of the invention, a target image is obtained, and the target image is any image input by a user; in response to the fact that the target image is an image capable of generating a user interface (UI) prototype graph, at least one sub-layout code and an image layout wireframe graph are generated according to layout detection, annotation information of each element component is generated according to element component detection, the annotation information comprises description information of each element component, and the description information comprises description information of each element component; the description information comprises position information, category information and style information; and inputting the target image, the at least one sub-layout code, the image layout wireframe diagram and the annotation information of the element component into a multi-modal large language model to generate a global code. Through the method, accurate conversion from the image to the code can be realized.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more specifically, to a method and device for automatically generating front-end code from an image. Background Art

[0002] In the development process of websites and applications, first, the product manager and the designer need to communicate many times to determine the design draft. After the design draft is determined, the developer needs to communicate with the product manager many times to edit and generate relatively accurate front-end code based on the above design draft, and finally generate a user interface (UI) prototype based on the front-end code. After generating the UI prototype, it may be necessary to communicate with the designer again, and so on and so forth. After multiple iterations, a UI prototype that meets the needs of the product manager is finally generated. Therefore, the process of generating the UI prototype is complicated and time-consuming, and after multiple people's participation, the generated UI prototype may not be able to accurately restore the above design draft.

[0003] In the prior art, in order to quickly generate UI prototypes from design drafts, methods such as Screen2Code and MasterGo are used to generate UI prototypes. Both Screen2Code and MasterGo can directly convert design drawings or design drafts into HTML / CSS (Cascading Style Sheets) codes. However, the above methods have limited recognition accuracy and are unable to handle complex layouts and restore design details, resulting in poor accuracy of the generated HTML / CSS codes and UI prototypes generated according to the above HTML / CSS codes.

[0004] To sum up, how to achieve accurate conversion from image to code is a problem that needs to be solved at present. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and device for automatically generating front-end code from an image, which uses a multimodal large language model to quickly achieve accurate conversion from image to code.

[0006] In a first aspect, an embodiment of the present invention provides a method for automatically generating front-end code from an image, the method comprising:

[0007] Acquire a target image, wherein the target image is any image input by a user;

[0008] In response to the target image being an image capable of generating a user interface UI prototype, generating at least one sub-layout code and an image layout wireframe according to layout detection, and generating annotation information of each element component according to element component detection, wherein the annotation information includes description information of each element component, and the description information includes position information, category information, and style information;

[0009] The target image, the at least one sub-layout code, the image layout wireframe diagram, and the annotation information of the element components are input into a multimodal large language model to generate a global code.

[0010] Optionally, the method further includes:

[0011] The UI prototype diagram is generated according to the global code.

[0012] Optionally, generating at least one sub-layout code according to the layout detection specifically includes:

[0013] determining at least one sub-layout of the target image according to the layout detection;

[0014] Obtaining a vector diagram of some element components included in the at least one sub-layout;

[0015] The at least one sub-layout and a vector diagram of some element components included in the at least one sub-layout are input into a multimodal large language model to generate at least one sub-layout code.

[0016] Optionally, generating at least one sub-layout code according to the layout detection specifically includes:

[0017] determining at least one sub-layout of the target image according to the layout detection;

[0018] The at least one sub-layout is input into the multimodal large language model to generate at least one sub-layout code.

[0019] Optionally, the generating the annotation information of each element component according to the element component detection specifically includes:

[0020] Generating description information of each element component according to the element component detection;

[0021] The description information of each element component is stored in the material library cache;

[0022] The description information of each element component is obtained in the material library cache, and the annotation information of each element component is generated according to the set rules, wherein the annotation information is the description information expressed in natural language.

[0023] Optionally, the method further includes:

[0024] An image element layout wireframe diagram is generated according to the element component detection.

[0025] Optionally, the step of inputting the target image, the at least one sub-layout code, the image layout wireframe diagram, and the annotation information of the element components into a multimodal large language model to generate a global code specifically includes:

[0026] The target image, the at least one sub-layout code, the image layout wireframe diagram, the annotation information of the element components, and the image element layout wireframe diagram are input into the multimodal large language model to generate a global code.

[0027] Optionally, the method further includes:

[0028] Get user feedback annotations;

[0029] The user feedback annotation is input into the multimodal large language model to update the global code.

[0030] Optionally, the method further includes:

[0031] Determine all element components in the target image according to the element component detection;

[0032] Background removal and vectorization are performed on some element components that can be vectorized among all the element components to generate vector graphics of the some element components.

[0033] Optionally, the method further includes:

[0034] The global code is optimized to generate an optimized global code.

[0035] In a second aspect, an embodiment of the present invention provides a device for automatically generating front-end code from an image, the device comprising:

[0036] An acquisition unit, used for acquiring a target image, wherein the target image is any image input by a user;

[0037] A processing unit, in response to the target image being an image capable of generating a user interface UI prototype image, is used to generate at least one sub-layout code and an image layout wireframe according to layout detection, and to generate annotation information of each element component according to element component detection, wherein the annotation information includes description information of each element component, and the description information includes position information, category information, and style information;

[0038] A generating unit is used to input the target image, the at least one sub-layout code, the image layout wireframe diagram and the annotation information of the element components into a multimodal large language model to generate a global code.

[0039] Optionally, the generating unit is further used for:

[0040] The UI prototype diagram is generated according to the global code.

[0041] Optionally, the processing unit is specifically used for:

[0042] determining at least one sub-layout of the target image according to the layout detection;

[0043] Obtaining a vector diagram of some element components included in the at least one sub-layout;

[0044] The at least one sub-layout and a vector diagram of some element components included in the at least one sub-layout are input into the multimodal large language model to generate at least one sub-layout code.

[0045] Optionally, the processing unit is specifically used for:

[0046] determining at least one sub-layout of the target image according to the layout detection;

[0047] The at least one sub-layout is input into the multimodal large language model to generate at least one sub-layout code.

[0048] Optionally, the processing unit is specifically used for:

[0049] Generating description information of each element component according to the element component detection;

[0050] The description information of each element component is stored in the material library cache;

[0051] The description information of each element component is obtained in the material library cache, and the annotation information of each element component is generated according to the set rules, wherein the annotation information is the description information expressed in natural language.

[0052] Optionally, the processing unit is further used for:

[0053] An image element layout wireframe diagram is generated according to the element component detection.

[0054] Optionally, the generating unit is specifically used for:

[0055] The target image, the at least one sub-layout code, the image layout wireframe diagram, the annotation information of the element components, and the image element layout wireframe diagram are input into a multimodal large language model to generate a global code.

[0056] Optionally, the device further includes an updating unit, configured to obtain user feedback annotations; input the user feedback annotations into the multimodal large language model, and update the global code.

[0057] Optionally, the processing unit is further used for:

[0058] Determine all element components in the target image according to the element component detection;

[0059] Background removal and vectorization are performed on some element components that can be vectorized among all the element components to generate vector graphics of the some element components.

[0060] Optionally, the device further includes an optimization unit, configured to optimize the global code and generate an optimized global code.

[0061] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement a method as described in the first aspect or any possible embodiment of the first aspect.

[0062] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement a method as described in the first aspect or any possible embodiment of the first aspect.

[0063] In an embodiment of the present invention, a target image is obtained, wherein the target image is any image input by a user; in response to the target image being an image that can generate a user interface UI prototype, at least one sub-layout code and an image layout wireframe are generated according to layout detection, and annotation information of each element component is generated according to element component detection, wherein the annotation information includes description information of each element component, and the description information includes location information, category information, and style information; the target image, the at least one sub-layout code, the image layout wireframe, and the annotation information of the element component are input into a multimodal large language model to generate a global code. Through the above method, the conversion from image to code can be achieved through a multimodal large language model, with fast generation speed and high conversion accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0065] Figure 1 This is a flow chart of a method for automatically generating front-end code from an image in an embodiment of the present invention;

[0066] Figure 2 is a schematic diagram of an image layout in an embodiment of the present invention;

[0067] Figure 3 It is a flow chart of another method for automatically generating front-end code from an image in an embodiment of the present invention;

[0068] Figure 4 is a flow chart of another method for automatically generating front-end code from an image in an embodiment of the present invention;

[0069] Figure 5 It is a flow chart of another method for automatically generating front-end code from an image in an embodiment of the present invention;

[0070] Figure 6 is a schematic diagram of a wireframe diagram of an image element layout in an embodiment of the present invention;

[0071] Figure 7 is a flow chart of another method for automatically generating front-end code from an image in an embodiment of the present invention;

[0072] Figure 8 It is a flow chart of another method for automatically generating front-end code from an image in an embodiment of the present invention;

[0073] Fig. 9 It is a flow chart of another method for automatically generating front-end code from an image in an embodiment of the present invention;

[0074] Fig.10 It is a schematic diagram of converting a target image into a UI prototype image in an embodiment of the present invention;

[0075] Fig.11 Schematic diagram of a device for automatically generating front-end code from an image in an embodiment of the present invention;

[0076] Fig.12 is a schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for those skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, flows, components and circuits are not described in detail.

[0078] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0079] Unless the context clearly requires otherwise, the words "include", "comprising" and similar words throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, the meaning is "including but not limited to".

[0080] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0081] In the prior art, in order to quickly generate UI prototypes from design drafts, methods such as Screen2Code and MasterGo are used. Specifically, Screen2Code is an open source warehouse. Users need to download the required software packages from the above open source warehouse, generate an application, and then upload the design drawings or screenshots to the application to directly generate clean codes, such as HTML codes, Cascading Style Sheets (CCS) codes, etc.; however, the above methods have limited recognition accuracy, insufficient processing capabilities for complex layouts, and difficulty in restoring design details and interactive effects. They also do not support the recognition and reuse of element components, and cannot restore the materials (such as pictures, icons or charts, etc.) on the original page (such as design drawings or screenshots) input, resulting in poor accuracy of the generated HTML / CSS code and the UI prototype generated according to the above HTML / CSS code; the common pix2code and sketch2code tools also have the same problems as Screen2Code. The MasterGo is mainly used to generate design drafts, and then convert the design drafts into HTML codes, CCS codes, etc. through the plug-ins in the MasterGo. Since the design drafts carry some design information, directly converting the design drafts into codes is more accurate than converting images into codes. However, MasterGo can only generate design drafts according to simple and fixed templates of a set number, and cannot handle complex layouts and restore design details, resulting in poor accuracy of the generated HTML / CSS codes and UI prototypes generated according to the above HTML / CSS codes. Therefore, how to improve the conversion accuracy and speed from images to codes is a problem that needs to be solved at present.

[0082] In order to solve the above problems, an embodiment of the present invention proposes a method for automatically generating front-end code from an image, as follows: Figure 1 As shown, the method includes:

[0083] Step S101: Acquire a target image.

[0084] Specifically, the target image is any image input by a user.

[0085] In an embodiment of the present invention, the target image may be a user interface (UI) design drawing, or a screenshot, etc. The format of the target image may be PNG format, JPEG format, or SVG format, or may be other common design file formats, which may be determined based on actual conditions.

[0086] Step S102: in response to the target image being an image capable of generating a user interface UI prototype, generating at least one sub-layout code and an image layout wireframe according to layout detection, and generating annotation information of each element component according to element component detection.

[0087] Specifically, the annotation information includes description information of each element component, and the description information includes position information, category information and style information; wherein the position information is the bounding box position of the element component, the category information is the category label, the style information includes the color and shape of the label and button, etc., and the description information also includes a component list of confidence scores, which is only for exemplary description here.

[0088] In an embodiment of the present invention, before generating a front-end code based on a target image, it is necessary to first inspect the target image through a multimodal large language model to determine whether the target image is an image that can generate a UI prototype image. If so, the layout detection and the element component detection are performed; if not, the processing is terminated.

[0089] In a possible implementation, the front-end code may also be referred to as code, global code, etc., that is, code used to generate a UI prototype image. The UI prototype image may also be referred to as a front-end page. The UI prototype image is provided to developers for display or research and development, and cannot be directly provided to users for interactive use.

[0090] In an embodiment of the present invention, the generating of at least one sub-layout code according to the layout detection specifically includes: determining at least one sub-layout of the target image according to the layout detection; obtaining a vector diagram of some element components included in the at least one sub-layout; inputting the at least one sub-layout and the vector diagram of some element components included in the at least one sub-layout into a multimodal large language model to generate at least one sub-layout code; or, determining at least one sub-layout of the target image according to the layout detection; inputting the at least one sub-layout into the multimodal large language model to generate at least one sub-layout code. The layout detection is used to detect the layout on the target image, for example, a card, a list, etc. on the target image. Figure 2 As stated, Figure 2 Five sub-layouts in the target image are displayed, namely, layout 1, layout 2, layout 3, layout 4 and layout 5. The element components in each sub-layout are not specifically displayed, and are only for exemplary description.

[0091] In a possible implementation, assuming that there are vector diagrams of some element components in the sub-layout 1 and the sub-layout 4, when the sub-layout 1 generates the sub-layout code, it is necessary to obtain the vector diagrams of some element components included in the sub-layout 1, and then input the sub-layout 1 and the vector diagrams of some element components included in the sub-layout 1 into the multimodal large language model to generate the sub-layout 1 code; similarly, when the sub-layout 4 generates the sub-layout code, it is necessary to obtain the vector diagrams of some element components included in the sub-layout 4, and then input the sub-layout 4 and the vector diagrams of some element components included in the sub-layout 4 into the multimodal large language model to generate the sub-layout 4 code.

[0092] In a possible implementation, assuming that no vector graphics exist for the element components in the sub-layout 2, sub-layout 3, and sub-layout 5, the sub-layout 2, sub-layout 3, and sub-layout 5 are respectively input into the multimodal large language model to generate sub-layout 2 code, sub-layout 3 code, and sub-layout 5 code. This is only an exemplary description.

[0093] In the embodiment of the present invention, the layout detection may also construct a hierarchical layout tree for representing the nesting and relative relationship between the sub-layouts.

[0094] In a possible implementation, generating annotation information for each element component according to the element component detection specifically includes: generating description information for each element component according to the element component detection; saving the description information of each element component in a material library cache; obtaining the description information of each element component in the material library cache, and generating annotation information for each element component according to set rules, wherein the annotation information is the description information expressed in natural language; wherein the element component detection is used to detect buttons, icons, pictures or labels on the target image; the annotation information can also be called a prompt word (Prompt), and the Prompt is used to guide or instruct the multimodal large language model to generate input text or instructions for specific output.

[0095] Specifically, the element component detection may also be referred to as element detection, component detection or UI component detection (UIelement Detection).

[0096] Step S103: input the target image, the at least one sub-layout code, the image layout wireframe diagram, and the annotation information of the element components into a multimodal large language model to generate a global code.

[0097] Specifically, the multimodal large language model (MLLM) is an artificial intelligence (AI) model. The MLLM has shown amazing results in different applications, especially in knowledge memory, text comprehension and text generation. The multimodal large language model is based on the large-scale language model. The multimodal large language model (LLM) is a model that can receive and reason about multimodal information. It can not only understand and generate text, but also process data in multiple modalities such as images, audio, and video, thus realizing cross-modal understanding and generation. The LLM is a language model constructed by a deep neural network containing more than tens of billions of parameters. It is trained by a large amount of unlabeled text using a self-supervised learning method, and can handle a variety of natural language tasks. It is an important path to artificial intelligence. The BERT model and the GPT model have promoted the development of large-scale language models. Specifically, the multimodal large language model is applied to ChatGPT, and can have natural and fluent conversations with users, answer various questions, etc. The embodiment of the present invention generates a global code from the target image through the multimodal large language model, which is a new application of the multimodal large language model.

[0098] In the embodiment of the present invention, the purpose of inputting the target image into the multimodal large language model is to obtain global design features, such as color scheme, font style, etc.

[0099] In a possible implementation, after step S103, the following steps are further included: Figure 3 As shown, Figure 3 A method for automatically generating front-end code for an image is as follows:

[0100] Step S104: Generate the UI prototype diagram according to the global code.

[0101] In a possible implementation, before step S104, the following steps are also included: Figure 4 As shown, Figure 4 A method for automatically generating front-end code for an image is as follows:

[0102] Step S105: Optimize the global code to generate an optimized global code.

[0103] Specifically, the global code optimization includes syntax checking and structural optimization of the global code, applying specific heuristic rules to ensure that the global code complies with best practices, and performing code simplification and performance optimization, such as deleting redundant code, optimizing CSS selectors, etc.

[0104] In a possible implementation, after step S104, the following steps are further included: Figure 5 As shown, Figure 5 A method for automatically generating front-end code for an image is as follows:

[0105] Step S106: Obtain user feedback annotation.

[0106] Specifically, the user feedback means that after the developer obtains the UI prototype, it may not meet the expectation, so the UI prototype needs to be modified, wherein the user feedback annotation can be called prompt.

[0107] Step S107: input the user feedback annotation into the multimodal large language model, update the global code, and then execute step S104.

[0108] Specifically, the target image, the at least one sub-layout code, the image layout wireframe, the annotation information of the element components, and the user feedback annotation are input into the multimodal large language model together to update the global code, and then update the generated UI prototype diagram, so as to iteratively modify and generate the final global code and UI prototype diagram.

[0109] In a possible implementation, while detecting the element components, a wireframe diagram of the image element layout can be generated according to the element component detection, as shown in FIG. Figure 6 As shown, the image element layout wireframe diagram frames all elements in the target image with white lines. All the elements can be buttons, icons, pictures, labels, etc. This is only an exemplary display and will be determined based on actual conditions.

[0110] In an embodiment of the present invention, after the image element layout wireframe diagram is determined, the target image, the at least one sub-layout code, the image layout wireframe diagram, and the annotation information of the element components are input into a multimodal large language model to generate a global code, which specifically includes: inputting the target image, the at least one sub-layout code, the image layout wireframe diagram, the annotation information of the element components, and the image element layout wireframe diagram into a multimodal large language model to generate a global code.

[0111] The above processing steps are fully described below through three complete embodiments, as follows: Specific embodiment 1:

[0113] In the embodiment of the present invention, a flow chart of a method for automatically generating front-end code from an image is proposed, such as Figure 7 As shown, the following steps are included:

[0114] Step S701, obtain user pictures.

[0115] The user picture may also be called a user image, a target image, etc.

[0116] Step S702: determine whether the user picture can generate an image of the UI prototype image through a multimodal large language model. If so, execute steps S703 and S704.

[0117] Specifically, step S703 and step S704 are executed in parallel.

[0118] Step S703: Perform layout detection on the user image, generate at least one sub-layout code and an image layout wireframe diagram, and then execute step S708.

[0119] Specifically, the target image generates four sub-layouts, namely sub-layout 1, sub-layout 2, sub-layout 3 and sub-layout 4.

[0120] Step S704: Element component detection is performed on the user image to generate all element components in the user image and description information of each element component.

[0121] Step S705: Save all the element components and description information of each element component in the material library cache.

[0122] Step S706: Obtain description information of each element component in the material library cache, and generate annotation information of each element component according to a set rule.

[0123] Step S707: before, after or simultaneously with executing step S706, background removal and vectorization processing are performed on some element components that can be vectorized among all the element components, vector graphics of the some element components are generated, and saved in the material library cache.

[0124] Specifically, the background removal processing is to separate the foreground and background of the local image after segmenting the local image of the element component in the target image, extract the independent element component, and generate attribute information for each element component, such as size, color, and transparency, and then vectorize the part of the element components that the multimodal large language model MLLM determines to be likely to be vectorized, wherein the MLLM can receive and generate data of different modalities.

[0125] Step S708: Input the at least one sub-layout code and the vector diagram of some element components included in the at least one sub-layout into the multimodal large language model to generate at least one sub-layout code.

[0126] Specifically, since the target image generates sub-layout 1, sub-layout 2, sub-layout 3 and sub-layout 4 after layout detection, a total of 4 sub-layouts, the sub-layout 1 and the vector graphics of some element components included in the sub-layout 1 are input into the multimodal large language model to generate sub-layout 1 code, for example, sub-layout 1.html; similarly, the sub-layout 2 generates sub-layout 2 code, for example, sub-layout 2.html; the sub-layout 3 generates layout 3 code, for example, sub-layout 3.html; the sub-layout 4 generates layout 4 code, for example, sub-layout 4.html.

[0127] Step S709: input the target image, the at least one sub-layout code, the image layout wireframe diagram, and the annotation information of the element components into a multimodal large language model to generate a global code.

[0128] In a possible implementation, after the global code is generated, the UI prototype diagram is generated according to the global code, and the UI prototype diagram is displayed to the developer.

[0129] In the embodiment of the present invention, the above method can realize direct conversion from image to code, and can restore the details in the image, while providing conversion accuracy and further improving the conversion speed. Specific embodiment 2:

[0131] In the embodiment of the present invention, Figure 7 On the basis of, after step S709, step S710 is also included, specifically as follows Figure 8 As shown, specifically including:

[0132] Step S710: Obtain user feedback annotations, input the user feedback annotations into the multimodal large language model, and update the global code.

[0133] Specifically, the user feedback annotation is proposed by the developer based on the global code generated in step S709.

[0134] In an embodiment of the present invention, by simultaneously inputting a target image and user feedback annotations (i.e., text input) into the LLM, the global code generated only by the target image is iterated, thereby improving the code quality and generating a more accurate updated global code. The UI prototype generated based on the updated global code can restore complex layout and style details. Specific embodiment three:

[0136] In the embodiment of the present invention, Figure 7 On the basis of step S704, step S711 is also included, specifically as follows Fig. 9 As shown, specifically including:

[0137] Step S711: Generate an image element layout wireframe diagram according to the element component detection.

[0138] Since the above-mentioned step S711 is added, the modification of step S709 is to input the target image, the at least one sub-layout code, the image layout wireframe, the annotation information of the element components, and the image element layout wireframe into the multimodal large language model to generate a global code.

[0139] In the embodiment of the present invention, the spatial relationship between element components can be accurately identified through the image layout wireframe diagram and the image element layout wireframe diagram, and a global code with a high degree of structural matching can be generated.

[0140] Through the above-mentioned embodiments, product managers can quickly create UI prototypes directly based on existing designs without having to communicate and modify repeatedly with designers and developers who write code. The product manager is a type of developer, but does not need to write code. Under normal circumstances, the product manager first determines the product requirements, and then communicates with designers and developers who write code based on the product requirements to generate a design draft, and generates a UI prototype based on the design draft. The LLM-based front-end code generation method proposed in the embodiment of the present invention can enable product managers to design prototypes more independently, improve the efficiency of prototype design, and shorten the time from idea to visual prototype; and can make full use of existing design assets to ensure that the new prototype is consistent with the existing design style.

[0141] In a possible implementation, after a directly usable application is generated by the method of automatically generating front-end code from an image, the following is specifically shown: Fig.10 As shown, the application can "upload pictures" and upload the pictures that need to generate UI prototypes. Fig.10 The foreign language image shown on the left, then click the "Generate Now" button to generate Fig.10 The "Preview" page shown on the right, where the "Preview" page is the UI prototype. When the "Preview" page is generated, the "code" is generated at the same time. Fig.10 Select the display page in the code without displaying the code in detail; Fig.10 In addition to the direct conversion from image to UI prototype, it can also provide the functions of converting image to design draft and web page to design draft. The specific demonstration is not given here. You can refer to the conversion method of MasterGo in the prior art and use the converted design draft as "upload image" to obtain more accurate code and UI prototype. Fig.10 The design draft in the embodiment of the present invention can also be understood as the UI prototype diagram. Fig.10 Additional functions can also be designed, such as main image selection, ratio selection, image size selection, etc., as additional conditions to generate more accurate code and UI prototypes; and, prompts can be manually input, such as "for example, change the click button in the middle of the image to an input box", etc., and the image and text can be simultaneously input into the UI intelligent generator built by the multimodal large language model to obtain more accurate code and UI prototypes. The UI prototype can be "regenerated", "edited again", "saved as a sample" and "viewed in full screen"; and the application is suitable for computer or mobile terminals.

[0142] In an embodiment of the present invention, a device for automatically generating front-end code from an image is provided. Fig.11As shown, it specifically includes: an acquisition unit 1101, a processing unit 1102 and a generation unit 1103; wherein the acquisition unit 1101 is used to acquire a target image, wherein the target image is any image input by a user; the processing unit 1102, in response to the target image being an image that can generate a user interface UI prototype image, is used to generate at least one sub-layout code and an image layout wireframe according to layout detection, and generate annotation information of each element component according to element component detection, wherein the annotation information includes description information of each element component, and the description information includes location information, category information and style information; the generation unit 1103 is used to input the target image, the at least one sub-layout code, the image layout wireframe and the annotation information of the element component into a multimodal large language model to generate a global code.

[0143] Furthermore, the generating unit is also used for:

[0144] The UI prototype diagram is generated according to the global code.

[0145] Furthermore, the processing unit is specifically used for:

[0146] determining at least one sub-layout of the target image according to the layout detection;

[0147] Obtaining a vector diagram of some element components included in the at least one sub-layout;

[0148] The at least one sub-layout and a vector diagram of some element components included in the at least one sub-layout are input into a multimodal large language model to generate at least one sub-layout code.

[0149] Furthermore, the processing unit is specifically used for:

[0150] determining at least one sub-layout of the target image according to the layout detection;

[0151] The at least one sub-layout is input into the multimodal large language model to generate at least one sub-layout code.

[0152] Furthermore, the processing unit is specifically used for:

[0153] Generating description information of each element component according to the element component detection;

[0154] The description information of each element component is stored in the material library cache;

[0155] The description information of each element component is obtained in the material library cache, and the annotation information of each element component is generated according to the set rules, wherein the annotation information is the description information expressed in natural language.

[0156] Furthermore, the processing unit is also used for:

[0157] An image element layout wireframe diagram is generated according to the element component detection.

[0158] Furthermore, the generating unit is specifically used for:

[0159] The target image, the at least one sub-layout code, the image layout wireframe diagram, the annotation information of the element components, and the image element layout wireframe diagram are input into a multimodal large language model to generate a global code.

[0160] Furthermore, the device also includes an updating unit, which is used to obtain user feedback annotations; input the user feedback annotations into the multimodal large language model, and update the global code.

[0161] Furthermore, the processing unit is also used for:

[0162] Determine all element components in the target image according to the element component detection;

[0163] Background removal and vectorization are performed on some element components that can be vectorized among all the element components to generate vector graphics of the some element components.

[0164] Furthermore, the device also includes an optimization unit, which is used to optimize the global code and generate an optimized global code.

[0165] Fig.12 Schematic diagram of the structure of the electronic device in the embodiment of the present invention. Fig.12 As shown, it includes a general computer hardware structure, which at least includes a processor 1201 and a memory 1202. The processor 1201 and the memory 1202 are connected via a bus 1203. The memory 1202 is suitable for storing instructions or programs executable by the processor 1201. The processor 1201 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 1201 executes the instructions stored in the memory 1202, thereby executing the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices. The bus 1203 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 1204 and the display device and the input / output (I / O) device 1205. The input / output (I / O) device 1205 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 1205 is connected to the system through an input / output (I / O) controller 1206.

[0166] Among them, the instructions stored in the memory 1202 are executed by at least one processor 1201 to achieve: obtaining a target image, wherein the target image is any image input by a user; in response to the target image being an image that can generate a user interface UI prototype image, generating at least one sub-layout code and an image layout wireframe diagram according to layout detection, and generating annotation information for each element component according to element component detection, wherein the annotation information includes description information of each element component, and the description information includes location information, category information and style information; inputting the target image, the at least one sub-layout code, the image layout wireframe diagram and the annotation information of the element component into a multimodal large language model to generate a global code.

[0167] Specifically, the electronic device includes: one or more processors 1201 and a memory 1202, Fig.12 Take a processor 1201 as an example. The processor 1201 and the memory 1202 may be connected via a bus or other means. Fig.12 The example of the bus connection is taken as an example. The memory 1202 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The processor 1201 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions and modules stored in the memory 1202, that is, the method of automatically generating front-end code by determining the image is implemented.

[0168] The memory 1202 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store a list of options, etc. In addition, the memory 1202 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1202 may optionally include a memory remotely arranged relative to the processor 1201, and these remote memories may be connected to an external device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0169] One or more modules are stored in the memory 1202, and when executed by one or more processors 1201, the method of automatically generating front-end code from an image in any of the above method embodiments is executed.

[0170] As will be appreciated by those skilled in the art, various aspects of the embodiments of the present invention may be implemented as a system, method or computer program product. Therefore, various aspects of the embodiments of the present invention may take the form of a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.), or an implementation that combines software aspects with hardware aspects, which may generally be referred to herein as a "circuit," "module," or "system." In addition, various aspects of the embodiments of the present invention may take the form of a computer program product implemented in one or more computer-readable media having a computer-readable program code implemented thereon.

[0171] Any combination of one or more computer-readable media can be utilized. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, (but not limited to) electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or apparatuses, or any suitable combination of the foregoing. More specific examples (non-exhaustive enumeration) of computer-readable storage media will include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of an embodiment of the present invention, a computer-readable storage medium can be any tangible medium that can contain or store a program used by an instruction execution system, device or apparatus or a program used in conjunction with an instruction execution system, device or apparatus.

[0172] A computer readable signal medium may include a propagated digital signal having a computer readable program code implemented therein, such as in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any of the following computer readable media: not a computer readable storage medium, and may communicate, propagate, or transmit a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0173] Program code embodied on a computer readable medium may be transmitted using any appropriate medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0174] The computer program code for performing operations for various aspects of the embodiments of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and conventional process programming languages ​​such as "C" programming language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer as a stand-alone software package; partially on the user's computer and partially on a remote computer; or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., by using the Internet of an Internet service provider).

[0175] The flowchart legend and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present invention described above describe various aspects of the embodiment of the present invention.It will be understood that each block of the flowchart legend and / or block diagram and the combination of the blocks in the flowchart legend and / or block diagram can be implemented by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that the instructions (executed by the processor of the computer or other programmable data processing device) create a device for implementing the function / action specified in the flowchart and / or block diagram block or block.

[0176] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing device, or other apparatus to operate in a particular manner, so that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in the flowchart and / or block diagram blocks or blocks.

[0177] The computer program instructions may also be loaded onto a computer, other programmable data processing device or other apparatus so that a series of operable steps are performed on the computer, other programmable device or other apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions / actions specified in the flowchart and / or block diagram block or blocks.

[0178] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0179] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portals for users to choose to authorize or refuse. Users' refusal to process personal information other than the necessary information required for basic functions will not affect the user's use of basic functions.

Claims

1. A method for automatically generating front-end code from an image, characterized in that: The method comprises: Acquire a target image, wherein the target image is any image input by a user; In response to the target image being an image capable of generating a user interface UI prototype, generating at least one sub-layout code and an image layout wireframe according to layout detection, and generating annotation information of each element component according to element component detection, wherein the annotation information includes description information of each element component, and the description information includes position information, category information, and style information; The target image, the at least one sub-layout code, the image layout wireframe diagram, and the annotation information of the element components are input into a multimodal large language model to generate a global code.

2. The method according to claim 1, characterized in that The method further comprises: The UI prototype diagram is generated according to the global code.

3. The method according to claim 1, characterized in that Generating at least one sub-layout code according to the layout detection specifically includes: determining at least one sub-layout of the target image according to the layout detection; Obtaining a vector diagram of some element components included in the at least one sub-layout; The at least one sub-layout and a vector diagram of some element components included in the at least one sub-layout are input into the multimodal large language model to generate at least one sub-layout code.

4. The method according to claim 1, characterized in that: Generating at least one sub-layout code according to the layout detection specifically includes: determining at least one sub-layout of the target image according to the layout detection; The at least one sub-layout is input into the multimodal large language model to generate at least one sub-layout code.

5. The method according to claim 1, characterized in that Generating the annotation information of each element component according to the element component detection specifically includes: Generating description information of each element component according to the element component detection; The description information of each element component is stored in the material library cache; The description information of each element component is obtained in the material library cache, and the annotation information of each element component is generated according to the set rules, wherein the annotation information is the description information expressed in natural language.

6. The method according to claim 1, characterized in that The method further comprises: An image element layout wireframe diagram is generated according to the element component detection.

7. The method according to claim 6, characterized in that The step of inputting the target image, the at least one sub-layout code, the image layout wireframe diagram, and the annotation information of the element components into a multimodal large language model to generate a global code specifically includes: The target image, the at least one sub-layout code, the image layout wireframe diagram, the annotation information of the element components, and the image element layout wireframe diagram are input into the multimodal large language model to generate a global code.

8. The method according to claim 1, characterized in that The method further comprises: Get user feedback annotations; The user feedback annotation is input into the multimodal large language model to update the global code.

9. The method according to claim 1, characterized in that: The method further comprises: Determine all element components in the target image according to the element component detection; Background removal and vectorization are performed on some element components that can be vectorized among all the element components to generate vector graphics of the some element components.

10. The method according to claim 1, characterized in that The method further comprises: The global code is optimized to generate an optimized global code.

11. A device for automatically generating front-end code from an image, characterized in that: The device comprises: An acquisition unit, used for acquiring a target image, wherein the target image is any image input by a user; A processing unit, in response to the target image being an image capable of generating a user interface UI prototype image, is used to generate at least one sub-layout code and an image layout wireframe according to layout detection, and to generate annotation information of each element component according to element component detection, wherein the annotation information includes description information of each element component, and the description information includes position information, category information, and style information; A generating unit is used to input the target image, the at least one sub-layout code, the image layout wireframe diagram and the annotation information of the element components into a multimodal large language model to generate a global code.

12. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images