Front-end code generation method and device, storage medium and electronic device

Abstract

The invention discloses a front-end code generation method and device, a storage medium and an electronic device, and relates to the technical field of code generation, and the method comprises the steps: obtaining a recognition result obtained by carrying out image recognition on an interface image through a multi-modal large language model; according to the target project code, a code knowledge base is constructed, and the code knowledge base is used for retrieval enhancement generation of the multi-modal large language model; and obtaining a first front-end code generated by the multi-modal large language model based on the recognition result and the code knowledge base. By adopting the technical scheme, the problem that the generation quality of the front-end code is low in the prior art is solved, and the effect of improving the generation quality of the front-end code can be further achieved.

Description

Technical Field

[0001] This invention relates to the field of code generation technology, and more specifically, to a method and apparatus for generating front-end code, a storage medium, and an electronic device. Background Technology

[0002] In the field of front-end development, most existing technical solutions for automatically converting UI design drafts into code rely on fixed image recognition algorithms and pre-set code templates. These solutions suffer from problems such as limited recognition of element types, inability to adapt to specific project technology stacks and code specifications, and poor usability of generated code. As a result, they are difficult to apply directly in complex real-world projects, and their effect on improving development efficiency is limited.

[0003] There is currently no effective solution to the problem of low-quality front-end code generation due to related technologies.

[0004] Therefore, it is necessary to improve the relevant technology to overcome the aforementioned defects. Summary of the Invention

[0005] This invention provides a method and apparatus for generating front-end code, a storage medium, and an electronic device to at least solve the problem of low quality of generated front-end code.

[0006] According to one aspect of the present invention, a method for generating front-end code is provided, comprising: obtaining a recognition result obtained by performing image recognition on an interface image through a multimodal large language model; constructing a code knowledge base based on target project code, wherein the code knowledge base is used for retrieval enhancement generation of the multimodal large language model; and obtaining first front-end code generated by the multimodal large language model based on the recognition result and the code knowledge base.

[0007] In an exemplary embodiment, obtaining the recognition result obtained by image recognition of an interface image through a multimodal large language model includes: inputting a first preset prompt word and the interface image into the multimodal large language model, wherein the first preset prompt word is used to prompt the multimodal large language model to perform a first task, the first task being to recognize the text information, page layout information, and element information of the interface image, the element information including element type, element size, and element position; and determining the recognition result based on the text information, page layout information, and element information output by the multimodal large language model performing the first task.

[0008] In one exemplary embodiment, constructing a code knowledge base based on target project code includes: parsing the target project code to obtain the code format and code component template of the target project code; vectorizing the code format to obtain a format vector, and vectorizing the code component template to obtain a template vector; and generating the code knowledge base based on the format vector and the template vector.

[0009] In one exemplary embodiment, obtaining the first front-end code generated by the multimodal large language model based on the recognition result and the code knowledge base includes: inputting a second preset prompt word and the recognition result into the multimodal large language model, wherein the second preset prompt word is used to prompt the multimodal large language model to perform a second task, the second task being to retrieve target code knowledge from the code knowledge base based on the recognition result and generate front-end code according to the target code knowledge; obtaining the first front-end code output by the multimodal large language model performing the second task, wherein the first front-end code satisfies the code specifications in the code knowledge base, and the components in the first front-end code are obtained by implementing the code component templates in the code knowledge base.

[0010] In an exemplary embodiment, after obtaining the first front-end code generated by the multimodal large language model based on the recognition result and the code knowledge base, the method further includes: compiling the first front-end code using a code compilation tool to obtain a first front-end compiled file; obtaining a first front-end page obtained by loading the first front-end compiled file; comparing the first front-end page with the interface image using the multimodal large language model to obtain a comparison result; and modifying the first front-end code according to the comparison result to obtain a second front-end code.

[0011] In one exemplary embodiment, modifying the first front-end code according to the comparison result to obtain the second front-end code includes: when it is determined that the comparison result indicates that the similarity between the first front-end page and the interface image is less than a preset similarity, marking the difference regions between the first front-end page and the interface image using the multimodal large language model; modifying the first front-end code according to the difference regions to obtain the second front-end code, wherein the similarity between the second front-end page corresponding to the second front-end code and the interface image is greater than the preset similarity.

[0012] In an exemplary embodiment, after modifying the first front-end code according to the comparison result to obtain the second front-end code, the method further includes: compiling the second front-end code using a code compilation tool to obtain a second front-end compiled file; obtaining a second front-end page obtained by loading the second front-end compiled file; and modifying the second front-end code to obtain a third front-end code when it is determined that there are elements in the second front-end page that do not conform to the preset element functions, wherein all page elements in the third front-end page corresponding to the third front-end code conform to the preset element functions.

[0013] According to another aspect of the present invention, a front-end code generation apparatus is also provided, comprising: an acquisition module, configured to acquire a recognition result obtained by image recognition of an interface image through a multimodal large language model; a construction module, configured to construct a code knowledge base based on target project code, wherein the code knowledge base is used for retrieval enhancement generation of the multimodal large language model; and a generation module, configured to acquire first front-end code generated by the multimodal large language model based on the recognition result and the code knowledge base.

[0014] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, wherein the computer program is configured to execute the above-described front-end code generation method at runtime.

[0015] According to another aspect of the present invention, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the aforementioned front-end code generation method through the computer program.

[0016] This invention obtains the recognition results of interface images through image recognition using a multimodal large language model; constructs a code knowledge base based on the target project code, wherein the code knowledge base is used for retrieval-enhanced generation by the multimodal large language model; and obtains the first front-end code generated by the multimodal large language model based on the recognition results and the code knowledge base. By employing the above technical solution, UI image content is recognized through a multimodal large language model, and a project-specific code knowledge base based on retrieval-enhanced generation is constructed. Front-end code that meets project requirements is generated based on the recognition results and knowledge base content, thereby achieving intelligent conversion from design drafts to high-quality front-end code. This solves the problem of low-quality front-end code generation in related technologies, thus improving the quality of front-end code generation. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with the description thereof, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of a computer terminal that executes the front-end code generation method of the embodiments of the present invention;

[0019] Figure 2 This is a flowchart of a method for generating front-end code according to an embodiment of the present invention;

[0020] Figure 3 This is a flowchart illustrating a method for generating front-end code according to an embodiment of the present invention;

[0021] Figure 4 This is a structural block diagram of a front-end code generation apparatus according to an embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms and terms such as "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] The methods and embodiments provided in this invention can be executed on a computer terminal or similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal that executes the front-end code generation method of this invention. For example... Figure 1 As shown, a computer terminal may include one or more ( Figure 1Only one is shown in the diagram. The processor 102 (which may include, but is not limited to, a microprocessor unit (MPU) or a programmable logic device (PLD)) and a memory 104 configured to store data are also included. In one exemplary embodiment, the computer terminal may further include a transmission device 106 configured for communication and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 Equivalent functions or ratios shown Figure 1 The functions shown have more different configurations.

[0025] The memory 104 may be configured to store computer programs, such as application software programs and modules, like the computer program corresponding to the front-end code generation method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the aforementioned method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0026] The transmission device 106 is configured to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module configured to communicate with the Internet wirelessly.

[0027] This embodiment provides a method for generating front-end code. Figure 2 This is a flowchart of a method for generating front-end code according to an embodiment of the present invention, the process including the following steps:

[0028] Step S202: Obtain the recognition result obtained by performing image recognition on the interface image through a multimodal large language model;

[0029] Among them, the aforementioned multimodal large language model can process information in modalities such as text and images.

[0030] Step S204: Construct a code knowledge base based on the target project code, wherein the code knowledge base is used for the retrieval enhancement generation of the multimodal large language model;

[0031] Step S206: Obtain the first front-end code generated by the multimodal large language model based on the recognition result and the code knowledge base.

[0032] Through the above steps, the recognition results obtained by image recognition of interface images using a multimodal large language model are acquired; a code knowledge base is constructed based on the target project code, wherein the code knowledge base is used for retrieval-enhanced generation by the multimodal large language model; and the first front-end code generated by the multimodal large language model based on the recognition results and the code knowledge base is obtained. By adopting the above technical solution, UI image content is recognized through a multimodal large language model, and a project-specific code knowledge base based on retrieval-enhanced generation is constructed. Front-end code that meets project requirements is generated based on the recognition results and knowledge base content, thereby achieving intelligent conversion from design drafts to high-quality front-end code. This solves the problem of low quality front-end code generation in related technologies, thus improving the quality of front-end code generation.

[0033] In an exemplary embodiment, obtaining the recognition result obtained by image recognition of an interface image through a multimodal large language model includes: inputting a first preset prompt word and the interface image into the multimodal large language model, wherein the first preset prompt word is used to prompt the multimodal large language model to perform a first task, the first task being to recognize the text information, page layout information, and element information of the interface image, the element information including element type, element size, and element position; and determining the recognition result based on the text information, page layout information, and element information output by the multimodal large language model performing the first task.

[0034] Optionally, in the above embodiments, for example, the first preset prompt is "Please identify the text content, page layout structure, element type and its position and size in the image. The identification result is output in JSON or segmented text form, including text content (such as title, label text), layout information (such as top and bottom structure, grid system) and element information (such as button, input box type, width, height, coordinates, etc.)".

[0035] For example, for a "product list page" design, the first preset prompt word guide model outputs the following recognition results: text information: "product name", "price", "inventory", etc.; page layout: search bar at the top, table layout in the middle, and pagination component at the bottom; element information: button (type: primary, size: 100x40px, position: top right corner), table (row height: 50px, number of columns: 4).

[0036] In one exemplary embodiment, constructing a code knowledge base based on target project code includes: parsing the target project code to obtain the code format and code component template of the target project code; vectorizing the code format to obtain a format vector, and vectorizing the code component template to obtain a template vector; and generating the code knowledge base based on the format vector and the template vector.

[0037] Optionally, in the above embodiments, the target project code is the source code of the current project, which is obtained by extracting code formatting specifications (such as indentation, naming rules, and comment requirements) and common component templates (such as standard implementations of forms, tables, and navigation bars). This content is converted into vector representations through an embedding model (such as Sentence-BERT) and stored in a vector database to form a searchable code knowledge base.

[0038] In one exemplary embodiment, obtaining the first front-end code generated by the multimodal large language model based on the recognition result and the code knowledge base includes: inputting a second preset prompt word and the recognition result into the multimodal large language model, wherein the second preset prompt word is used to prompt the multimodal large language model to perform a second task, the second task being to retrieve target code knowledge from the code knowledge base based on the recognition result and generate front-end code according to the target code knowledge; obtaining the first front-end code output by the multimodal large language model performing the second task, wherein the first front-end code satisfies the code specifications in the code knowledge base, and the components in the first front-end code are obtained by implementing the code component templates in the code knowledge base.

[0039] Optionally, in the above embodiments, for example, a second preset prompt, "Please retrieve relevant content from the code knowledge base based on the recognition results, and generate front-end code that conforms to the project specifications based on the recognition results and the retrieved content," guides the multimodal large language model to perform the code generation task. The model first retrieves content related to the recognition results from the code knowledge base (such as button component template implementation and layout style implementation), and then combines the recognition results to generate complete front-end code. The code conforms to the indentation, naming, and style specifications stipulated by the project, and the component implementation conforms to the project standards.

[0040] In an exemplary embodiment, after obtaining the first front-end code generated by the multimodal large language model based on the recognition result and the code knowledge base, the method further includes: compiling the first front-end code using a code compilation tool to obtain a first front-end compiled file; obtaining a first front-end page obtained by loading the first front-end compiled file; comparing the first front-end page with the interface image using the multimodal large language model to obtain a comparison result; and modifying the first front-end code according to the comparison result to obtain a second front-end code.

[0041] Optionally, in the above embodiments, a compilation tool (such as Vue's hot compilation module) is used to compile the generated first front-end code into a runnable HTML / CSS / JS file, which is then rendered as an actual page in the browser. Subsequently, the multimodal large language model compares the rendered page screenshot with the original UI design, identifies visual differences (such as color discrepancies and layout misalignments), and generates optimization suggestions or directly modifies the code. For example, after the first front-end code is rendered, the page button color is blue, while it is red in the design. After comparison, the multimodal large language model outputs: "Button color mismatch, it is recommended to change to #FF0000". Based on this, the system automatically modifies the color values ​​in the CSS and generates the second front-end code.

[0042] In one exemplary embodiment, modifying the first front-end code according to the comparison result to obtain the second front-end code includes: when it is determined that the comparison result indicates that the similarity between the first front-end page and the interface image is less than a preset similarity, marking the difference regions between the first front-end page and the interface image using the multimodal large language model; modifying the first front-end code according to the difference regions to obtain the second front-end code, wherein the similarity between the second front-end page corresponding to the second front-end code and the interface image is greater than the preset similarity.

[0043] Optionally, in the above embodiments, the difference identification and code modification process is refined. The system uses an image similarity algorithm (such as SSIM) to quantify the differences between the page and the design draft. If the similarity is below a threshold (e.g., 90%), the system uses a multimodal large language model to locate specific difference areas (e.g., "inconsistent title font size" or "image position offset") and adjusts the style or layout attributes in the code accordingly. For example, if the similarity calculation is 85%, the model marks "title area offset 20px to the left". The system modifies the margin-left property in the CSS to align the second front-end page with the design draft, increasing the similarity to 95%.

[0044] In an exemplary embodiment, after modifying the first front-end code according to the comparison result to obtain the second front-end code, the method further includes: compiling the second front-end code using a code compilation tool to obtain a second front-end compiled file; obtaining a second front-end page obtained by loading the second front-end compiled file; and modifying the second front-end code to obtain a third front-end code when it is determined that there are elements in the second front-end page that do not conform to the preset element functions, wherein all page elements in the third front-end page corresponding to the third front-end code conform to the preset element functions.

[0045] Optionally, in the above embodiments, a functional verification step is further added. After visual optimization, the interactive functions of the rendered page are tested, such as checking whether buttons are clickable, whether input boxes support input, and whether tables can be sorted. If any missing or abnormal functions are found, the system automatically repairs the code, adding event listeners or logic implementations to ensure that the generated front-end code not only has high visual fidelity but also complete interactive functions. For example, the code requirement is that the "search button" in the second front-end page needs to be bound to a click event. After the second front-end page generated by the large language model with prompt words is reviewed and it is detected that the "search button" is not bound to a click event, the @click="handleSearch" method is added to the Vue code to generate the third front-end code, enabling the button to have search functionality.

[0046] Obviously, the embodiments described above are merely some embodiments of the present invention, and not all embodiments. To better understand the above-described front-end code generation method, the process is explained below with reference to embodiments, but this is not intended to limit the technical solutions of the embodiments of the present invention. Specifically:

[0047] In an optional embodiment, such as Figure 3 As shown, the process of generating front-end code includes the following steps:

[0048] 1. User inputs UI image;

[0049] 2. Multimodal large model performs content recognition and outputs recognition results, describing layout, elements, positions, sizes, and other information in different segments. Below is an example of a complete recognition result:

[0050] Page Type: This is a cover page, typically used at the beginning of a presentation or report. It primarily displays basic information such as the topic, author information, and date.

[0051] Page layout overview:

[0052] Top area: includes the logo area and the title area.

[0053] Logo area: The company logo is displayed in the upper left corner, and the logo is against a gray gradient background in the upper right corner.

[0054] Title area: The page theme is displayed in prominent red font in the center.

[0055] Central area: including the image area.

[0056] Image area: Displays an image of size N*M.

[0057] Bottom area: The author's name "XXX" is displayed in the lower right corner, and the date "2021.11" and company information "XX Company" are displayed below the name.

[0058] Function Description:

[0059] Information display function: The main function of the cover page is to provide basic background information, including the subject, author and date, to lay the groundwork for subsequent content.

[0060] Visual guidance function: Attract viewers' attention and stimulate their interest in exploration through eye-catching titles and atmospheric images of snow-capped mountains.

[0061] Hierarchical structure:

[0062] First layer: Cover page.

[0063] —Second level: Top level.

[0064] —Third layer: Logo area. Content: Company logo.

[0065] —Third layer: Title area. Content: Subject text *********.

[0066] —Second level: Middle level.

[0067] —Third layer: Image area. Content: Images of mountain climbers.

[0068] —Second level: Bottom level.

[0069] —Third layer: Author information area.

[0070] —Third layer: Date area. Content: Time "2021.11".

[0071] —Third layer: Company information area. Content: Company name.

[0072] Element recognition:

[0073] Title: Text title, text size 28p, left margin 200px.

[0074] Image: 300px wide, 400px high, 100px left margin.

[0075] …”.

[0076] 3. Build a RAG knowledge base, including code style guides, component knowledge, template code, etc.

[0077] The project has accumulated standard code for various standard pages. It can define code implementation styles for various common page scenarios, such as the implementation of tree components, tables, and pagination components. Furthermore, the template code uses common components accumulated from business logic; after generating the RAG knowledge base, the final generated code can directly use these common components or styles.

[0078] Code style guidelines are the coding standards for the relevant languages ​​specified for a project, such as code appearance, declarations, naming conventions, and statement syntax (if-else, switch, for, try-catch, etc.). They constrain code style, the logic of variables and functions, and comments. Under these guidelines, large models can be assisted in generating code with the same style and style. For example:

[0079] Column width: The code column width is controlled within 120 characters.

[0080] Line breaks: When an expression exceeds or is about to exceed the specified column width, follow these rules for line breaks: Start a new line after a comma; statements must end with a semicolon, do not omit the semicolon; start a new line before operators, placing the operator at the beginning of the new line; start a new line at lower priority operators; it is recommended to indent the new line by 8 spaces, and the next level of statement by 4 spaces, to ensure neat formatting and readability.

[0081] Indentation: Indentation should be one tab (4 spaces) per line. Do not use the tab character in the code.

[0082] Statement: [Recommendation] Only one statement per line.

[0083] Location: [Recommendation] Variables should be placed at the beginning of a block, rather than always declaring them where they are first used.

[0084] Naming conventions: Names should describe "what" rather than "how". By avoiding names that expose the underlying implementation, you can preserve a layer of abstraction that simplifies complexity.

[0085] Naming Principles: Difficulty in choosing the right name may indicate a need for further analysis or definition of the item's purpose. Make names long enough to be meaningful, and short enough to avoid verbosity. Unique names are used solely in programming to distinguish items. Expressive names are meant to help people read; therefore, providing names that people can understand makes sense. However, ensure that the chosen name conforms to the rules and standards of the applicable language.

[0086] Note: Comments should comprise at least 30% of the total code; when modifying code, always keep surrounding comments up-to-date; for special or difficult-to-understand business logic, complete comments are mandatory; avoid adding comments at the end of lines of code; end-of-line comments make code harder to read.

[0087] 4. The large model retrieves relevant content from the RAG knowledge base based on the recognition results to generate front-end code.

[0088] 5. Analyze the rendering code effects.

[0089] The project uses a unified webpack + vue + antdesign technology stack. The generated code can be directly rendered into a web interface using Vue's hot-compile module, and the rendering result can be viewed directly in the browser. This allows for a quick comparison of the quality of the generated code and its accuracy in reproducing the UI design.

[0090] 6. Based on the generated code and UI diagrams, the multimodal model re-understands the UI content, compares it with the elements and screenshots in the current code, and performs another comparison. It identifies differences and areas of insufficient fidelity, proactively optimizes these differences, and improves fidelity until the code achieves the expected results.

[0091] Through the above embodiments, this solution accurately identifies the layout, elements, and style information of UI design drafts using a multimodal large model. It then incorporates project code standards and component templates from the RAG knowledge base to generate high-quality front-end code that meets the technology stack requirements. A real-time rendering engine provides an intuitive preview of the generated effect, and a multi-round comparison and optimization mechanism automatically corrects visual differences from the original image, achieving automated conversion from design to code. Furthermore, the generated code quality is significantly improved, making it more readily compatible with project development needs.

[0092] From the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0093] This embodiment also provides a front-end code generation apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0094] Figure 4 This is a structural block diagram of a front-end code generation apparatus according to an embodiment of the present invention, the apparatus comprising:

[0095] The acquisition module 42 is used to acquire the recognition results obtained by performing image recognition on the interface image through a multimodal large language model;

[0096] Module 44 is used to build a code knowledge base based on the target project code, wherein the code knowledge base is used for the retrieval enhancement generation of the multimodal large language model;

[0097] The generation module 46 is used to obtain the first front-end code generated by the multimodal large language model based on the recognition results and the code knowledge base.

[0098] The front-end code generation apparatus of this invention obtains the recognition result obtained by image recognition of interface images through a multimodal large language model; constructs a code knowledge base based on the target project code, wherein the code knowledge base is used for retrieval-enhanced generation of the multimodal large language model; and obtains the first front-end code generated by the multimodal large language model based on the recognition result and the code knowledge base. By adopting the above technical solution, UI image content is recognized through a multimodal large language model, and a project-specific code knowledge base based on retrieval-enhanced generation is constructed. Front-end code that meets project requirements is generated based on the recognition result and the knowledge base content, thereby achieving intelligent conversion from design drafts to high-quality front-end code. This solves the problem of low quality front-end code generation in related technologies, thus improving the quality of front-end code generation.

[0099] In an exemplary embodiment, the acquisition module 42 is configured to input the first preset prompt word and the interface image into the multimodal large language model, wherein the first preset prompt word is used to prompt the multimodal large language model to perform a first task, the first task being to recognize the text information, page layout information and element information of the interface image, the element information including element type, element size and element position; and to determine the recognition result based on the text information, page layout information and element information output by the multimodal large language model in performing the first task.

[0100] In an exemplary embodiment, the construction module 44 is configured to parse the target project code to obtain the code format and code component template of the target project code; to vectorize the code format to obtain a format vector, and to vectorize the code component template to obtain a template vector; and to generate the code knowledge base based on the format vector and the template vector.

[0101] In an exemplary embodiment, the generation module 46 is configured to input the second preset prompt word and the recognition result into the multimodal large language model, wherein the second preset prompt word is used to prompt the multimodal large language model to perform a second task, the second task being to retrieve target code knowledge from the code knowledge base based on the recognition result and generate front-end code based on the target code knowledge; and to obtain the first front-end code output by the multimodal large language model performing the second task, wherein the first front-end code satisfies the code specifications in the code knowledge base, and the components in the first front-end code are obtained by implementing the code component templates in the code knowledge base.

[0102] In an exemplary embodiment, the generation module 46 is configured to: compile the first front-end code using a code compilation tool to obtain a first front-end compiled file; obtain a first front-end page obtained by loading the first front-end compiled file; compare the first front-end page with the interface image using the multimodal large language model to obtain a comparison result; and modify the first front-end code according to the comparison result to obtain a second front-end code.

[0103] In an exemplary embodiment, the generation module 46 is configured to, when determining that the comparison result indicates that the similarity between the first front-end page and the interface image is less than a preset similarity, mark the difference regions between the first front-end page and the interface image using the multimodal large language model; modify the first front-end code according to the difference regions to obtain the second front-end code, wherein the second front-end page corresponding to the second front-end code has a similarity greater than the preset similarity with the interface image.

[0104] In an exemplary embodiment, the generation module 46 is configured to compile the second front-end code using a code compilation tool to obtain a second front-end compiled file; obtain a second front-end page obtained by loading the second front-end compiled file; and modify the second front-end code to obtain a third front-end code if it is determined that there are elements in the second front-end page that do not conform to the preset element functions. The page elements in the third front-end page corresponding to the third front-end code all conform to the preset element functions.

[0105] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0106] S1, Obtain the recognition result obtained by performing image recognition on the interface image through a multimodal large language model;

[0107] S2, Construct a code knowledge base based on the target project code, wherein the code knowledge base is used for retrieval enhancement generation of the multimodal large language model;

[0108] S3, obtain the first front-end code generated by the multimodal large language model based on the recognition result and the code knowledge base.

[0109] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0110] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0111] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0112] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0113] S1, Obtain the recognition result obtained by performing image recognition on the interface image through a multimodal large language model;

[0114] S2, Construct a code knowledge base based on the target project code, wherein the code knowledge base is used for retrieval enhancement generation of the multimodal large language model;

[0115] S3, obtain the first front-end code generated by the multimodal large language model based on the recognition result and the code knowledge base.

[0116] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0117] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0118] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0119] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating front-end code, characterized in that, include: Obtain the recognition results obtained by performing image recognition on interface images using a multimodal large language model; A code knowledge base is constructed based on the target project code, wherein the code knowledge base is used for retrieval enhancement and generation of the multimodal large language model; Obtain the first front-end code generated by the multimodal large language model based on the recognition results and the code knowledge base.

2. The method for generating front-end code according to claim 1, characterized in that, Obtain the recognition results obtained by performing image recognition on interface images using a multimodal large language model, including: The first preset prompt word and the interface image are input into the multimodal large language model, wherein the first preset prompt word is used to prompt the multimodal large language model to perform a first task, the first task being to identify the text information, page layout information and element information of the interface image, the element information including element type, element size and element position; The recognition result is determined based on the text information, page layout information, and element information output by the multimodal large language model when performing the first task.

3. The method for generating front-end code according to claim 1, characterized in that, Build a code knowledge base based on the target project's code, including: The target project code is parsed to obtain the code format and code component template of the target project code; The code format is vectorized to obtain a format vector, and the code component template is vectorized to obtain a template vector; The code knowledge base is generated based on the format vector and the template vector.

4. The method for generating front-end code according to claim 3, characterized in that, Obtaining the first front-end code generated by the multimodal large language model based on the recognition results and the code knowledge base includes: The second preset prompt word and the recognition result are input into the multimodal large language model, wherein the second preset prompt word is used to prompt the multimodal large language model to perform a second task, the second task being to retrieve target code knowledge from the code knowledge base based on the recognition result, and to generate front-end code based on the target code knowledge; Obtain the first front-end code output by the multimodal large language model executing the second task, wherein the first front-end code satisfies the code specifications in the code knowledge base, and the components in the first front-end code are obtained by implementing the code component templates in the code knowledge base.

5. The method for generating front-end code according to claim 1, characterized in that, After obtaining the first front-end code generated by the multimodal large language model based on the recognition results and the code knowledge base, the method further includes: The first front-end code is compiled using a code compilation tool to obtain a first front-end compiled file; Obtain the first front-end page obtained by loading the first front-end compiled file; The comparison results are obtained by comparing the first front-end page with the interface image using the multimodal large language model. Based on the comparison results, the first front-end code is modified to obtain the second front-end code.

6. The method for generating front-end code according to claim 5, characterized in that, Based on the comparison results, the first front-end code is modified to obtain the second front-end code, including: If the comparison result indicates that the similarity between the first front-end page and the interface image is less than a preset similarity, the difference regions between the first front-end page and the interface image are marked by the multimodal large language model. The first front-end code is modified according to the difference region to obtain the second front-end code, wherein the similarity between the second front-end page corresponding to the second front-end code and the interface image is greater than the preset similarity.

7. The method for generating front-end code according to claim 5, characterized in that, After modifying the first front-end code based on the comparison results to obtain the second front-end code, the method further includes: The second front-end code is compiled using a code compilation tool to obtain the second front-end compiled file; Obtain the second front-end page obtained by loading the second front-end compiled file; If it is determined that there are elements in the second front-end page that do not conform to the preset element functions, the second front-end code is modified to obtain the third front-end code, wherein all page elements in the third front-end page corresponding to the third front-end code conform to the preset element functions.

8. A front-end code generation device, characterized in that, include: The acquisition module is used to acquire the recognition results obtained by performing image recognition on the interface image through a multimodal large language model; A building module is used to build a code knowledge base based on the target project code, wherein the code knowledge base is used for the retrieval enhancement generation of the multimodal large language model; The generation module is used to obtain the first front-end code generated by the multimodal large language model based on the recognition results and the code knowledge base.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 7 through the computer program.

Images