Front-end code generation method and device, equipment, storage medium and program product

Abstract

The invention relates to the technical field of artificial intelligence, and particularly provides a front-end code generation method and device, equipment, a storage medium and a program product.The front-end code generation method comprises the steps that a user interface image and a user instruction text are obtained; target visual features are generated based on the user interface image through a visual encoder, the target visual features comprise a first visual feature and a second visual feature, the first visual feature is a visual feature obtained by performing visual encoding processing based on a fixed resolution on the user interface image, and the second visual feature is a visual feature obtained by performing visual encoding processing based on the fixed resolution on the user interface image; the second visual feature is a visual feature obtained by performing dynamic resolution-based visual coding processing on the user interface image; and through a large language model, generating a front-end code based on the target visual features and the user instruction text. By implementing the technical scheme of the embodiment of the invention, the accuracy of front-end code generation can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a front-end code generation method, apparatus, device, storage medium, and program product. Background Technology

[0002] In front-end development, front-end engineers need to carefully study UI design drafts, understand design intent, layout, color scheme, font selection, interactive effects, etc., and choose the appropriate front-end technology stack, including HyperText Markup Language (HTML), Cascading Style Sheets (CSS), JavaScript frameworks, and front-end component libraries (such as Element). This requires a lot of coding work and consumes a lot of time and energy in project development.

[0003] Automatic front-end code generation can improve coding efficiency. Currently, methods for automatically generating front-end code typically extract edge information from front-end interface design images, input the edge region images into a component recognition model, identify the position, text content, and component category of front-end interface components, assemble target data according to a preset data format based on the component's position, category, and text content, and generate front-end code using a preset code framework. However, the above methods are insufficient for handling complex user interfaces, and edge automation struggles to accurately distinguish between various front-end interface components, resulting in a high error rate in the generated front-end code. Summary of the Invention

[0004] This application is made in view of the above-mentioned problems. This application provides a front-end code generation method, apparatus, device, storage medium, and program product.

[0005] In a first aspect, embodiments of this application provide a front-end code generation method, including: Obtain user interface images and user command text; Based on the user interface image, a target visual feature is generated using a visual encoder. The target visual feature includes a first visual feature and a second visual feature. The first visual feature is a visual feature obtained by performing visual encoding processing on the user interface image based on a fixed resolution, and the second visual feature is a visual feature obtained by performing visual encoding processing on the user interface image based on a dynamic resolution. The front-end code is generated using a large language model based on the target visual features and the user command text.

[0006] Furthermore, according to a front-end code generation method of the first aspect of the present application, the first visual feature is a visual feature of fixed size, and the second visual feature is a visual feature of dynamic size.

[0007] Furthermore, according to a front-end code generation method of the first aspect of the present application, the target visual feature is obtained by the visual encoder by splicing the first visual feature and the second visual feature.

[0008] Furthermore, according to a first aspect of the embodiments of this application, a front-end code generation method includes training the large language model; Training the large language model includes: Based on the first sample data, a multimodal model including the visual encoder and the large language model is pre-trained, wherein the first sample data includes a first user interface image and the corresponding first target front-end code; The multimodal model is fine-tuned based on the second sample data, which includes a second user interface image, a second user instruction text, a first initial front-end code, and a corresponding second target front-end code.

[0009] Furthermore, according to a front-end code generation method based on a first aspect of the embodiments of this application, training the large language model further includes: The multimodal model is fine-tuned based on the third sample, which includes the third user instruction text, the second initial front-end code, and the corresponding third target front-end code.

[0010] The multimodal model is pre-trained based on the fourth sample data, wherein the fourth sample data includes images and corresponding image descriptions, or the fourth sample data includes Chinese and English document images and corresponding text content.

[0011] Furthermore, according to a front-end code generation method of the first aspect of the present application, obtaining the first sample data includes: The first prompt word for generating the front-end interface description from the documentation; Based on the first prompt word, the first major language model is used to generate a front-end interface description; Based on the aforementioned front-end interface description, a second prompt word is constructed to generate the front-end interface description code. Based on the second prompt word, the first target front-end code is generated using the second major language model; The first user interface image is generated based on the first target front-end code.

[0012] Secondly, embodiments of this application provide a front-end code generation apparatus, including: The acquisition module is used to acquire user interface images and user command text; The feature generation module is used to generate target visual features based on the user interface image through a visual encoder. The target visual features include a first visual feature and a second visual feature. The first visual feature is a visual feature obtained by performing visual encoding processing on the user interface image based on a fixed resolution, and the second visual feature is a visual feature obtained by performing visual encoding processing on the user interface image based on a dynamic resolution. The code generation module is used to generate the front-end code based on the target visual features and the user instruction text using a large language model.

[0013] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the methods described above.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0015] Fifthly, embodiments of this application provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0016] As will be described in detail below, the front-end code generation method, apparatus, device, storage medium, and program product according to embodiments of this application extract target visual features of user interface images by combining fixed resolution and dynamic resolution in a visual encoder, and combine them with user instruction text to directly generate front-end code from a large language model, thereby improving the accuracy of front-end code generation and reducing the error rate.

[0017] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0018] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1This is a flowchart illustrating a front-end code generation method according to an embodiment of this application.

[0020] Figure 2 This is an architectural diagram illustrating a front-end code generation method according to an embodiment of this application.

[0021] Figure 3 This is a flowchart illustrating the model training process in the front-end code generation method of this application.

[0022] Figure 4 This is a flowchart illustrating the sample data acquisition process in the front-end code generation method of this application embodiment.

[0023] Figure 5 This is a block diagram illustrating a front-end code generation apparatus according to an embodiment of this application.

[0024] Figure 6 This is a schematic diagram illustrating a computer program product according to an embodiment of this application.

[0025] Figure 7 This is a hardware block diagram illustrating an electronic device according to an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0027] See Figure 1 and Figure 2 A front-end code generation method, comprising: S101, Obtain the user interface image and user command text.

[0028] Users can input user interface (UI) images and user command text so that the system executing the front-end code generation method can obtain the user interface images and user command text.

[0029] In this step, the user interface (UI) image can be a UI design draft created by the user or a UI image obtained from other sources. The user instruction text is used to indicate the code generation requirements and / or rules. The user instruction text can be entered by the user or preset by the system to ensure that code generation can still be executed even if the user does not input any information. For example, if the user-input instruction text is "Generate Element front-end code based on the UI design draft," the system executing the method of this application embodiment can determine to use the Element framework for code generation based on this instruction text, thereby achieving front-end code output that matches the user's requirements.

[0030] S102, using a visual encoder, a target visual feature is generated based on the user interface image. The target visual feature includes a first visual feature and a second visual feature. The first visual feature is a visual feature obtained by performing visual encoding processing on the user interface image based on a fixed resolution, and the second visual feature is a visual feature obtained by performing visual encoding processing on the user interface image based on a dynamic resolution.

[0031] In this step, the target visual features may include the aforementioned first visual features and second visual features. These can be the first and second visual features alone, or they can be a combination of the first and second visual features through methods such as splicing. By simultaneously utilizing visual encoding processing with both fixed and dynamic resolutions, the robustness to UI images at different resolutions can be improved while maintaining the stability of the visual features. This, in turn, increases the accuracy of the front-end code generated based on the target visual features and reduces the generation error rate.

[0032] In one example, the first visual feature is a fixed-size visual feature, and the second visual feature is a dynamically sized visual feature.

[0033] In one example, the target visual features are obtained by concatenating the first and second visual features by the visual encoder 201. This allows both fixed-size and dynamic-size features to be included in a single feature representation, thereby supporting more accurate front-end code generation later.

[0034] In practical applications, user interface images typically have a wide resolution range, ranging from approximately 320×480px to 7680×4320px. Traditional CLIP (Contrastive Language-Image Pre-training) visual encoders 201 have an input resolution of 336×336px, which is insufficient for processing content-rich user interface images. The visual encoder 201 provided in this application embodiment supports both fixed-resolution and dynamic-resolution visual encoding. The visual encoder 201 may include: A first encoder with a fixed resolution, such as CLIP or SigLIP (Sigmoid Loss for Language-Image Pre-training). Secondary encoders with dynamic resolution, such as Convolutional Neural Networks (CNN) and Vision Transformers (ViT) with dynamic patch count.

[0035] The same user interface image is input into the first encoder and the second encoder respectively, resulting in a fixed-size first visual feature (e.g., visual token1) and a dynamically sized second visual feature (e.g., visual token2). These two features are then concatenated into a single target visual feature (e.g., target visual token). For example, for the fixed-resolution first encoder, an input image of size H×W is resized to 336×336 and input into visual encoder 201, resulting in a visual token1 of size 257×1024 (257 being the sequence length and 1024 being the sequence dimension). For the dynamically sized second encoder, the input image of size H×W remains at its original size, resulting in a visual token2 of size H / 16×W / 16×1024. Visual token2 is then resized to (H / 16×W / 16)×1024 and finally concatenated into a target visual token of size (257+H / 16×W / 16)×1024. In this way, the model can support user interface images of various sizes, meeting the needs of real-world scenarios.

[0036] When the target visual feature is a combination of the first visual feature and the second visual feature through splicing or other methods, the target visual feature can be sent to the large language model, which will then generate front-end code based on the target visual feature and the user's instruction text.

[0037] When the target visual features are the first visual features and the second visual features, the first visual features and the second visual features can be combined by the large language model or other modules through splicing to form new visual features, and then the front-end code can be generated by the large language model. Alternatively, the front-end code can be generated directly by the large language model based on the first visual features and the second visual features.

[0038] In this embodiment, visual features include visual attribute information characterizing user interface components. For example, visual attribute information includes the component layout, component category, and text within the component; it also includes icons, colors, and states within the component. This embodiment allows for more complete identification of component visual attributes, reducing the error rate in front-end code.

[0039] S103 generates front-end code based on target visual features and user command text using a large language model.

[0040] In this step, the large language model 202 generates front-end code based on the target visual features and user command text. The target visual features and user command text can be input into the large language model 202, which then generates the front-end code.

[0041] In this embodiment, user interface image input of any resolution can be supported, and front-end code is generated based on the user interface image and user command text, thereby improving the accuracy of front-end code generation and reducing the error rate.

[0042] In this embodiment, corresponding front-end code can be generated end-to-end based on user interface images and user command text, avoiding information loss caused by multiple models being linked together, resulting in higher accuracy and lower error rate of the generated front-end code.

[0043] In this embodiment, the large language model 202 and the visual encoder 201 are combined to form a multimodal model 203. In practical use, user interface images and user command text can be input into the multimodal model 203, which then directly outputs front-end code. Users can also continue to input new commands based on the generated code, allowing the model to further adjust and optimize the generated code.

[0044] In one embodiment, the large language model 202 is trained, see [reference needed]. Figure 2 and Figure 3 Training large language models includes: S301, based on the first sample data, pre-train a multimodal model including a visual encoder and a large language model, wherein the first sample data includes a first user interface image and the corresponding first target front-end code.

[0045] In this step, the first sample data is used to pre-train the multimodal model 203. The visual encoder 201 can be jointly trained with the large language model 202, or the parameters of the large language model 202 can be trained only with the parameters of the visual encoder 201 fixed.

[0046] Furthermore, the first sample data may also include the first user instruction text. For example, by inputting a UI design draft, the corresponding Element front-end code can be generated.

[0047] S302, based on the second sample data, fine-tune the training of the multimodal model. The second sample data includes the second user interface image, the second user instruction text, the first initial front-end code, and the corresponding second target front-end code.

[0048] In this step, the second user instruction text guides the fine-tuning training of the multimodal model 203, containing modification or generation requirements. For example: "Generate the front-end component code for xx based on the user interface image, where the dropdown component's options are xx." Here, the first initial front-end code is the corresponding original code, and the second target front-end code is the target front-end code obtained by adjusting the first initial front-end code according to the content of the second user instruction text.

[0049] In this embodiment, model training can employ an autoregressive approach, i.e., pre-training followed by fine-tuning. This method enables the multimodal model 203, composed of the visual encoder 201 and the large language model 202, to not only generate accurate front-end code but also to further adjust it based on user feedback, thereby generating front-end code that better meets the requirements.

[0050] For example, multimodal instruction fine-tuning data may include second user instruction text in the following forms: "Generate code for the xx front-end component library based on the UI design," "Generate code corresponding to the new version of the UI design based on the previous version of the UI design and the previous version of the code," "Generate code for the xx front-end component based on the UI design, where the dropdown component's option is xx." During training, the model can be trained based on the above second user instruction text and its corresponding second user interface image, the first initial front-end code, and the second target front-end code.

[0051] In one implementation, training a large language model further includes: The multimodal model is fine-tuned based on the third sample, which includes the third user instruction text, the second initial front-end code, and the corresponding third target front-end code.

[0052] In this implementation, the third user instruction text is used to instruct the multimodal model to modify or supplement the existing front-end code. For example, the code instruction fine-tuning data may include third user instruction text in the form of: "Add component xxx to the front-end code," or "Modify the dropdown options in the front-end code to xx." During training, fine-tuning can be performed based on the third user instruction text, the corresponding second initial front-end code, and the third target front-end code, thereby optimizing the model's performance in code modification and supplementation tasks.

[0053] In one implementation, training a large language model further includes: The multimodal model is pre-trained based on the fourth sample data, which includes images and corresponding image descriptions, or images of Chinese and English documents and corresponding text content.

[0054] When the fourth sample data includes "image and corresponding image description", the visual encoder 201 is used to extract the visual features of the image and combine them with the image description to train the large language model 202, so that the large language model 202 can better recognize the semantics of the image, thereby improving the accuracy of front-end code generation.

[0055] When the fourth sample data includes "Chinese and English document images and corresponding text content", this data can be Optical Character Recognition (OCR) type data. During training, the visual encoder 201 extracts the visual features of the Chinese and English document images and combines them with the corresponding text content to train the large language model 202, enabling the large language model 202 to better recognize and understand the Chinese and English content in the images, thereby further improving the accuracy of front-end code generation.

[0056] In one implementation, training a large language model further includes: The large language model is pre-trained based on the fifth sample data, which includes front-end code or Chinese and English text.

[0057] When pre-training based on the fifth sample data, the visual encoder 201 is not required; instead, the large language model 202 is trained directly. When the fifth sample data includes front-end code, training can enhance the ability of the large language model 202 to understand and generate front-end code, thereby improving the accuracy of generating front-end code. When the fifth sample data includes Chinese and English text, training can enhance the ability of the large language model 202 to recognize and understand Chinese and English text, thereby improving its multilingual processing capabilities.

[0058] In this embodiment of the application, the trained multimodal model 203 can be integrated into the front-end development process, for example, deployed as a Visual Studio Code plugin, thereby achieving automated code generation and improving development efficiency.

[0059] In one implementation, see Figure 4 Obtain the first sample data, including: S401, the first prompt word for building the front-end interface description of the documentation.

[0060] In this step, "documentation" refers to the documentation for the front-end UI components. To construct the first prompt, you can prepare a list of component names from the component library, their corresponding documentation, and sample code. Randomly select one or more components from the list of component names, and based on the selected components and their documentation, generate a first prompt that provides a documented description of the front-end UI (e.g., a webpage description).

[0061] For example, in one example, the first prompt is: haloe3 is a front - end component library based on Vue3; The following is the component library documentation of haloe3: [Component Name 1]: [Documentation of Component Name 1]; [Component Name 2]: [Documentation of Component Name 2]; Based on these haloe3 components: <haloe3 component name list>, diverse website layout ideas are generated, and each idea has unique design elements; Example: On the login page of a car company website, in the middle is [input_input box], below the [input_input box] is an [alert_warning prompt] indicating the specifications of the username and password, below that is a blue submit [button_button], and at the bottom is a [checkbox_checkbox] with disclaimer and privacy terms.

[0062] S402, according to the first prompt, use the first large - language model to generate a front - end interface description.

[0063] In this step, the first large - language model can be called to generate a front - end interface description. The first large - language model can be an open - source large model (such as the LLaMA large model), or a commercial "Application Programming Interface (API) (such as ChatGPT), and this step does not make a limitation.

[0064] S403, based on the front - end interface description, construct a second prompt for generating code for the front - end interface description.

[0065] In this step, the second prompt can be constructed by combining the front - end interface description, the documentation of relevant components, and the example code, for guiding code generation.

[0066] For example, in one example, the second prompt is: You are now a very excellent programmer for writing front - end code. haloe3 is a front - end component library based on Vue3.

[0067] Now you need to write code according to the haloe3 front - end component library documentation, example code, and user requirements; haloe3 front - end component library documentation and example code: [Component Name 1]: [Documentation of Component Name 1]; [Example code of Component Name 1]; [Component Name 2]: [Documentation of Component Name 2]; [Component Name 2 Sample Code].

[0068] S404: Based on the second prompt word, the second major language model is used to generate the first target front-end code.

[0069] In this step, a second large language model can be invoked to generate the front-end code for the first target. The second large language model can be an open-source large model (such as the DeepSeek large model) or a commercial API (such as GPT-4o).

[0070] S405, Generate the first user interface image based on the first target front-end code.

[0071] In this step, the first target front-end code can be rendered as an image to obtain a first user interface image. The first user interface image and the corresponding first target front-end code together constitute the first sample data. Obtaining the first sample data based on this embodiment not only improves the efficiency of sample data acquisition but also ensures the accuracy and standardization of the sample data, making it more suitable for model training.

[0072] See Figure 5 A front-end code generation device, comprising: The acquisition module 501 is used to acquire user interface images and user command text; The feature generation module 502 is used to generate target visual features based on the user interface image through a visual encoder. The target visual features include a first visual feature and a second visual feature, or a combination of the first visual feature and the second visual feature. The first visual feature is a visual feature obtained by performing visual encoding processing on the user interface image based on a fixed resolution, and the second visual feature is a visual feature obtained by performing visual encoding processing on the user interface image based on a dynamic resolution. The code generation module 503 is used to generate front-end code based on target visual features and user instruction text using a large language model.

[0073] In one implementation, the first visual feature is a fixed-size visual feature, and the second visual feature is a dynamic-size visual feature.

[0074] In one implementation, the target visual features are obtained by stitching together the first visual features and the second visual features by a visual encoder.

[0075] In one implementation, visual features include visual attribute information used to characterize user interface components.

[0076] In one implementation, visual attribute information includes the component layout, component category, and text within the user interface components.

[0077] In one implementation, visual attribute information includes the component's icon, component's color, and component state.

[0078] In one implementation, the front-end code generation device includes a training module, on which a large language model is trained; The training module is used to train large language models; Training a large language model includes: Based on the first sample data, a multimodal model including a visual encoder and a large language model is pre-trained, wherein the first sample data includes a first user interface image and the corresponding first target front-end code; Based on the second sample data, the multimodal model is fine-tuned and trained. The second sample data includes a second user interface image, a second user instruction text, a first initial front-end code, and a corresponding second target front-end code.

[0079] In one implementation, the training module is further configured to: The multimodal model is fine-tuned based on the third sample, which includes the third user instruction text, the second initial front-end code, and the corresponding third target front-end code.

[0080] In one implementation, the training module is further configured to: The multimodal model is pre-trained based on the fourth sample data, which includes images and corresponding image descriptions, or Chinese and English document images and corresponding text content.

[0081] In one implementation, the training module is further configured to acquire first sample data: Obtain the first sample data, including: The first prompt word for generating the front-end interface description from the documentation; Based on the first prompt word, use the first major language model to generate a front-end interface description; Based on the front-end interface description, construct a second prompt word for the code generated from the front-end interface description; Based on the second prompt word, the second major language model is used to generate the first target front-end code; Generate the first user interface image based on the first target front-end code.

[0082] An exemplary embodiment of this application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform a method according to an embodiment of this application.

[0083] Exemplary embodiments of this application also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this application.

[0084] refer to Figure 6 An exemplary embodiment of this application also provides a computer program product 600, including a computer program 601, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this application.

[0085] refer to Figure 7 The present invention describes a structural block diagram of an electronic device 700 that can serve as a server or client of this application, which is an example of a hardware device that can be applied to various aspects of this application. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0086] Electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. The RAM 703 may also store various programs and data required for device operation. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0087] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, output unit 707, storage unit 708, and communication unit 709. Input unit 706 can be any type of device capable of inputting information to electronic device 700. Input unit 706 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 707 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 708 may include, but is not limited to, disk and optical disk. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0088] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the methods of the embodiments of this application can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. In some embodiments, the computing unit 701 can be configured to perform the methods of the embodiments of this application by any other suitable means (e.g., by means of firmware).

[0089] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0090] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” and “having” are open-ended terms meaning “including but not limited to” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to” and is used interchangeably with it.

[0091] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0092] It should also be noted that in the system and method of this application, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of this application.

[0093] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0094] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0095] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A front-end code generation method, characterized in that, include: Obtain user interface images and user command text; Based on the user interface image, a target visual feature is generated using a visual encoder. The target visual feature includes a first visual feature and a second visual feature. The first visual feature is a visual feature obtained by performing visual encoding processing on the user interface image based on a fixed resolution, and the second visual feature is a visual feature obtained by performing visual encoding processing on the user interface image based on a dynamic resolution. The front-end code is generated using a large language model based on the target visual features and the user command text.

2. The method according to claim 1, characterized in that, The first visual feature is a fixed-size visual feature, and the second visual feature is a dynamic-size visual feature.

3. The method according to claim 1, characterized in that, The target visual features are obtained by the visual encoder by splicing the first visual features and the second visual features.

4. The method according to claim 1, characterized in that, The method includes training the large language model; Training the large language model includes: Based on the first sample data, a multimodal model including the visual encoder and the large language model is pre-trained, wherein the first sample data includes a first user interface image and the corresponding first target front-end code; The multimodal model is fine-tuned based on the second sample data, which includes a second user interface image, a second user instruction text, a first initial front-end code, and a corresponding second target front-end code.

5. The method according to claim 4, characterized in that, Training the large language model further includes: The multimodal model is fine-tuned based on the third sample, which includes the third user instruction text, the second initial front-end code, and the corresponding third target front-end code. The multimodal model is pre-trained based on the fourth sample data, wherein the fourth sample data includes images and corresponding image descriptions, or the fourth sample data includes Chinese and English document images and corresponding text content.

6. The method according to claim 4, characterized in that, Obtaining the first sample data includes: The first prompt word for generating the front-end interface description from the documentation; Based on the first prompt word, the first major language model is used to generate a front-end interface description; Based on the aforementioned front-end interface description, a second prompt word is constructed to generate the front-end interface description code. Based on the second prompt word, the first target front-end code is generated using the second major language model; The first user interface image is generated based on the first target front-end code.

7. A front-end code generation device, characterized in that, include: The acquisition module is used to acquire user interface images and user command text; The feature generation module is used to generate target visual features based on the user interface image through a visual encoder. The target visual features include a first visual feature and a second visual feature. The first visual feature is a visual feature obtained by performing visual encoding processing on the user interface image based on a fixed resolution, and the second visual feature is a visual feature obtained by performing visual encoding processing on the user interface image based on a dynamic resolution. The code generation module is used to generate the front-end code based on the target visual features and the user instruction text using a large language model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the method described in any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the method described in any one of claims 1 to 6.

Images