Front-end page code generation method and device based on basic large language model

Abstract

The embodiment of the invention discloses a front-end page code generation method and device based on a basic large language model.The method comprises the steps that a front-end page design drawing is obtained, a front-end code generation type model is adopted for conducting code generation processing on the front-end page design drawing, and the front-end page design drawing is obtained; the front-end code generation type model is obtained by performing migration training on a basic large language model based on a front-end visual prompt generation structure.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, storage medium and electronic device for generating front-end page code based on a basic large language model. Background Art

[0002] With the development of Internet technology, the application of the Internet is becoming more and more extensive, and more and more users are browsing the web through the Internet. The web pages browsed by users are the front end of the website system. General web page development can be called front-end development.

[0003] With the increasing demand for client and mobile web pages, the demand for front-end page development is becoming increasingly heavy. While ensuring quality, a large number of coding tasks need to be completed in a short period of time. Usually, front-end development is mainly carried out by manual coding. Manual coding usually requires multiple iterations during the front-end page development process. Continuous communication and collaboration between designers and engineers is required, which takes a lot of time and energy, resulting in slow development speed and low efficiency, and it is impossible to complete a large number of coding tasks in a short period of time. Summary of the invention

[0004] The present application provides a method, device, storage medium and electronic device for generating a front-end page code. The technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for generating front-end page code based on a basic large language model, the method comprising:

[0006] Get the front-end page design;

[0007] A front-end code generation model is used to perform code generation processing on the front-end page design drawing to obtain the front-end page code for the front-end page design drawing. The front-end code generation model is obtained by performing migration training on the basic large language model based on the front-end visual prompt generation structure.

[0008] The front-end visual prompt generation structure includes a front-end prompt generation module, a front-end dimensional transformation linear module and a large language generation module based on a basic large language model.

[0009] Optionally, the front-end page design drawing is subjected to code generation processing using a front-end code generation model to obtain a front-end page code for the front-end page design drawing, including:

[0010] The front-end page design drawing is input into the target generative model, the front-end page design drawing is subjected to visual prompt encoding processing by the front-end prompt generation module to obtain front-end soft prompt data, the front-end soft prompt data is subjected to multimodal dimensional transformation by the front-end dimensional transformation linear module to obtain front-end soft prompt input features for the large language generative module, the large language generative module is controlled to perform code generation processing on the front-end soft prompt input features to obtain the front-end page code for the front-end page design drawing, and the front-end page code is output.

[0011] Optionally, the method further includes:

[0012] Create an initial front-end code generation model for the base large language model based on the front-end visual cue generation structure;

[0013] Get sample training data;

[0014] The sample training data is used to perform front-end code generation migration training on the initial front-end code generation model to obtain a trained front-end code generation model.

[0015] Optionally, the generating structure based on the front-end visual prompts creates an initial front-end code generation model for the basic large language model, including:

[0016] Get the basic large language model;

[0017] Based on the front-end visual cue generation structure and the basic large language model, an initial front-end code generation model is created, which includes at least a front-end cue generation module, a front-end dimensional transformation linear module and a large language generation module.

[0018] Optionally, the using the sample training data to perform front-end code generation migration training on the initial front-end code generation model to obtain a trained front-end code generation model includes:

[0019] Obtaining sample task training data corresponding to at least one training stage task;

[0020] In the training task stage, the sample task training data is used to perform front-end code generation migration training on the initial front-end code generation model until the initial front-end code generation model completes all the training task stages, thereby obtaining a trained front-end code generation model.

[0021] Optionally, the training task phase includes a large language module instruction optimization phase, a model migration warm-up training phase, a model migration adjustment training phase, and a visual instruction tuning phase, and the initial front-end code generation model includes a front-end prompt generation module, a front-end dimensional transformation linear module, and a large language generation module.

[0022] The using the sample task training data to perform front-end code generation migration training on the initial front-end code generation model includes:

[0023] In the large language model instruction optimization stage, the first sample task training data is input into the large language generation module to perform instruction generation code training to obtain an instruction generation result, and the large language generation module is fine-tuned based on the instruction generation result to obtain the large language generation module after parameter fine-tuning;

[0024] In the model migration warm-up training stage, the word vector converter from the front-end prompt generation module to the large language generation module is trained using the second sample task training data, the front-end dimensional transformation linear module is initialized based on the word vector converter, and the initial front-end code generation model is pre-trained for model migration using the second sample task training data to obtain the initial front-end code generation model after model migration pre-training;

[0025] In the model migration adjustment training stage, the front-end prompt generation module and the front-end dimensional transformation linear module are subjected to model migration training using the third sample task training data to obtain the front-end prompt generation module and the front-end dimensional transformation linear module after the model migration training;

[0026] In the visual instruction tuning stage, the fourth sample task training data is used to perform front-end code generation training on the initial front-end code generation model to obtain a front-end code generation result, and the model parameters of the initial front-end code generation model are fine-tuned based on the front-end code generation result to obtain a front-end code generation model.

[0027] Optionally, the using the second sample task training data to perform model migration pre-training on the initial front-end code generation model to obtain the initial front-end code generation model after model migration pre-training includes:

[0028] The second sample task training data is used to perform model migration pre-training on the front-end dimensional transformation linear module in the initial front-end code generation model, and the model parameters of the front-end prompt generation module and the large language generation module are controlled to remain unchanged, so as to obtain the front-end dimensional transformation linear module after model migration pre-training.

[0029] Optionally, fine-tuning model parameters of the initial front-end code generation model based on the front-end code generation result includes:

[0030] Based on the front-end code generation result, model parameters of the front-end prompt generation module and the front-end dimensional transformation linear module in the initial front-end code generation model are fine-tuned, and the model parameters of the large language generation module are controlled to remain unchanged.

[0031] Optionally, before the front-end code generation model is trained by the fourth sample task training data to obtain the front-end code generation result, the method further includes:

[0032] Acquire source sample task training data consisting of sample front-end page design drawings, perform graph specification normalization processing on the source sample task training data, and obtain the source sample task training data after graph specification normalization processing

[0033] The source sample task training data is processed by prompt word annotation to obtain fourth sample task training data of the front-end graph code pair type, where the front-end graph code pair type is the type corresponding to the sample front-end page design drawing and the prompt word corresponding to the sample front-end page design drawing.

[0034] In a second aspect, an embodiment of the present application provides a front-end page code generation device based on a basic large language model, the device comprising:

[0035] Image acquisition module, used to obtain the front-end page design image;

[0036] The code generation module is used to use a front-end code generation model to perform code generation processing on the front-end page design drawing to obtain the front-end page code for the front-end page design drawing. The front-end code generation model is obtained by performing migration training on the basic large language model based on the front-end visual prompt generation structure.

[0037] Optionally, the front-end visual prompt generation structure includes a front-end prompt generation module, a front-end dimensional transformation linear module and a large language generation module based on a basic large language model, and the code generation module is used to:

[0038] The front-end page design drawing is input into the target generative model, the front-end page design drawing is subjected to visual prompt encoding processing by the front-end prompt generation module to obtain front-end soft prompt data, the front-end soft prompt data is subjected to multimodal dimensional transformation by the front-end dimensional transformation linear module to obtain front-end soft prompt input features for the large language generative module, the large language generative module is controlled to perform code generation processing on the front-end soft prompt input features to obtain the front-end page code for the front-end page design drawing, and the front-end page code is output.

[0039] Optionally, the device further comprises:

[0040] A model building module for creating an initial front-end code generation model for a base large language model based on the front-end visual cue generation structure;

[0041] A data acquisition module, used to acquire sample training data;

[0042] The model training module is used to use the sample training data to perform front-end code generation migration training on the initial front-end code generation model to obtain a trained front-end code generation model.

[0043] Optionally, the model training module is used to:

[0044] Get the basic large language model;

[0045] Based on the front-end visual cue generation structure and the basic large language model, an initial front-end code generation model is created, which includes at least a front-end cue generation module, a front-end dimensional transformation linear module and a large language generation module.

[0046] Optional, model training module for:

[0047] Obtaining sample task training data corresponding to at least one training stage task;

[0048] In the training task stage, the sample task training data is used to perform front-end code generation migration training on the initial front-end code generation model until the initial front-end code generation model completes all the training task stages, thereby obtaining a trained front-end code generation model.

[0049] Optionally, the training task phase includes a large language module instruction optimization phase, a model migration warm-up training phase, a model migration adjustment training phase, and a visual instruction tuning phase, and the initial front-end code generation model includes a front-end prompt generation module, a front-end dimensional transformation linear module, and a large language generation module.

[0050] The model training module is used to:

[0051] In the large language model instruction optimization stage, the first sample task training data is input into the large language generation module to perform instruction generation code training to obtain an instruction generation result, and the large language generation module is fine-tuned based on the instruction generation result to obtain the large language generation module after parameter fine-tuning;

[0052] In the model migration warm-up training stage, the word vector converter from the front-end prompt generation module to the large language generation module is trained using the second sample task training data, the front-end dimensional transformation linear module is initialized based on the word vector converter, and the initial front-end code generation model is pre-trained for model migration using the second sample task training data to obtain the initial front-end code generation model after model migration pre-training;

[0053] In the model migration adjustment training stage, the front-end prompt generation module and the front-end dimensional transformation linear module are subjected to model migration training using the third sample task training data to obtain the front-end prompt generation module and the front-end dimensional transformation linear module after the model migration training;

[0054] In the visual instruction tuning stage, the fourth sample task training data is used to perform front-end code generation training on the initial front-end code generation model to obtain a front-end code generation result, and the model parameters of the initial front-end code generation model are fine-tuned based on the front-end code generation result to obtain a front-end code generation model.

[0055] Optionally, the model training module is used to: use the second sample task training data to perform model migration pre-training on the front-end dimensional transformation linear module in the initial front-end code generation model, and control the model parameters of the front-end prompt generation module and the large language generation module to remain unchanged, so as to obtain the front-end dimensional transformation linear module after model migration pre-training.

[0056] Optionally, the model training module is used to: fine-tune the model parameters of the front-end prompt generation module in the initial front-end code generation model based on the front-end code generation result, and control the model parameters of the front-end dimensional transformation linear module and the large language generation module to remain unchanged.

[0057] Optionally, the data acquisition module is used to:

[0058] Acquire source sample task training data consisting of sample front-end page design drawings, perform graph specification normalization processing on the source sample task training data, and obtain the source sample task training data after graph specification normalization processing

[0059] The source sample task training data is processed by prompt word annotation to obtain fourth sample task training data of the front-end graph code pair type, where the front-end graph code pair type is the type corresponding to the sample front-end page design drawing and the prompt word corresponding to the sample front-end page design drawing.

[0060] In a third aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0061] In a fourth aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0062] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:

[0063] In one or more embodiments of the present application, an electronic device obtains a front-end page design drawing, and uses a front-end code generation model to perform code generation processing on the front-end page design drawing to obtain a front-end page code for the front-end page design drawing. The front-end code generation model is obtained after transfer training of a basic large language model based on a front-end visual cue generation structure. Using the front-end code generation model after transfer training can avoid spending a lot of time and effort, improve the efficiency of front-end design and development, and reduce front-end development costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 It is a flowchart of a method for generating front-end page code provided in an embodiment of the present application;

[0066] Figure 2 It is a schematic diagram of a front-end page design diagram provided in an embodiment of the present application;

[0067] Figure 3 It is a flowchart of another method for generating front-end page code provided in an embodiment of the present application;

[0068] Figure 4 It is a schematic diagram of the model structure of a front-end code generation model provided in an embodiment of the present application;

[0069] Figure 5 It is a schematic diagram of a model training provided in an embodiment of the present application;

[0070] Figure 6 It is a model effect verification diagram of a front-end code generation model provided in an embodiment of the present application;

[0071] Figure 7 It is a structural schematic diagram of a front-end page code generation device provided in an embodiment of the present application;

[0072] Figure 8 It is a structural schematic diagram of another front-end page code generation device provided in an embodiment of the present application;

[0073] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0074] Fig.10It is a schematic diagram of the structure of the operating system and user space provided in the embodiment of the present application;

[0075] Fig.11 yes Fig.10 The architecture diagram of the Android operating system;

[0076] Fig.12 yes Fig.10 Architecture diagram of the IOS operating system. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0078] In the description of the present application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present application, it should be noted that, unless otherwise clearly specified and limited, "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood in specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are an "or" relationship.

[0079] The present application is described in detail below with reference to specific embodiments.

[0080] In one embodiment, Figure 1As shown, a front-end page code generation method based on a basic large language model is proposed, which can be implemented by a computer program and can be run on a front-end page code generation device based on the von Neumann system. The computer program can be integrated in an application or run as an independent tool application. The front-end page code generation device can be an electronic device, including but not limited to: a server, a personal computer, a tablet computer, a handheld device, a vehicle-mounted device, a wearable device, a computing device or other processing device connected to a wireless modem. In different networks, terminal devices can be called different names, such as: user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user device, cellular phone, cordless phone, 5G network or electronic device in future evolution network, etc.

[0081] Specifically, the front-end page code generation method includes:

[0082] S102: Obtaining the front-end page design drawing;

[0083] Front-end refers to "client-side development" and is the opposite of back-end, or "server-side development". Front-end code mainly runs on the client side. It implements the display pages of user-visible applications and web pages (collectively referred to as applications in this application), and needs to consider user experience, including interface layout, interactive effects, page loading speed, etc.

[0084] Specifically, the front-end page design drawing can be understood as a front-end design style picture, a front-end design style image, a front-end design style picture collection (such as a front-end design style video). It can be a front-end style sketch hand-drawn by a front-end design user, or it can be other pictures containing front-end style.

[0085] like Figure 2 As shown, Figure 2 It is a schematic diagram of the front-end page design. Figure 2 The front-end page design can be as follows Figure 2 According to the front-end design user's hand-drawn front-end style sketch shown, for example, the electronic device provides Figure 2 The dialog window between the user and the front-end code generation model is shown in the figure. The front-end design user enters the following Figure 2 The front-end style sketch in the figure carries the prompt word "Can you write the html code for this webpagepicture?" input by the user. At this time, the electronic device obtains the following information: Figure 2A front-end style sketch is drawn, and then a front-end code generation model is used to perform code generation processing on the front-end page design drawing to obtain the front-end page code for the front-end page design drawing.

[0086] Exemplarily, a front-end design user manually draws a front-end style on a drawing software, and then saves it as an image in an image format such as jpg or png. The image is the front-end page design drawing, and the electronic device generates the subsequent front-end page code by obtaining the front-end page design drawing.

[0087] S104: Using a front-end code generation model to perform code generation processing on the front-end page design drawing to obtain the front-end page code for the front-end page design drawing, the front-end code generation model is obtained by performing transfer training on a basic large language model based on a front-end visual prompt generation structure.

[0088] In this specification, a front-end code generation model is pre-trained, and the front-end code generation model is obtained after migration training of the basic large language model based on the front-end visual prompt generation structure. The basic large language model can be quickly applied to a new front-end code automatic generation field without retraining a new model. Only fine-tuning training of the basic large language model is required. Usually, the basic large language model is adapted to the text character understanding processing scenario and cannot be directly adapted to the visual understanding processing scenario. Based on this, with the help of the front-end visual prompt generation structure, the basic large language model (which can be called LLM model) can be migrated and converted into a multimodal large-scale language model (which can be called MLLM) compatible with multimodal scenarios such as text and images. The multimodal large-scale language model can generate front-end code, using the front-end page design drawing as the model input. The multimodal large-scale language model obtained by migration training can be called a front-end code generation model. The front-end code generation model automatically generates front-end code based on the front-end page design drawing to obtain the front-end page code for the front-end page design drawing, thereby assisting design developers in designing and coding front-end web pages and reducing web page development and design costs.

[0089] The front-end code generation model is obtained by performing migration training on the basic large language model based on the front-end visual prompt generation structure. The basic large language model may be a Wenxinyiyan model, an LLM model, etc. The type of the basic large language model is not limited in this specification.

[0090] In a feasible implementation, the front-end visual prompt generation structure includes a front-end prompt generation module, a front-end dimensional transformation linear module, and a large language generation module based on a basic large language model.

[0091] The process of using the front-end code generation model to perform code generation processing on the front-end page design diagram to obtain the front-end page code for the front-end page design diagram may be as follows:

[0092] The electronic device inputs the front-end page design drawing into the target generative model, the model structure of the target generative model is also the front-end visual prompt generation structure, the front-end prompt generation module performs visual prompt encoding processing on the front-end page design drawing to obtain front-end soft prompt data, the front-end dimensional transformation linear module performs multimodal dimensional transformation on the front-end soft prompt data to obtain the front-end soft prompt input features for the large language generative module, the large language generative module is controlled to perform code generation processing on the front-end soft prompt input features to obtain the front-end page code for the front-end page design drawing, and the front-end page code is output.

[0093] Exemplarily, the front-end prompt generation module (which may be referred to as the front-end VPG) takes the front-end page design drawing as input, and encodes the visual data input such as the front-end page design drawing into the front-end soft prompt data of fixed length. Then, the front-end dimension transformation linear module (which may be referred to as the front-end Projector) is used to align the dimension of the front-end soft prompt data with the word embedding dimension of the large language generation module to obtain the front-end soft prompt input feature (at this time, the conversion from visual image features to character input features is completed). Finally, the large language generation module is controlled to perform code generation processing on the front-end soft prompt input feature. The large language generation module LLM based on the basic large language model will generate the front-end code according to the front-end soft prompt input feature transmitted from the soft prompt. At this time, the front-end page code for the front-end page design drawing is obtained.

[0094] In one or more embodiments of the present specification, an electronic device obtains a front-end page design drawing, and uses a front-end code generation model to perform code generation processing on the front-end page design drawing to obtain the front-end page code for the front-end page design drawing. The front-end code generation model is based on the front-end visual prompt generation structure and performs migration training on a basic large language model. This can avoid spending a lot of time and effort, improve the efficiency of front-end design and development, and reduce front-end development costs.

[0095] See also Figure 3 , Figure 3 This is a flow chart of another embodiment of a front-end page code generation method proposed in this application. Specifically:

[0096] S202: Creating an initial front-end code generation model for a basic large language model based on a front-end visual cue generation structure;

[0097] For example, Figure 4 As shown, Figure 4The present invention is a schematic diagram of the model structure of a front-end code generation model. The model structure of the (initial) front-end code generation model adopts a front-end visual prompt generation structure, which is composed of at least a front-end prompt generation module, a front-end dimensional transformation linear module and a large language generation module. The front-end visual prompt generation structure in this specification is based on a general basic large language model. A front-end prompt generation module and a front-end dimensional transformation linear module for visual dimensional transformation are trained on the basic large language model.

[0098] The front-end prompt generation module in the front-end visual prompt generation structure is used to bridge the mode plug-in between the front-end image vision and the basic large language model in the front-end page code generation scenario. The front-end visual prompt generation structure is introduced to convert the front-end vision (such as the front-end design drawing) into soft prompt data (soft prompt) representing the front-end features. After processing, the soft prompt data representing the front-end features can be used for subsequent generation and processing by the large language generation module corresponding to the basic large language model.

[0099] The front-end dimensionality transformation linear module in the front-end visual prompt generation structure is used to perform visual dimensionality transformation, aligning the dimension of the front-end soft prompt data with the word embedding dimension of the large language generation module to obtain the front-end soft prompt input feature (at this time, the conversion from visual image features to character input features is completed).

[0100] In a feasible implementation, the electronic device first obtains a basic large language model, and then creates an initial front-end code generation model including at least a front-end prompt generation module, a front-end dimensional transformation linear module and a large language generation module based on the front-end visual prompt generation structure and the basic large language model.

[0101] The basic large language model may be a Vicuna-7B model or the like.

[0102] S204: Obtain sample training data;

[0103] The sample training data is determined based on the training phase tasks, which may include but are not limited to using the CodeAlpaca dataset, COCO dataset, Laion-COCO dataset, Pix2code, etc.

[0104] Exemplarily, the front-end code generation model can be divided into multiple training stage tasks, and different training stage tasks can use different or the same sample training data, which is determined based on the actual application environment.

[0105] Optionally, in some embodiments, the multiple training stage tasks divided by the front-end code generation model can be a large language model instruction optimization stage, a model migration warm-up training stage, a model migration adjustment training stage, a visual instruction tuning stage, etc.

[0106] The sample training data used in the large language model instruction optimization stage can be called the first sample task training data;

[0107] The sample training data used in the model transfer warm-up training phase can be called the second sample task training data;

[0108] The sample training data used in the model transfer adjustment training phase can be called the third sample task training data;

[0109] The sample training data used in the visual command tuning stage can be referred to as the fourth sample task training data;

[0110] S206: Performing front-end code generation migration training on the initial front-end code generation model using the sample training data to obtain a trained front-end code generation model.

[0111] Exemplarily, the model training technology in the related art can be used to perform front-end code generation migration training on the initial front-end code generation model using sample training data, so as to obtain a trained front-end code generation model.

[0112] For example, Figure 5 As shown, Figure 5 This is a schematic diagram of model training, wherein the sample training data is used to perform front-end code generation migration training on the initial front-end code generation model to obtain a trained front-end code generation model, which may include the following steps, specifically:

[0113] S3002: Obtain sample task training data corresponding to at least one training phase task;

[0114] In the training task stage, as shown in Table 1 below, it can be divided into 4 training stage tasks. The sample task training data used by the training stage tasks is used to perform front-end code generation migration training on the initial front-end code generation model until the initial front-end code generation model completes all training task stages to obtain a trained front-end code generation model.

[0115] Table 1

[0116]

[0117] S3004: In the large language model instruction optimization stage, the first sample task training data is input into the large language generation formula module to perform instruction generation code training to obtain an instruction generation result, and the large language generation formula module is fine-tuned based on the instruction generation result to obtain the large language generation formula module after parameter fine-tuning;

[0118] In the large language model instruction optimization stage: use the first sample task training data to fine-tune the instructions of the language generation module based on the large language model LLM model to improve the code generation capability of the language generation module.

[0119] The first sample task training data is usually training data of the instruction-input-code triple type. The first sample task training data may be the CodeAlpaca dataset. Using the first sample task training data to perform model training in the large language model instruction optimization stage can improve the processing capability of the language generation module in the code generation dimension.

[0120] Furthermore, the CodeAlpaca dataset is a dataset of 20,000 "instruction-input-code" triplets. The following is an instruction describing a task, which provides more contextual input pairs. Sample data in the CodeAlpaca dataset is as follows:

[0121] ###Directive: Generates a code snippet that extracts all URLs in a given string.

[0122] ###Input: This string contains some URLs, such as<https: / / www.google.com> and<https: / / www.facebook.com> .

[0123] ###Code response: import restring="This string contains some URLs, such as<https: / / www.google.com><https: / / www.facebook.com> "urls=re.findall('http[s]?: / / (?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\\\(\\\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+',string)print(ur ls);

[0124] Specifically, in the large language model instruction optimization stage, the electronic device inputs the first sample task training data (such as the instructions and input sample data in the CodeAlpaca data set) into the large language generation module of the initial front-end code generation model to perform instruction generation code training to obtain an instruction generation result. The instruction generation result is actually the code generated by the large language generation module of the initial front-end code generation model according to the instruction and input, and then the module code generation loss is calculated based on the instruction generation result and the code data in the first sample task training data by using the relevant loss function, and the module code generation loss is used to perform parameter fine-tuning processing on the large language generation module until the large language generation module completes the model training in the large language model instruction optimization stage, thereby obtaining the large language generation module after parameter fine-tuning.

[0125] Considering that in the large language model instruction optimization stage, it is impossible to fine-tune the full parameter instructions of the large language generative module for a single model processing part (such as the model processing part based on the A10040GB graphics processor), DeepSpeed ​​technology can be used to split the large language generative module training in the large language model instruction optimization stage to achieve model parallelism. DeepSpeed ​​technology mainly uses the Zero Redundancy Optimizer (Zero) to provide the ZeRO-Offload mode for video memory optimization in the model training stage, for example, it is divided into ZeRO-1, 2, and 3, and can be split to different degrees. In addition, the ZeRO-Offload mode can delegate certain calculation parameters in the training stage to the memory and CPU calculation, saving video memory.

[0126] For example, in the actual large language model instruction optimization stage, n-block model processing parts can be used for training, and the total memory can be 220GB. The batch size parameter per_device_train_batch_size of each GPU for training can be set to 2, the configured gradient accumulation number gradient_accumulation_steps is 8, the configured learning rate learning_rate is 0.00002, and the training round num_train_epochs is 10. By adopting the ZeRO-2 mode and the ZeRO-Offload mode, a part of the calculation parameters are put into the memory and CPU for calculation, which can greatly save the memory usage of the large language model instruction optimization stage.

[0127] S3006: In the model migration warm-up training stage, the word vector converter from the front-end prompt generation module to the large language generation module is trained using the second sample task training data, the front-end dimensional transformation linear module is initialized based on the word vector converter, and the initial front-end code generation model is pre-trained for model migration using the second sample task training data to obtain the initial front-end code generation model after model migration pre-training;

[0128] Exemplarily, the second sample task training data is used to perform model migration pre-training on the front-end dimensional transformation linear module in the initial front-end code generation model, and the model parameters of the front-end prompt generation module and the large language generation module are controlled to remain unchanged, so as to obtain the front-end dimensional transformation linear module after model migration pre-training.

[0129] The model migration warm-up training phase can be expressed as the “model migration warm-up training phase”, which is exemplified as follows:

[0130] When migrating a model, first perform migration warm-up training, which can accelerate the convergence of the front-end code generation model, reduce the number of training rounds, and avoid the performance degradation caused by the new large language generation module. The model migration warm-up training phase is divided into two steps:

[0131] The first step is word vector conversion:

[0132] Word vector conversion is from the source large language model LLM src To the target large language model LLM gt Train a word vector converter, which can be regarded as a linear layer. Then initialize the Projector with the word vector converter. For example, the source word vector model (equivalent to LLM src ) can be opt-6.7b, the target word vector model (equivalent to LLM gt ) can be vicuna-7b (large language module), the multimodal model corresponding to the front-end prompt generation module can be blip2_pretrained_opt6.7b, and the initialized Projector (dimensionality transformation linear layer) is obtained. The training dataset used for word vector conversion can be the coco dataset, sbu dataset, etc.

[0133] In the model training phase, in order to avoid using the randomly initialized dimensionality transformation linear layer Projector to tune the front-end prompt generation module, which may damage the existing fine-grained visual perception ability of the front-end prompt generation module. The front-end prompt generation module is usually a pre-trained model with strong visual perception ability. In this phase, the above limitation can be solved by preheating the Projector before jointly tuning the front-end prompt generation module and the dimensionality transformation linear layer. For preheating the Projector, please refer to the second step "Projector warm-up".

[0134] The second step is to warm up the Projector layer: Warming up the Projector layer can prevent performance degradation and speed up VPG training. With the help of the word embedding converter, the Projector layer of LLMgt is initialized to speed up the warm-up of the Projector layer.

[0135] Optionally, the second step is to perform a dimensional transformation linear layer projector warm-up, which can fix the parameters of the front-end prompt generation module VPG and the large language model LLM, and only train the parameters of the dimensional transformation linear layer Projector. Here, the parameter input, the multimodal model is blip2_pretrained_opt6.7b, the initialized Projector and the large language model LLM model vicuna-7b. That is, the execution of the model migration pre-training of the initial front-end code generation model using the second sample task training data to obtain the initial front-end code generation model after the model migration pre-training can be:

[0136] The second sample task training data is used to perform model migration pre-training on the front-end dimensional transformation linear module in the initial front-end code generation model, and the model parameters of the front-end prompt generation module and the large language generation module are controlled to remain unchanged, so as to obtain the front-end dimensional transformation linear module after model migration pre-training.

[0137] For example, in the second step of dimension transformation linear layer warm-up Projector warm-up, the optimizer can be Adam, the batch size is 16, the learning rate is 0.0005, and the training round is 1. The training datasets are COCO dataset and SBU dataset;

[0138] S3008: In the model migration adjustment training stage, the front-end prompt generation module and the front-end dimensional transformation linear module are subjected to model migration training using the third sample task training data to obtain the front-end prompt generation module and the front-end dimensional transformation linear module after the model migration training;

[0139] Exemplarily, the training goal of the Fine-tuning stage of model migration is to align the front-end prompt generation module to the large language generation module, so that the visual input of the front-end prompt generation module can be controlled to be encoded into fixed-length front-end soft prompt data. Then, the front-end dimension transformation linear layer Projector can align the data character dimension of the front-end soft prompt data with the word embedding dimension of the large language generation module to obtain processable input features, and the subsequent large language generation module can directly generate front-end code based on the input features.

[0140] In the model migration adjustment training phase, considering that only training the front-end dimensional transformation linear module is not enough, which will cause the model to underfit, the model fine-tuning in the model migration Fine-tuning phase simultaneously trains the front-end prompt generation module VPG and the front-end dimensional transformation linear module Projector.

[0141] For example, in the model migration adjustment training phase, the batch size can be configured to 6, the number of gradient accumulations can be configured to 8, the learning rate can be configured to 0.00005, and the number of training rounds can be configured to 10. The training datasets used for the third sample task training data can be COCO datasets, VG datasets, Laion-COCO datasets, etc. When COCO datasets, VG datasets, and Laion-COCO datasets are used at the same time, the corresponding training set ratios are: 0.1, 0.2, and 0.7. The image input size is 224*224.

[0142] S3010: In the visual instruction tuning stage, the fourth sample task training data is used to perform front-end code generation training on the initial front-end code generation model to obtain a front-end code generation result, and based on the front-end code generation result, the model parameters of the initial front-end code generation model are fine-tuned to obtain a front-end code generation model.

[0143] For example, the visual instruction tuning stage can also be called Visual Instruction Tuning. In S3004-S3008, the public training set can be used. In the visual instruction tuning stage, the fourth sample task training data of the front-end graph code pair type (front-end design diagram and prompt word) is introduced.

[0144] The image-code pair dataset of the fourth sample task training data (also called Pix2code training data) is used to fine-tune the visual instructions of the migrated multimodal model. Here, the large language model LLM model is also fixed, and the front-end prompt generation module VPG and the front-end dimensional transformation linear module Projector are trained.

[0145] Exemplarily, the fine-tuning of model parameters of the initial front-end code generation model based on the front-end code generation result may be:

[0146] Based on the front-end code generation result, the model parameters of the front-end prompt generation module and the front-end dimensional transformation linear module in the initial front-end code generation model are fine-tuned, and the model parameters of the large language generation module LLM are controlled to remain unchanged.

[0147] For example, in the visual instruction tuning stage: BATCH_SIZE can be configured to 6, the number of gradient accumulations can be configured to 8, the number of training rounds can be configured to 4, and the learning rate can be configured to 1e-5.

[0148] Exemplarily, the following uses the fourth sample task training data to explain the production process, that is, before the front-end code generation model is trained using the fourth sample task training data to obtain the front-end code generation result, the following steps are also included:

[0149] A2: acquiring source sample task training data consisting of sample front-end page design drawings, and performing graph specification normalization processing on the source sample task training data to obtain the source sample task training data after the graph specification normalization processing;

[0150] The source sample task training data can be understood as the source sample task training data that only includes the sample front-end page design drawings without prompt word annotations. The source sample task training data can also include the front-end text code related to the sample front-end page design drawings.

[0151] The purpose of image specification normalization is to convert sample front-end page design images with different image size specifications into images with uniform specifications.

[0152] For example, assuming that the image size for model training is usually 224*224, and the image size in the sample front-end page design image is not consistent, the image size is first scaled to 224*224 and normalized to obtain the source sample task training data after the image specification normalization processing.

[0153] A4: Perform prompt word annotation processing on the source sample task training data to obtain the fourth sample task training data of the front-end graph code pair type, where the front-end graph code pair type is the type corresponding to the sample front-end page design drawing and the prompt word corresponding to the sample front-end page design drawing.

[0154] The prompt word labeling process can be understood as configuring the prompt word Prompt for the source sample task training data.

[0155] The text data in the source sample task training data is just the front-end code, without any prompts, so prompt words need to be added to it;

[0156] For example, two prompts are shown as follows:

[0157] 1.["The following code is for reference only, and you can modify it according to your needs. The code is as follows:\n", (The following code is for reference only, you can modify it according to your needs)

[0158] 2. "I can write HTML code for this. Here is an example of what the code might look like:\n"]

[0159] In one or more embodiments of the present specification, a trained front-end code generation model can be obtained through training and fine-tuning in the above manner, and a visual prompt generation framework is used to perform model migration. The training cost of the multimodal front-end code generation model is extremely low. After using the front-end code generation model, the corresponding front-end web page code can be automatically generated according to the front-end design sketch, front-end design screenshot, etc., thereby improving the efficiency of front-end design and development and reducing the front-end development cost.

[0160] like Figure 6 As shown, Figure 6 This is a model effect verification diagram of a front-end code generation model. The front-end code generation model is trained using the above method. The electronic device provides Figure 5 The dialog window between the user and the front-end code generation model is shown in the figure. The front-end design user enters the following Figure 5 Front-end page design diagram in ( Figure 5 The front-end page design diagram is indicated by a rectangular box (the content of the diagram is omitted), the front-end page design diagram can be a screenshot of the front-end webpage. When the front-end webpage screenshot is input, the user inputs the prompt word "can you write the source code for this webpage screenshot?" At this time, the electronic device obtains the front-end style sketch, and then uses the front-end code generation model to perform code generation processing on the front-end page design diagram to obtain the front-end page code for the front-end page design diagram. The front-end page code is as follows: Figure 6As shown, it can be seen that after using the front-end code generation model, the corresponding front-end web page code can be automatically generated according to the front-end design screenshots, etc., which improves the efficiency of front-end design and development and reduces the front-end development cost.

[0161] The following will be combined Figure 7 , the front-end page code generation device provided in the embodiment of the present application is introduced in detail. It should be noted that, Figure 7 The front-end page code generation device shown is used to execute this application Figure 1 to Figure 6 For the convenience of explanation, only the part related to the embodiment of the present application is shown. For the specific technical details not disclosed, please refer to the present application. Figure 1 to Figure 6 The embodiment shown.

[0162] See also Figure 7 , which shows a schematic diagram of the structure of the front-end page code generation device of an embodiment of the present application. The front-end page code generation device 1 can be implemented as all or part of the user terminal through software, hardware or a combination of both. According to some embodiments, the front-end page code generation device 1 includes a graph acquisition module 11 and a code generation module 12, which are specifically used to:

[0163] The image acquisition module 11 is used to acquire the front-end page design image;

[0164] The code generation module 12 is used to use a front-end code generation model to perform code generation processing on the front-end page design drawing to obtain the front-end page code for the front-end page design drawing. The front-end code generation model is obtained by performing migration training on the basic large language model based on the front-end visual prompt generation structure.

[0165] Optionally, the front-end visual prompt generation structure includes a front-end prompt generation module, a front-end dimensional transformation linear module and a large language generation module based on a basic large language model, and the code generation module 12 is used to:

[0166] The front-end page design drawing is input into the target generative model, the front-end page design drawing is subjected to visual prompt encoding processing by the front-end prompt generation module to obtain front-end soft prompt data, the front-end soft prompt data is subjected to multimodal dimensional transformation by the front-end dimensional transformation linear module to obtain front-end soft prompt input features for the large language generative module, the large language generative module is controlled to perform code generation processing on the front-end soft prompt input features to obtain the front-end page code for the front-end page design drawing, and the front-end page code is output.

[0167] Optional, such as Figure 8 As shown, the device also includes:

[0168] A model building module 13, for creating an initial front-end code generation model for a basic large language model based on a front-end visual cue generation structure;

[0169] A data acquisition module 14 is used to acquire sample training data;

[0170] The model training module 15 is used to perform front-end code generation migration training on the initial front-end code generation model using the sample training data to obtain a trained front-end code generation model.

[0171] Optionally, the model training module 15 is used to:

[0172] Get the basic large language model;

[0173] Based on the front-end visual cue generation structure and the basic large language model, an initial front-end code generation model is created, which includes at least a front-end cue generation module, a front-end dimensional transformation linear module and a large language generation module.

[0174] Optionally, the model training module 15 is used to:

[0175] Obtaining sample task training data corresponding to at least one training stage task;

[0176] In the training task stage, the sample task training data is used to perform front-end code generation migration training on the initial front-end code generation model until the initial front-end code generation model completes all the training task stages, thereby obtaining a trained front-end code generation model.

[0177] Optionally, the training task phase includes a large language module instruction optimization phase, a model migration warm-up training phase, a model migration adjustment training phase, and a visual instruction tuning phase, and the initial front-end code generation model includes a front-end prompt generation module, a front-end dimensional transformation linear module, and a large language generation module.

[0178] The model training module 15 is used to:

[0179] In the large language model instruction optimization stage, the first sample task training data is input into the large language generation module to perform instruction generation code training to obtain an instruction generation result, and the large language generation module is fine-tuned based on the instruction generation result to obtain the large language generation module after parameter fine-tuning;

[0180] In the model migration warm-up training stage, the word vector converter from the front-end prompt generation module to the large language generation module is trained using the second sample task training data, the front-end dimensional transformation linear module is initialized based on the word vector converter, and the initial front-end code generation model is pre-trained for model migration using the second sample task training data to obtain the initial front-end code generation model after model migration pre-training;

[0181] In the model migration adjustment training stage, the front-end prompt generation module and the front-end dimensional transformation linear module are subjected to model migration training using the third sample task training data to obtain the front-end prompt generation module and the front-end dimensional transformation linear module after the model migration training;

[0182] In the visual instruction tuning stage, the fourth sample task training data is used to perform front-end code generation training on the initial front-end code generation model to obtain a front-end code generation result, and the model parameters of the initial front-end code generation model are fine-tuned based on the front-end code generation result to obtain a front-end code generation model.

[0183] Optionally, the model training module 15 is used to: use the second sample task training data to perform model migration pre-training on the front-end dimensional transformation linear module in the initial front-end code generation model, and control the model parameters of the front-end prompt generation module and the large language generation module to remain unchanged, so as to obtain the front-end dimensional transformation linear module after model migration pre-training.

[0184] Optionally, the model training module 15 is used to: fine-tune the model parameters of the front-end prompt generation module in the initial front-end code generation model based on the front-end code generation result, and control the model parameters of the front-end dimensional transformation linear module and the large language generation module to remain unchanged.

[0185] Optionally, the data acquisition module 15 is used to:

[0186] Acquire source sample task training data consisting of sample front-end page design drawings, perform graph specification normalization processing on the source sample task training data, and obtain the source sample task training data after graph specification normalization processing

[0187] The source sample task training data is processed by prompt word annotation to obtain fourth sample task training data of the front-end graph code pair type, where the front-end graph code pair type is the type corresponding to the sample front-end page design drawing and the prompt word corresponding to the sample front-end page design drawing.

[0188] It should be noted that the front-end page code generation device provided in the above embodiment only uses the division of the above functional modules as an example when executing the front-end page code generation method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the front-end page code generation device provided in the above embodiment and the front-end page code generation method embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0189] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0190] In an embodiment of the present application, an electronic device obtains a front-end page design drawing, and uses a front-end code generation model to perform code generation processing on the front-end page design drawing to obtain a front-end page code for the front-end page design drawing. The front-end code generation model is obtained after transfer training of a basic large language model based on a front-end visual cue generation structure. Using the front-end code generation model after transfer training can avoid spending a lot of time and energy, improve the efficiency of front-end design and development, and reduce front-end development costs.

[0191] The present application also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded and executed by a processor as described above. Figure 1 to Figure 6 The front-end page code generation method of the embodiment shown in the figure can be found in the specific execution process. Figure 1 to Figure 6 The specific description of the illustrated embodiment will not be repeated here.

[0192] The present application also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor as described above. Figure 1 to Figure 6 The front-end page code generation method of the embodiment shown in the figure can be found in the specific execution process. Figure 1 to Figure 6 The specific description of the illustrated embodiment will not be repeated here.

[0193] Please refer to Fig. 9 , which shows a block diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. The electronic device in the present application may include one or more of the following components: a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, the memory 120, the input device 130, and the output device 140 may be connected via the bus 150.

[0194] The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0195] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 120 includes a non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The operating system may be an Android system, including a system deeply developed based on the Android system, an IOS system developed by Apple, including a system deeply developed based on the IOS system or other systems. The data storage area may also store data created by the electronic device during use, such as a phone book, audio and video data, chat record data, etc.

[0196] See also Fig.10As shown, the memory 120 can be divided into an operating system space and a user space. The operating system runs in the operating system space, and native and third-party applications run in the user space. In order to ensure that different third-party applications can achieve good operating results, the operating system allocates corresponding system resources to different third-party applications. However, different application scenarios in the same third-party application also have different requirements for system resources. For example, in the local resource loading scenario, the third-party application has higher requirements for disk reading speed; in the animation rendering scenario, the third-party application has higher requirements for GPU performance. The operating system and third-party applications are independent of each other, and the operating system often cannot perceive the current application scenario of the third-party application in a timely manner, resulting in the operating system being unable to perform targeted system resource adaptation according to the specific application scenario of the third-party application.

[0197] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to open up data communication between third-party applications and the operating system so that the operating system can obtain the current scenario information of third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0198] Taking the Android operating system as an example, the programs and data stored in the memory 120 are as follows: Fig.11As shown, the memory 120 may store a Linux kernel layer 320, a system runtime library layer 340, an application framework layer 360 and an application layer 380, wherein the Linux kernel layer 320, the system runtime library layer 340 and the application framework layer 360 belong to the operating system space, and the application layer 380 belongs to the user space. The Linux kernel layer 320 provides underlying drivers for various hardware of electronic devices, such as display drivers, audio drivers, camera drivers, Bluetooth drivers, Wi-Fi drivers, power management, etc. The system runtime library layer 340 provides the main feature support for the Android system through some C / C++ libraries. For example, the SQLite library provides database support, the OpenGL / ES library provides 3D drawing support, and the Webkit library provides browser kernel support, etc. The Android runtime library (Android runtime) is also provided in the system runtime library layer 340, which mainly provides some core libraries that allow developers to use the Java language to write Android applications. The application framework layer 360 provides various APIs that may be used when building applications. Developers can also use these APIs to build their own applications, such as activity management, window management, view management, notification management, content provider, package management, call management, resource management, and location management. At least one application runs in the application layer 380. These applications can be native applications that come with the operating system, such as contact applications, text messaging applications, clock applications, camera applications, etc.; they can also be third-party applications developed by third-party developers, such as game applications, instant messaging applications, photo beautification applications, etc.

[0199] Taking the operating system as an IOS system as an example, the programs and data stored in the memory 120 are as follows: Fig.12As shown, the IOS system includes: a core operating system layer 420 (Core OS layer), a core service layer 440 (Core Services layer), a media layer 460 (Media layer), and a touchable layer 480 (Cocoa Touch Layer). The core operating system layer 420 includes the operating system kernel, drivers, and underlying program frameworks, which provide functions closer to the hardware for use by the program framework located in the core service layer 440. The core service layer 440 provides system services and / or program frameworks required by the application, such as the foundation framework, account framework, advertising framework, data storage framework, network connection framework, geographic location framework, motion framework, etc. The media layer 460 provides audio-visual interfaces for the application, such as graphics and image related interfaces, audio technology related interfaces, video technology related interfaces, and wireless playback (AirPlay) interfaces for audio and video transmission technologies. The touchable layer 480 provides various commonly used interface-related frameworks for application development, and the touchable layer 480 is responsible for the user's touch interaction operations on the electronic device. For example, local notification service, remote push service, advertising framework, game tool framework, message user interface (UI) framework, user interface UIKit framework, map framework, etc.

[0200] exist Fig.12 Among the frameworks shown, the frameworks related to most applications include but are not limited to: the basic framework in the core service layer 440 and the UIKit framework in the touchable layer 480. The basic framework provides many basic object classes and data types, provides the most basic system services for all applications, and has nothing to do with UI. The classes provided by the UIKit framework are basic UI class libraries for creating touch-based user interfaces. iOS applications can provide UIs based on the UIKit framework, so it provides the basic architecture of applications for building user interfaces, drawing, processing and user interaction events, responding to gestures, etc.

[0201] Among them, the method and principle of implementing data communication between third-party applications and the operating system in the IOS system can be referred to the Android system, and this application will not go into details here.

[0202] Among them, the input device 130 is used to receive input instructions or data, and the input device 130 includes but is not limited to a keyboard, a mouse, a camera, a microphone or a touch device. The output device 140 is used to output instructions or data, and the output device 140 includes but is not limited to a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined, and the input device 130 and the output device 140 are touch screen displays, which are used to receive touch operations on or near the user using any suitable object such as a finger or a touch pen, and to display the user interface of each application. The touch screen display is usually set on the front panel of the electronic device. The touch screen display can be designed as a full screen, a curved screen or a special-shaped screen. The touch screen display can also be designed as a combination of a full screen and a curved screen, or a combination of a special-shaped screen and a curved screen, which is not limited in the embodiments of the present application.

[0203] In addition, those skilled in the art will appreciate that the structure of the electronic device shown in the above drawings does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown, or combine certain components, or arrange the components differently. For example, the electronic device also includes a radio frequency circuit, an input unit, a sensor, an audio circuit, a wireless fidelity (WiFi) module, a power supply, a Bluetooth module and other components, which will not be described in detail here.

[0204] In the embodiment of the present application, the execution subject of each step may be the electronic device described above. Optionally, the execution subject of each step is the operating system of the electronic device. The operating system may be an Android system, an IOS system, or other operating systems, which is not limited in the embodiment of the present application.

[0205] The electronic device of the embodiment of the present application may also be equipped with a display device, which may be any device capable of realizing a display function, such as a cathode ray tube display (CR), a light-emitting diode display (LED), an electronic ink screen, a liquid crystal display (LCD), a plasma display panel (PDP), etc. The user may use the display device on the electronic device 101 to view displayed text, images, videos and other information. The electronic device may be a smart phone, a tablet computer, a gaming device, an AR (Augmented Reality) device, a car, a data storage device, an audio playback device, a video playback device, a notebook, a desktop computing device, a wearable device such as an electronic watch, an electronic glasses, an electronic helmet, an electronic bracelet, an electronic necklace, electronic clothing and other devices.

[0206] exist Fig. 9 In the electronic device shown, where the electronic device may be an electronic device, the processor 110 may be used to call an application stored in the memory 120 and specifically perform the following operations:

[0207] Get the front-end page design;

[0208] A front-end code generation model is used to perform code generation processing on the front-end page design drawing to obtain the front-end page code for the front-end page design drawing. The front-end code generation model is obtained by performing migration training on the basic large language model based on the front-end visual prompt generation structure.

[0209] In one embodiment, the front-end visual prompt generation structure includes a front-end prompt generation module, a front-end dimensional transformation linear module, and a large language generation module based on a basic large language model. The processor 110 performs code generation processing on the front-end page design diagram using the front-end code generation model to obtain the front-end page code for the front-end page design diagram, and performs the following operations:

[0210] The front-end page design drawing is input into the target generative model, the front-end page design drawing is subjected to visual prompt encoding processing by the front-end prompt generation module to obtain front-end soft prompt data, the front-end soft prompt data is subjected to multimodal dimensional transformation by the front-end dimensional transformation linear module to obtain front-end soft prompt input features for the large language generative module, the large language generative module is controlled to perform code generation processing on the front-end soft prompt input features to obtain the front-end page code for the front-end page design drawing, and the front-end page code is output.

[0211] In one embodiment, when executing the front-end page code generation method, the processor 110 further performs the following operations:

[0212] Create an initial front-end code generation model for the base large language model based on the front-end visual cue generation structure;

[0213] Get sample training data;

[0214] The sample training data is used to perform front-end code generation migration training on the initial front-end code generation model to obtain a trained front-end code generation model.

[0215] In one embodiment, the processor 110 performs the following steps when executing the process of creating an initial front-end code generation model for a basic large language model based on a front-end visual cue generation structure:

[0216] Get the basic large language model;

[0217] Based on the front-end visual cue generation structure and the basic large language model, an initial front-end code generation model is created, which includes at least a front-end cue generation module, a front-end dimensional transformation linear module and a large language generation module.

[0218] In one embodiment, the processor 110 performs the following steps when performing the front-end code generation migration training on the initial front-end code generation model using the sample training data to obtain the trained front-end code generation model:

[0219] Obtaining sample task training data corresponding to at least one training stage task;

[0220] In the training task stage, the sample task training data is used to perform front-end code generation migration training on the initial front-end code generation model until the initial front-end code generation model completes all the training task stages, thereby obtaining a trained front-end code generation model.

[0221] In one embodiment, the training task phase includes a large language module instruction optimization phase, a model migration warm-up training phase, a model migration adjustment training phase, and a visual instruction tuning phase. The initial front-end code generation model includes a front-end prompt generation module, a front-end dimensional transformation linear module, and a large language generation module. The processor 110 performs the following steps when performing the front-end code generation migration training on the initial front-end code generation model using the sample task training data:

[0222] In the large language model instruction optimization stage, the first sample task training data is input into the large language generation module to perform instruction generation code training to obtain an instruction generation result, and the large language generation module is fine-tuned based on the instruction generation result to obtain the large language generation module after parameter fine-tuning;

[0223] In the model migration warm-up training stage, the word vector converter from the front-end prompt generation module to the large language generation module is trained using the second sample task training data, the front-end dimensional transformation linear module is initialized based on the word vector converter, and the initial front-end code generation model is pre-trained for model migration using the second sample task training data to obtain the initial front-end code generation model after model migration pre-training;

[0224] In the model migration adjustment training stage, the front-end prompt generation module and the front-end dimensional transformation linear module are subjected to model migration training using the third sample task training data to obtain the front-end prompt generation module and the front-end dimensional transformation linear module after the model migration training;

[0225] In the visual instruction tuning stage, the fourth sample task training data is used to perform front-end code generation training on the initial front-end code generation model to obtain a front-end code generation result, and the model parameters of the initial front-end code generation model are fine-tuned based on the front-end code generation result to obtain a front-end code generation model.

[0226] In one embodiment, the processor 110 performs the following steps when performing the model migration pre-training on the initial front-end code generation model using the second sample task training data to obtain the initial front-end code generation model after the model migration pre-training:

[0227] The second sample task training data is used to perform model migration pre-training on the front-end dimensional transformation linear module in the initial front-end code generation model, and the model parameters of the front-end prompt generation module and the large language generation module are controlled to remain unchanged, so as to obtain the front-end dimensional transformation linear module after model migration pre-training.

[0228] In one embodiment, the processor 110 performs the following steps when fine-tuning the model parameters of the initial front-end code generation model based on the front-end code generation result:

[0229] Based on the front-end code generation result, model parameters of the front-end prompt generation module and the front-end dimensional transformation linear module in the initial front-end code generation model are fine-tuned, and the model parameters of the large language generation module are controlled to remain unchanged.

[0230] In one embodiment, before executing the method of performing front-end code generation training on the initial front-end code generation model using the fourth sample task training data to obtain the front-end code generation result, the processor 110 may further perform the following steps:

[0231] Acquire source sample task training data consisting of sample front-end page design drawings, perform graph specification normalization processing on the source sample task training data, and obtain the source sample task training data after graph specification normalization processing

[0232] The source sample task training data is processed by prompt word annotation to obtain fourth sample task training data of the front-end graph code pair type, where the front-end graph code pair type is the type corresponding to the sample front-end page design drawing and the prompt word corresponding to the sample front-end page design drawing.

[0233] In an embodiment of the present application, an electronic device obtains a front-end page design drawing, and uses a front-end code generation model to perform code generation processing on the front-end page design drawing to obtain a front-end page code for the front-end page design drawing. The front-end code generation model is obtained after transfer training of a basic large language model based on a front-end visual cue generation structure. Using the front-end code generation model after transfer training can avoid spending a lot of time and energy, improve the efficiency of front-end design and development, and reduce front-end development costs.

[0234] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0235] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A method for generating front-end page code based on a basic large language model, characterized in that: The method comprises: Get the front-end page design; A front-end code generation model is used to perform code generation processing on the front-end page design drawing to obtain the front-end page code for the front-end page design drawing. The front-end code generation model is obtained by performing migration training on the basic large language model based on the front-end visual prompt generation structure.

2. The method according to claim 1, characterized in that The front-end visual prompt generation structure includes a front-end prompt generation module, a front-end dimensional transformation linear module and a large language generation module based on a basic large language model. The front-end page design diagram is subjected to code generation processing using the front-end code generation model to obtain the front-end page code for the front-end page design diagram, including: The front-end page design drawing is input into the target generative model, the front-end page design drawing is subjected to visual prompt encoding processing by the front-end prompt generation module to obtain front-end soft prompt data, the front-end soft prompt data is subjected to multimodal dimensional transformation by the front-end dimensional transformation linear module to obtain front-end soft prompt input features for the large language generative module, the large language generative module is controlled to perform code generation processing on the front-end soft prompt input features to obtain the front-end page code for the front-end page design drawing, and the front-end page code is output.

3. The method according to claim 1, characterized in that The method further comprises: Create an initial front-end code generation model for the base large language model based on the front-end visual cue generation structure; Get sample training data; The sample training data is used to perform front-end code generation migration training on the initial front-end code generation model to obtain a trained front-end code generation model.

4. The method according to claim 3, characterized in that: The method of creating an initial front-end code generation model for a basic large language model based on a front-end visual prompt generation structure includes: Get the basic large language model; Based on the front-end visual cue generation structure and the basic large language model, an initial front-end code generation model is created, which includes at least a front-end cue generation module, a front-end dimensional transformation linear module and a large language generation module.

5. The method according to claim 4, characterized in that The method of performing front-end code generation migration training on the initial front-end code generation model using the sample training data to obtain a trained front-end code generation model includes: Obtaining sample task training data corresponding to at least one training stage task; In the training task stage, the sample task training data is used to perform front-end code generation migration training on the initial front-end code generation model until the initial front-end code generation model completes all the training task stages, thereby obtaining a trained front-end code generation model.

6. The method according to claim 5, characterized in that The training task phase includes a large language module instruction optimization phase, a model migration warm-up training phase, a model migration adjustment training phase, and a visual instruction tuning phase. The initial front-end code generation model includes a front-end prompt generation module, a front-end dimensional transformation linear module, and a large language generation module. The using the sample task training data to perform front-end code generation migration training on the initial front-end code generation model includes: In the large language model instruction optimization stage, the first sample task training data is input into the large language generation module to perform instruction generation code training to obtain an instruction generation result, and the large language generation module is fine-tuned based on the instruction generation result to obtain the large language generation module after parameter fine-tuning; In the model migration warm-up training stage, the word vector converter from the front-end prompt generation module to the large language generation module is trained using the second sample task training data, the front-end dimensional transformation linear module is initialized based on the word vector converter, and the initial front-end code generation model is pre-trained for model migration using the second sample task training data to obtain the initial front-end code generation model after model migration pre-training; In the model migration adjustment training stage, the front-end prompt generation module and the front-end dimensional transformation linear module are subjected to model migration training using the third sample task training data to obtain the front-end prompt generation module and the front-end dimensional transformation linear module after the model migration training; In the visual instruction tuning stage, the fourth sample task training data is used to perform front-end code generation training on the initial front-end code generation model to obtain a front-end code generation result, and the model parameters of the initial front-end code generation model are fine-tuned based on the front-end code generation result to obtain a front-end code generation model.

7. The method according to claim 6, characterized in that The using the second sample task training data to perform model migration pre-training on the initial front-end code generation model to obtain the initial front-end code generation model after model migration pre-training includes: The second sample task training data is used to perform model migration pre-training on the front-end dimensional transformation linear module in the initial front-end code generation model, and the model parameters of the front-end prompt generation module and the large language generation module are controlled to remain unchanged, so as to obtain the front-end dimensional transformation linear module after model migration pre-training.

8. The method according to claim 6, characterized in that The fine-tuning of model parameters of the initial front-end code generation model based on the front-end code generation result includes: Based on the front-end code generation result, model parameters of the front-end prompt generation module and the front-end dimensional transformation linear module in the initial front-end code generation model are fine-tuned, and the model parameters of the large language generation module are controlled to remain unchanged.

9. The method according to claim 6, characterized in that Before the fourth sample task training data is used to perform front-end code generation training on the initial front-end code generation model to obtain the front-end code generation result, the method further includes: Acquire source sample task training data consisting of sample front-end page design drawings, and perform graph specification normalization processing on the source sample task training data to obtain the source sample task training data after the graph specification normalization processing; The source sample task training data is processed by prompt word annotation to obtain fourth sample task training data of the front-end graph code pair type, where the front-end graph code pair type is the type corresponding to the sample front-end page design drawing and the prompt word corresponding to the sample front-end page design drawing.

10. A front-end page code generation device based on a basic large language model, characterized in that: The device comprises: Image acquisition module, used to obtain the front-end page design image; The code generation module is used to use a front-end code generation model to perform code generation processing on the front-end page design drawing to obtain the front-end page code for the front-end page design drawing. The front-end code generation model is obtained by performing migration training on the basic large language model based on the front-end visual prompt generation structure.

Images