Method, system and equipment for automatically generating GUI (Graphical User Interface) code based on graphical interface

Abstract

The invention discloses a GUI code automatic generation method, system and device based on a graphical interface. The method comprises the steps that UI interface image training data and corresponding UI code training data are obtained; uI interface image training data and the UI code training data are input into an initial code generation model for training, a preset code generation model is obtained, and the initial code generation model is composed of a language module, a vision module and a multi-modal information fusion module; the multi-modal information fusion module comprises a multi-head self-attention layer, a multi-head cross attention layer and a feedforward network; and realizing automatic generation of the GUI code of the graphical interface based on a preset code generation model. According to the method, the preset code generation model is constructed according to the language module, the vision module and the multi-modal information fusion module, various controls in the UI design drawing are effectively identified to generate corresponding graphical user interface codes by fusing the feature information of the images and the codes, and the accuracy of GUI code generation is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic code generation, and in particular to a method, system and device for automatically generating GUI code based on a graphical interface. Background Art

[0002] In recent years, the field of automated interface generation has become a research hotspot, especially the task of converting graphical user interface (GUI) images into code. In the traditional graphical interface development process, the design and implementation of the GUI is usually divided into two stages: first, the designer uses a graphic design tool to create a UI design drawing, and then the developer writes the corresponding code based on the design drawing. This process is often time-consuming and labor-intensive, and is prone to human errors. Therefore, how to achieve automatic generation of GUI code has become a problem that needs to be solved urgently.

[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present invention is to provide a method, system and device for automatically generating GUI code based on a graphical interface, aiming to solve the technical problem of how to realize automatic generation of GUI code.

[0005] To achieve the above object, the present invention provides a method for automatically generating GUI code based on a graphical interface, the method comprising:

[0006] Obtain UI interface image training data and corresponding UI code training data;

[0007] Input the UI interface image training data and the UI code training data into the initial code generation model for training to obtain a preset code generation model, wherein the initial code generation model is composed of a language module, a visual module and a multimodal information fusion module, wherein the multimodal information fusion module includes a multi-head self-attention layer, a multi-head cross-attention layer and a feedforward network;

[0008] Automatic generation of GUI code of a graphical interface is achieved based on the preset code generation model.

[0009] Optionally, the step of inputting the UI interface image training data and the UI code training data into an initial code generation model for training to obtain a preset code generation model includes:

[0010] Dividing the UI interface image training data to obtain a plurality of UI interface image blocks;

[0011] Input multiple UI interface image blocks into the visual model in the initial code generation model to obtain the image feature vector corresponding to each UI interface image block;

[0012] Inputting the UI code training data into the language module in the initial code generation model to obtain a text feature vector;

[0013] The multimodal information fusion module in the initial code generation model is trained according to the image feature vector and the text feature vector to obtain a preset code generation model.

[0014] Optionally, the step of training a multimodal information fusion module in the initial code generation model according to the image feature vector and the text feature vector to obtain a preset code generation model includes:

[0015] The image feature vector and the text feature vector are respectively input into the multi-head self-attention layer in the multimodal information fusion module in the initial code generation model for training to obtain an image feature weight matrix and a text feature weight matrix;

[0016] Based on the image feature weight matrix and the text feature weight matrix, a feature fusion weight matrix is ​​obtained through a multi-head cross attention layer in the multimodal information fusion module;

[0017] Obtaining a prediction code vector matrix through a feedforward network in the multimodal information fusion module according to the feature fusion weight matrix;

[0018] Generate a predicted GUI code according to the predicted code vector matrix;

[0019] When the predicted GUI code meets the preset cross entropy loss condition, the trained initial code generation model is used as the preset code generation model.

[0020] Optionally, the step of inputting the image feature vector into a multi-head self-attention layer in a multimodal information fusion module in the initial code generation model for training to obtain an image feature weight matrix includes:

[0021] Based on the multi-head self-attention layer in the multimodal information fusion module in the initial code generation model, a local feature weight matrix of the image is obtained through a multi-head self-attention mechanism algorithm according to the image feature vector;

[0022] The multi-head self-attention mechanism algorithm is:

[0023]

[0024] In the formula, P is the image local feature weight matrix, Y is the image feature vector, is the attention head weight matrix, and The weight matrix for model training and fitting, head i is the i-th attention head;

[0025] Residual connection and layer normalization are performed on the local feature weight matrix of the image to obtain an image feature weight matrix.

[0026] Optionally, the step of obtaining a feature fusion weight matrix based on the image feature weight matrix and the text feature weight matrix through a multi-head cross attention layer in the multimodal information fusion module includes:

[0027] Based on the multi-head cross attention layer in the multimodal information fusion module, a feature fusion weight matrix is ​​obtained according to the image feature weight matrix and the text feature weight matrix through a multi-head cross attention mechanism algorithm;

[0028] The multi-head cross attention mechanism algorithm is:

[0029] MHCA=MultiHead(Q,K,V)=Concat(head 1 ,head 2 ,…,head h )·W O

[0030] head i =Attention(L′·W i Q ,P′·W i K ,P′·W i V )

[0031]

[0032] Q=L′·W Q

[0033] K=P′·W K

[0034] V=P′·W V

[0035] In the formula, MHCA is the feature fusion weight matrix, L′ is the text feature weight matrix, P′ is the image feature weight matrix, and W O is the attention head weight matrix, W Q , W K and W V The weight matrix fitted for model training;

[0036] Residual connection and layer normalization are performed on the feature fusion weight matrix to obtain a feature fusion weight matrix.

[0037] Optionally, the step of obtaining a prediction code vector matrix through a feedforward network in the multimodal information fusion module according to the feature fusion weight matrix comprises:

[0038] Inputting the feature fusion weight matrix into the feedforward network in the multimodal information fusion module to obtain a prediction code vector matrix;

[0039] The prediction code vector matrix is ​​subjected to residual connection and layer normalization to obtain a prediction code vector matrix.

[0040] Optionally, the step of inputting the feature fusion weight matrix into a feedforward network in the multimodal information fusion module to obtain a prediction code vector matrix includes:

[0041] Based on the feedforward network in the multimodal information fusion module, a prediction code vector matrix is ​​obtained through a Gelu activation function according to the feature fusion weight matrix;

[0042] The Gelu activation function is:

[0043] FFN(MHCA′)=Gelu(k 2 ·Gelu(k 1 MHCA′+b 1 )+b 2 )

[0044] In the formula, k 1 and k 2 is the coefficient, b 1 and b 2 is the bias parameter, MHCA′ is the predicted code vector matrix, and FFN(MHCA′) is the predicted code vector matrix output by the FFN layer.

[0045] In addition, to achieve the above-mentioned purpose, the present invention also proposes a GUI code automatic generation system based on a graphical interface, and the GUI code automatic generation system based on a graphical interface includes:

[0046] An acquisition module is used to acquire UI interface image training data and corresponding UI code training data;

[0047] A training module, used to input the UI interface image training data and the UI code training data into an initial code generation model for training to obtain a preset code generation model, wherein the initial code generation model is composed of a language module, a visual module, and a multimodal information fusion module, wherein the multimodal information fusion module includes a multi-head self-attention layer, a multi-head cross-attention layer, and a feedforward network;

[0048] A generation module is used to realize automatic generation of GUI code of a graphical interface based on the preset code generation model.

[0049] In addition, to achieve the above-mentioned objectives, the present invention also proposes a GUI code automatic generation device based on a graphical interface, the device comprising: a memory, a processor, and a GUI code automatic generation program based on a graphical interface stored in the memory and executable on the processor, the GUI code automatic generation program based on a graphical interface being configured to implement the steps of the GUI code automatic generation method based on a graphical interface as described above.

[0050] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a GUI code automatic generation program based on a graphical interface is stored. When the GUI code automatic generation program based on a graphical interface is executed by a processor, the steps of the GUI code automatic generation method based on a graphical interface as described above are implemented.

[0051] The present invention first obtains UI interface image training data and corresponding UI code training data, and then inputs the UI interface image training data and the UI code training data into the initial code generation model for training to obtain a preset code generation model, wherein the initial code generation model is composed of a language module, a visual module and a multimodal information fusion module, wherein the multimodal information fusion module includes a multi-head self-attention layer, a multi-head cross-attention layer and a feedforward network, and then the GUI code of the graphical interface is automatically generated based on the preset code generation model. Compared with the prior art, developers are required to write corresponding codes according to the design drawings, which is time-consuming and laborious. The present invention constructs a preset code generation model according to the language module, the visual module and the multimodal information fusion module, and effectively identifies various controls in the UI design drawings by fusing the feature information of the image and the code to automatically generate the corresponding graphical user interface code, thereby improving the accuracy of GUI code generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a structural schematic diagram of a GUI code automatic generation device based on a graphical interface in a hardware operating environment involved in an embodiment of the present invention;

[0053] Figure 2 It is a flowchart of the first embodiment of the method for automatically generating GUI code based on the graphical interface of the present invention;

[0054] Figure 3 A schematic diagram of a network framework of a first embodiment of a method for automatically generating GUI code based on a graphical interface according to the present invention;

[0055] Figure 4It is a structural block diagram of the first embodiment of the GUI code automatic generation system based on the graphical interface of the present invention.

[0056] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0058] Reference Figure 1 , Figure 1 A schematic diagram of a device structure is automatically generated based on a graphical interface GUI code for a hardware operating environment involved in an embodiment of the present invention.

[0059] like Figure 1 As shown, the GUI code automatic generation device based on the graphical interface may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage system independent of the aforementioned processor 1001.

[0060] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the GUI code automatic generation device based on the graphical interface, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0061] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a GUI code automatic generation program based on a graphical interface.

[0062] exist Figure 1In the GUI code automatic generation device based on a graphical interface shown, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the GUI code automatic generation device based on a graphical interface of the present invention can be set in the GUI code automatic generation device based on a graphical interface, and the GUI code automatic generation device based on a graphical interface calls the GUI code automatic generation program based on a graphical interface stored in the memory 1005 through the processor 1001, and executes the GUI code automatic generation method based on a graphical interface provided in an embodiment of the present invention.

[0063] The embodiment of the present invention provides a method for automatically generating GUI code based on a graphical interface, referring to Figure 2 , Figure 2 It is a flowchart diagram of the first embodiment of the method for automatically generating GUI code based on a graphical interface of the present invention.

[0064] In this embodiment, the method for automatically generating GUI code based on a graphical interface includes the following steps:

[0065] Step S10: Obtain UI interface image training data and corresponding UI code training data.

[0066] It is easy to understand that the execution subject of this embodiment can be a graphical interface-based GUI code automatic generation system with functions such as data processing, network communication and program running, or it can be other computer devices with similar functions, etc. This embodiment is not limited.

[0067] In this embodiment, in order to solve the limitations of data set size and complexity, the required DSL tags can be randomly generated according to needs, and then the tags can be compiled into corresponding UI codes. Finally, the rendered interface effect diagram is generated through the graphics rendering engine and a screenshot is saved, thereby obtaining multiple UI interface images (i.e., UI interface image training data) and the DSL codes corresponding to each UI interface image (i.e., UI code training data).

[0068] For DSL codes, all indents and line breaks are first removed, and special <pad>The markers are used to pad the sequence to ensure that all sequences in the batch have the same length, so that the model can process them in batches. At the same time, the start and end positions of each code sequence are added. <start>and <end>Mark to mark the start and end of the code. The DSL code after the above processing is used as UI code training data.

[0069] For UI interface images, all images are first scaled to a size of 224*224, and then standardized on the three RGB channels. The pixel mean of each channel is set to [0.485, 0.456, 0.406], and the standard deviation is set to [0.229, 0.224, 0.225]. The interface images after the above processing are used as UI interface image training data.

[0070] Step S20: Input the UI interface image training data and the UI code training data into the initial code generation model for training to obtain a preset code generation model, wherein the initial code generation model is composed of a language module, a visual module and a multimodal information fusion module, and the multimodal information fusion module includes a multi-head self-attention layer, a multi-head cross-attention layer and a feedforward network.

[0071] Furthermore, the UI interface image training data is divided and processed to obtain multiple UI interface image blocks; the multiple UI interface image blocks are input into the visual model in the initial code generation model to obtain the image feature vector corresponding to each UI interface image block; the UI code training data is input into the language module in the initial code generation model to obtain the text feature vector; the multimodal information fusion module in the initial code generation model is trained according to the image feature vector and the text feature vector to obtain the preset code generation model, wherein the initial code generation model is composed of a language module, a visual module and a multimodal information fusion module, and the multimodal information fusion module includes a multi-head self-attention layer, a multi-head cross-attention layer and a feedforward network. Reference Figure 3 , Figure 3 This is a schematic diagram of the network framework of the first embodiment of the method for automatically generating GUI code based on a graphical interface of the present invention.

[0072] After pre-training, the visual model in the visual module has powerful image feature processing capabilities. Its pre-trained weights are fine-tuned on the image code dataset so that the model can adapt to the needs of image code generation tasks. The language model in the language module has powerful generation capabilities and the ability to understand language structure. Its pre-trained weights are fine-tuned on the image code dataset so that the model can understand the semantic information of the code, ensure the grammatical correctness and semantic coherence of the code during the generation process, and generate high-quality hierarchical code.

[0073] It should also be noted that the visual module can use the ViT model as an encoder to convert the input image information into a feature vector representation, but this embodiment does not limit the image processing model.

[0074] It should be understood that the ViT model uses a self-attention mechanism to capture global features and local associations in images. Its working principle is to divide the image into a series of image blocks and use the pixel values ​​of these image blocks as input sequences, so that the Transformer model can directly process image data. Compared with traditional convolutional neural networks, the ViT model does not need to use convolutional layers when processing images, but relies entirely on the self-attention mechanism.

[0075] The ViT model is responsible for converting the input image data into a high-dimensional feature vector representation. By dividing the image into image blocks and inputting them into the ViT model, the feature vector representation of the entire image is obtained.

[0076] In the specific implementation, the pre-trained ViT model weights are used and fine-tuned on the task-specific dataset to adapt to the needs of the specific task. The fine-tuning process involves adjusting the parameters of the model so that it can better extract task-related features and achieve better performance.

[0077] In this embodiment, the language module can use the GPT-2 model as a decoder. The model has powerful text generation capabilities, but this embodiment does not limit the text processing model.

[0078] It should be understood that the GPT-2 model uses the Transformer architecture, the core of which is the stacking of multi-head self-attention mechanism and feedforward neural network layers.

[0079] It should also be noted that by using pre-trained visual models and language models, the present invention can quickly and accurately generate graphical interface code, reducing the time and labor costs required for traditional manual coding. The automated generation process significantly shortens the development cycle and accelerates product iteration and update speed.

[0080] Because the model has been fine-tuned, it can ensure the grammatical correctness and semantic coherence of the generated code, reducing common grammatical errors and logical loopholes in manual writing. The automatically generated code has high consistency, avoiding the problems of inconsistent styles and standards caused by individual differences in manual writing.

[0081] Furthermore, the multimodal information fusion module in the initial code generation model is trained according to the image feature vector and the text feature vector, and the processing method of obtaining the preset code generation model is as follows: the image feature vector and the text feature vector are respectively input into the multi-head self-attention layer in the multimodal information fusion module in the initial code generation model for training to obtain the image feature weight matrix and the text feature weight matrix; based on the image feature weight matrix and the text feature weight matrix, the multi-head cross-attention layer in the multimodal information fusion module is used to obtain the feature fusion weight matrix; according to the feature fusion weight matrix, the feedforward network in the multimodal information fusion module is used to obtain the predicted code vector matrix; according to the predicted code vector matrix, the predicted GUI code is generated; when the predicted GUI code meets the preset cross entropy loss condition, the trained initial code generation model is used as the preset code generation model.

[0082] The preset cross entropy loss condition is that the cross entropy loss between the predicted GUI code and the real GUI code converges gradually and stabilizes.

[0083] It should also be noted that a multi-head self-attention mechanism is introduced in this embodiment. This mechanism enables each feature vector to dynamically pay attention to other feature vectors, thereby improving the processing ability of the intrinsic connection between local features. Among them, the number of attention heads can be 8, or more, and the hidden layer dimension is 768 or other values.

[0084] Furthermore, the image feature vector is input into the multi-head self-attention layer corresponding to the visual module in the multimodal information fusion module in the initial code generation model for training, and the processing method for obtaining the image feature weight matrix is ​​as follows: based on the multi-head self-attention layer corresponding to the visual module in the multimodal information fusion module in the initial code generation model, the image local feature weight matrix is ​​obtained through the multi-head self-attention mechanism algorithm according to the image feature vector; based on the image feature vector, the image local feature weight matrix is ​​residually connected and layer normalized to obtain the image feature weight matrix.

[0085] The multi-head self-attention mechanism algorithm is:

[0086]

[0087] In the formula, P is the image local feature weight matrix, Y is the image feature vector, is the attention head weight matrix, and The weight matrix for model training and fitting, head i is the i-th attention head, d k =96.

[0088] Furthermore, the text feature vector is input into the multi-head self-attention layer corresponding to the language module in the multimodal information fusion module in the initial code generation model for training, and the processing method for obtaining the image feature weight matrix is ​​as follows: based on the multi-head self-attention layer corresponding to the language module in the multimodal information fusion module in the initial code generation model, the text feature weight matrix is ​​obtained through the multi-head self-attention mechanism algorithm according to the text feature vector; based on the text feature vector, the text feature weight matrix is ​​residually connected and layer normalized to obtain the text feature weight matrix.

[0089] The multi-head self-attention mechanism algorithm is:

[0090]

[0091] In the formula, L is the text feature weight matrix, X is the text feature vector, is the attention head weight matrix, and The weight matrix for model training and fitting, head i is the i-th attention head, d k =96.

[0092] It should be understood that the multi-head self-attention layer enables each feature vector to dynamically pay attention to other feature vectors, thereby improving the ability to handle the intrinsic connections between local features.

[0093] It should also be noted that the output of the multi-head self-attention layer is processed with residual connection and layer normalization to make the learning process of the model more stable, which helps to alleviate the gradient vanishing and gradient exploding problems during training and thus improve the performance of the model.

[0094] Furthermore, based on the image feature weight matrix and the text feature weight matrix, a processing method for obtaining the feature fusion weight matrix is ​​as follows: based on the multi-head cross attention layer in the multi-modal information fusion module, the feature fusion weight matrix is ​​obtained according to the image feature weight matrix and the text feature weight matrix through the multi-head cross attention mechanism algorithm; the feature fusion weight matrix is ​​residually connected and layer normalized to obtain the feature fusion weight matrix.

[0095] The multi-head cross attention mechanism algorithm is:

[0096] MHCA=MultiHead(Q,K,V)=Concat(head 1 ,head 2 ,…,head h )·W O

[0097] head i =Attention(L′·W i Q ,P′·W i K ,P′·W i V )

[0098]

[0099] Q=L′·W Q

[0100] K=P′·W K

[0101] V=P′·W V

[0102] In the formula, MHCA is the feature fusion weight matrix, L′ is the text feature weight matrix, P′ is the image feature weight matrix, and W O is the attention head weight matrix, W Q , W K and W V The weight matrix for model training and fitting, d k =96.

[0103] It should also be noted that the feedforward network is used to further extract and abstract feature information to enhance the model's ability to represent image features and text features. The Gelu activation function is used in the FFN layer, and d model is 512, d ff The size of the dataset is 2048 and the dropout is 20%.

[0104] In this embodiment, the feature fusion weight matrix is ​​input into the feedforward network in the multimodal information fusion module through the Gelu activation function to obtain the prediction code vector matrix; the prediction code vector matrix is ​​residually connected and layer normalized to obtain the prediction code vector matrix.

[0105] The Gelu activation function is:

[0106] FFN(MHCA′)=Gelu(k 2 ·Gelu(k 1 MHCA′+b 1 )+b 2 )

[0107] In the formula, k 1 and k 2 is the coefficient, b 1 and b 2 is the bias parameter, MHCA′ is the predicted code vector matrix, and FFN(MHCA′) is the predicted code vector matrix output by the FFN layer.

[0108] In the specific implementation, residual connections and layer normalization are performed on the output of the feedforward network layer and the output of the language model after the multi-head self-attention layer (i.e., the text feature weight matrix) to further improve the model performance.

[0109] Step S30: Automatically generate GUI code of a graphical interface based on the preset code generation model.

[0110] In a specific implementation, the acquired UI interface image can be directly input into a trained preset code generation model to output the GUI code corresponding to the UI interface image.

[0111] In this embodiment, firstly, UI interface image training data and corresponding UI code training data are obtained, and then the UI interface image training data and the UI code training data are input into the initial code generation model for training to obtain a preset code generation model, the initial code generation model is composed of a language module, a visual module and a multimodal information fusion module, the multimodal information fusion module includes a multi-head self-attention layer, a multi-head cross-attention layer and a feedforward network, and then the GUI code of the graphical interface is automatically generated based on the preset code generation model. Compared with the prior art, developers are required to write corresponding codes according to the design drawings, which is time-consuming and laborious, and this embodiment constructs a preset code generation model according to the language module, the visual module and the multimodal information fusion module, and effectively identifies various controls in the UI design drawing through the multimodal information fusion module according to the feature information of the fused image and the code to automatically generate the corresponding graphical user interface code, thereby improving the accuracy and quality of GUI code generation.

[0112] Reference Figure 4 , Figure 4 It is a structural block diagram of the first embodiment of the GUI code automatic generation system based on the graphical interface of the present invention.

[0113] like Figure 4 As shown, the GUI code automatic generation system based on the graphical interface proposed in the embodiment of the present invention includes:

[0114] Acquisition module 4001, used to acquire UI interface image training data and corresponding UI code training data;

[0115] The training module 4002 is used to input the UI interface image training data and the UI code training data into the initial code generation model for training to obtain a preset code generation model, wherein the initial code generation model is composed of a language module, a visual module and a multimodal information fusion module, wherein the multimodal information fusion module includes a multi-head self-attention layer, a multi-head cross-attention layer and a feedforward network;

[0116] The generation module 4003 is used to realize automatic generation of GUI code of the graphical interface based on the preset code generation model.

[0117] Other embodiments or specific implementations of the GUI code automatic generation system based on a graphical interface of the present invention can refer to the above-mentioned method embodiments, which will not be repeated here.

[0118] It should be noted that, in this article, the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0119] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0121] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.< / end> < / start> < / pad>

Claims

1. A method for automatically generating GUI code based on a graphical interface, characterized in that: The GUI code automatic generation method based on the graphical interface comprises the following steps: Obtain UI interface image training data and corresponding UI code training data; Input the UI interface image training data and the UI code training data into the initial code generation model for training to obtain a preset code generation model, wherein the initial code generation model is composed of a language module, a visual module and a multimodal information fusion module, wherein the multimodal information fusion module includes a multi-head self-attention layer, a multi-head cross-attention layer and a feedforward network; Automatic generation of GUI code of a graphical interface is achieved based on the preset code generation model.

2. The method according to claim 1, characterized in that The step of inputting the UI interface image training data and the UI code training data into the initial code generation model for training to obtain a preset code generation model includes: Dividing the UI interface image training data to obtain a plurality of UI interface image blocks; Input multiple UI interface image blocks into the visual model in the initial code generation model to obtain the image feature vector corresponding to each UI interface image block; Inputting the UI code training data into the language module in the initial code generation model to obtain a text feature vector; The multimodal information fusion module in the initial code generation model is trained according to the image feature vector and the text feature vector to obtain a preset code generation model.

3. The method according to claim 2, characterized in that The step of training the multimodal information fusion module in the initial code generation model according to the image feature vector and the text feature vector to obtain a preset code generation model includes: The image feature vector and the text feature vector are respectively input into the multi-head self-attention layer in the multimodal information fusion module in the initial code generation model for training to obtain an image feature weight matrix and a text feature weight matrix; Based on the image feature weight matrix and the text feature weight matrix, a feature fusion weight matrix is ​​obtained through a multi-head cross attention layer in the multimodal information fusion module; Obtaining a prediction code vector matrix through a feedforward network in the multimodal information fusion module according to the feature fusion weight matrix; Generate a predicted GUI code according to the predicted code vector matrix; When the predicted GUI code meets the preset cross entropy loss condition, the trained initial code generation model is used as the preset code generation model.

4. The method according to claim 3, characterized in that The step of inputting the image feature vector into the multi-head self-attention layer in the multimodal information fusion module in the initial code generation model for training to obtain the image feature weight matrix includes: Based on the multi-head self-attention layer in the multimodal information fusion module in the initial code generation model, a local feature weight matrix of the image is obtained through a multi-head self-attention mechanism algorithm according to the image feature vector; The multi-head self-attention mechanism algorithm is: In the formula, P is the image local feature weight matrix, Y is the image feature vector, is the attention head weight matrix, and The weight matrix for model training and fitting, head i is the i-th attention head; Residual connection and layer normalization are performed on the local feature weight matrix of the image to obtain an image feature weight matrix.

5. The method according to claim 4, characterized in that The step of obtaining a feature fusion weight matrix based on the image feature weight matrix and the text feature weight matrix through a multi-head cross attention layer in the multimodal information fusion module includes: Based on the multi-head cross attention layer in the multimodal information fusion module, a feature fusion weight matrix is ​​obtained according to the image feature weight matrix and the text feature weight matrix through a multi-head cross attention mechanism algorithm; The multi-head cross attention mechanism algorithm is: MHCA=MultiHead(Q,K,V)=Concat(head1,head2,…,head h )·W O head i =Attention(L′·W i Q ,P′·W i K ,P′·W i V ) Q=L′·W Q K=P′·W K V=P′·W V In the formula, MHCA is the feature fusion weight matrix, L′ is the text feature weight matrix, P′ is the image feature weight matrix, and W O is the attention head weight matrix, W Q , W K and W V The weight matrix fitted for model training; Residual connection and layer normalization are performed on the feature fusion weight matrix to obtain a feature fusion weight matrix.

6. The method according to claim 3, characterized in that The step of obtaining a prediction code vector matrix through a feedforward network in the multimodal information fusion module according to the feature fusion weight matrix comprises: Inputting the feature fusion weight matrix into the feedforward network in the multimodal information fusion module to obtain a prediction code vector matrix; The prediction code vector matrix is ​​subjected to residual connection and layer normalization to obtain a prediction code vector matrix.

7. The method according to claim 6, characterized in that The step of inputting the feature fusion weight matrix into the feedforward network in the multimodal information fusion module to obtain a prediction code vector matrix includes: Based on the feedforward network in the multimodal information fusion module, a prediction code vector matrix is ​​obtained through a Gelu activation function according to the feature fusion weight matrix; The Gelu activation function is: FFN(MHCA′)=Gelu(k2·Gelu(k1·MHCA′+b1)+b2) Wherein, k1 and k2 are coefficients, b1 and b2 are bias parameters, MHCA′ is the predicted code vector matrix, and FFN(MHCA′) is the predicted code vector matrix output by the FFN layer.

8. A GUI code automatic generation system based on a graphical interface, characterized in that: The GUI code automatic generation system based on the graphical interface includes: An acquisition module is used to acquire UI interface image training data and corresponding UI code training data; A training module, used to input the UI interface image training data and the UI code training data into an initial code generation model for training to obtain a preset code generation model, wherein the initial code generation model is composed of a language module, a visual module, and a multimodal information fusion module, wherein the multimodal information fusion module includes a multi-head self-attention layer, a multi-head cross-attention layer, and a feedforward network; A generation module is used to realize automatic generation of GUI code of a graphical interface based on the preset code generation model.

9. A GUI code automatic generation device based on a graphical interface, characterized in that: The device includes: a memory, a processor, and a graphical interface-based GUI code automatic generation program stored in the memory and executable on the processor, wherein the graphical interface-based GUI code automatic generation program is configured to implement the steps of the graphical interface-based GUI code automatic generation method as described in any one of claims 1 to 7.

Images