Code generation model training method, code generation method, device, and medium

Abstract

The application provides a code generation model training method, a code generation method, equipment and a medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: obtaining multi-modal model training data; wherein the multi-modal model training data comprises first text data and first image data, and the first text data is data for describing the features of the first image data; performing model training on a preset artificial intelligence model based on the first text data and the first image data to obtain a code generation model; wherein the artificial intelligence model is constructed by combining a multi-modal large language model and a diffusion model; and the code generation model is used for learning at least second image data and outputting codes corresponding to the second image data. The application can improve the ability of an artificial intelligence-based code generation model to realize automatic programming based on images or text, enable the code generation model to automatically generate codes of a website interface with complex structure, and thus improve the efficiency of code writing.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a code generation model training method, a code generation method, a device, and a medium. Background Art

[0002] With the continuous development and progress of artificial intelligence technology, automated programming with the help of large language models (LLMs) has become a reality. In related technologies, large language model automated programming can only automatically write code at the level of simple functions according to text prompts. For complex website interfaces that are difficult to clearly describe in words, it still takes a lot of time and human resources to manually write the code data corresponding to the website interface. This results in very low efficiency in writing code for image data similar to website interfaces. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a training method, code generation method, device and medium for a code generation model, aiming to enhance the automated programming capability of the code generation model, so that the code generation model can automatically generate code for a complex website interface, thereby improving the efficiency of code writing.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a method for training a code generation model, the method comprising:

[0005] Acquire multimodal model training data; wherein the multimodal model training data includes first text data and first image data, and the first text data is data for describing features of the first image data;

[0006] Performing model training on a preset artificial intelligence model based on the first text data and the first image data to obtain a code generation model;

[0007] Among them, the artificial intelligence model is constructed by combining a multimodal large language model and a diffusion model; the code generation model is used to learn at least the second image data and output a code corresponding to the second image data.

[0008] In some embodiments, before performing model training on a preset artificial intelligence model based on the first text data and the first image data to obtain a code generation model, the method further includes:

[0009] Get the preset artificial intelligence model;

[0010] Among them, the artificial intelligence model includes an encoder component, a denoiser component, a decoder component and a classification head; the encoder component is created based on a multimodal large language model, and the denoiser component is created based on a diffusion model.

[0011] In some embodiments, the performing model training on a preset artificial intelligence model based on the first text data and the first image data includes:

[0012] performing comparative learning on the first text data and the first image data based on the encoder component to obtain a synthetic vector representing a matching relationship between the first text data and the first image data;

[0013] Performing diffusion learning on the synthetic vector based on the denoiser component to obtain a diffusion target;

[0014] A hidden representation corresponding to the diffusion target is generated based on the decoder component, and a code label distribution corresponding to the hidden representation is calculated based on the classification head.

[0015] In some embodiments, performing diffusion learning on the synthesized vector based on the denoiser component to obtain a diffusion target comprises:

[0016] gradually superimposing Gaussian noise on the synthetic vector based on the denoiser component;

[0017] Self-attention processing is performed on the synthetic vector, the predicted noise corresponding to the synthetic vector, and the Gaussian noise to obtain a diffusion target corresponding to the synthetic vector.

[0018] In some embodiments, the method further comprises:

[0019] Obtaining a preset model training loss function; wherein the model training loss function is calculated based on the error between the predicted noise and the Gaussian noise, the error between the hidden representation and the embedded code, and a standard cross entropy loss function; wherein the embedded code is the source code in the first text data;

[0020] The denoiser component, the decoder component and the classification head are trained based on the model training loss function.

[0021] In some embodiments, the method further comprises at least one of the following:

[0022] Pre-training the denoiser component and the decoder component for unsupervised code generation based on preset Gaussian noise;

[0023] Denoiser input data is obtained by calculating a preset code segment based on the encoder component, and pre-training the denoiser component and the decoder component for denoising continuous code segments is performed based on the denoiser input data.

[0024] In some embodiments, obtaining multimodal model training data includes at least one of the following:

[0025] Obtaining website source code from a preset code hosting platform as first text data, and obtaining a website interface corresponding to the website source code as first image data, so as to construct multimodal model training data;

[0026] Determining a target website that meets a preset website ranking condition, crawling the code of the target website as first text data, and crawling a website screenshot corresponding to the code of the target website as first image data, so as to construct multimodal model training data;

[0027] Obtaining component source code from a preset component library as first text data, and obtaining a component image corresponding to the component source code as first image data, so as to construct multimodal model training data;

[0028] Multimodal representation learning is performed on the model training data based on momentum distillation to remove original noise data in the model training data.

[0029] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a code generation method, the method comprising:

[0030] Acquiring model input data; wherein the model input data at least includes second image data;

[0031] The second image data is input into the code generation model as described in the first aspect above, so as to learn the second image data based on the code generation model and output a code corresponding to the second image data.

[0032] To achieve the above objectives, a third aspect of an embodiment of the present application provides a training device for a code generation model, the device comprising:

[0033] A first acquisition module is configured to acquire multimodal model training data; wherein the multimodal model training data includes first text data and first image data, and the first text data is data for describing features of the first image data;

[0034] A model training module is used to perform model training on a preset artificial intelligence model based on the first text data and the first image data to obtain a code generation model; wherein the artificial intelligence model is constructed in combination with a multimodal large language model and a diffusion model; the code generation model is used to learn at least the second image data and output a code corresponding to the second image data.

[0035] To achieve the above objectives, a fourth aspect of the embodiments of the present application provides a code generation device, comprising:

[0036] A second acquisition module is used to acquire model input data; wherein the model input data at least includes second image data;

[0037] A code generation module is used to input the second image data into the code generation model as described in the first aspect above, so as to learn the second image data based on the code generation model and output a code corresponding to the second image data.

[0038] To achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, and the memory stores a computer program. When the electronic device is applied to a vehicle, the processor implements the method described in the first aspect when executing the computer program, and / or, the processor implements the method described in the second aspect when executing the computer program.

[0039] To achieve the above-mentioned objectives, the sixth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect above, and / or implements the method described in the second aspect above.

[0040] To achieve the above-mentioned objectives, the seventh aspect of an embodiment of the present application proposes a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the method provided in the first aspect above, and / or implements the method provided in the second aspect above.

[0041] The embodiment of the present application obtains an artificial intelligence model by combining a multimodal large language model and a diffusion model, and then obtains multimodal model training data (including first text data and first image data, the first text data is data for describing the features of the first image data), thereby training the preset artificial intelligence model based on the first text data and the first image data to obtain a code generation model. That is, the embodiment of the present application combines the multimodal large language model and the diffusion model training to obtain a code generation model based on multimodality and diffusion, thereby using the code generation model to learn the second image data and output the code corresponding to the second image data. In this way, compared to the traditional method of manually writing code data corresponding to a complex website interface, the embodiment of the present application can enhance the ability of the artificial intelligence-based code generation model to realize automated programming based on images or text. The code generation model obtained by adopting the training method of the code generation model provided by the embodiment of the present application can automatically generate code for a website interface with a complex structure, thereby improving the efficiency of code writing. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of the steps of the code generation model training method provided in some embodiments of the present application;

[0043] Figure 2 A schematic diagram of the model architecture involved in the training method of the code generation model provided in an embodiment of the present application;

[0044] Figure 3 for Figure 1 Schematic diagram of the detailed steps of step S102;

[0045] Figure 4 for Figure 3 Schematic diagram of the detailed steps of step S302;

[0046] Figure 5 A flowchart of the steps of training a model based on a loss function involved in some embodiments of the code generation model training method provided in the embodiments of the present application;

[0047] Figure 6 A schematic diagram of the overall process involved in some embodiments of the code model training method provided in the embodiments of the present application;

[0048] Figure 7 A flowchart of the steps of the code generation method provided in some embodiments of the present application;

[0049] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0051] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0053] First, a brief explanation of the professional and technical terms involved in the embodiments of the present application is given.

[0054] Multi-modal Large Language Model (MLLM).

[0055] MLLMs are typically based on large language models (LLMs) and incorporate other non-textual modal information to accomplish various multimodal tasks. MLLMs can create stories from images without requiring mathematical reasoning like optical character recognition (OCR), offering a potential path toward achieving artificial general intelligence (AGI).

[0056] Compared to unimodal LLMs, MLLMs are more in line with how humans perceive the world. Humans have multiple senses to receive multimodal information, which often complements and synergizes with each other. Therefore, using multimodal information generally leads to better cognition and task completion. Furthermore, MLLMs have a more powerful and user-friendly interface than unimodal LLMs. Because MLLMs support multimodal input, users can input and convey information to MLLMs in a more flexible manner. Furthermore, MLLMs support a wider range of tasks. Unimodal LLMs are typically only capable of completing tasks related to plain text, while MLLMs can accomplish additional tasks through multimodality, such as image description and visual knowledge question answering.

[0057] Diffusion Models.

[0058] Diffusion models have demonstrated excellent performance in image generation, and their application areas are constantly expanding. Diffusion models primarily use an embedding layer to convert discrete tokens into continuous embeddings, adding Gaussian noise to simulate the expansion process, and then selecting the vocabulary token closest to the embedding to map the denoised embeddings back to discrete text.

[0059] Next, the overall concept of the embodiment of the present application is described.

[0060] With the continuous advancement of artificial intelligence technology, automated programming using LLM has become a reality. However, related technologies, such as large language model automated programming, can only provide intelligent prompts and write simple functions according to the correct prompts. However, complex website interfaces are difficult to describe clearly with simple text. Therefore, for such complex website interfaces that are difficult to describe clearly in text, a considerable amount of time and human resources are still required to manually write the corresponding code data. This results in extremely low efficiency in coding for image data similar to website interfaces.

[0061] Based on this, the embodiments of the present application provide a training method for a code generation model, a code generation method, an electronic device, a computer-readable storage medium, and a computer program product, aiming to enhance the automated programming capabilities of the code generation model, so that the code generation model can automatically generate code for a complex website interface, thereby improving the efficiency of code writing.

[0062] The embodiment of the present application obtains an artificial intelligence model by combining a multimodal large language model and a diffusion model, and then obtains multimodal model training data (including first text data and first image data, the first text data is data for describing the features of the first image data), thereby training the preset artificial intelligence model based on the first text data and the first image data to obtain a code generation model. That is, the embodiment of the present application applies the diffusion model to the field of code generation, and obtains a code generation model based on multimodality and diffusion by combining a multimodal large language model and a diffusion model training, thereby using the code generation model to learn the second image data and output the code corresponding to the second image data. In this way, the ability of the artificial intelligence-based code generation model to realize automated programming based on images or text can be improved. Compared with the traditional method of manually writing code data corresponding to complex website interfaces, the code generation model obtained by adopting the model training method of the embodiment of the present application can automatically generate code for website interfaces with complex structures, thereby improving the efficiency of code writing.

[0063] Next, the training method of the code generation model, the code generation method, the device, the electronic device, the computer-readable storage medium and the computer program product provided in the embodiments of the present application are specifically described through the following embodiments, and the training method of the code generation model and the code generation method provided in the embodiments of the present application are first described in detail.

[0064] It should be noted that the training method and code generation method of the code generation model provided in the embodiment of the present application relate to the field of artificial intelligence technology. The training method and code generation method of the code generation model provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, can also be software running in a terminal or a server side. In certain embodiments, the terminal can be an electronic device such as a smart phone, a tablet computer, a notebook computer, a desktop computer. The server side can be a server terminal device, can be configured as an independent physical server, can also be configured as a server cluster or a distributed system consisting of multiple physical servers, can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN) and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms. Software can be an application, a computer program and a storage medium carrying the computer program, etc., that implements the training method and / or code generation method of the code generation model. It should be understood that, based on different design requirements of actual applications, in different feasible embodiments, the training method of the code generation model and / or the terminal, server side, and software of the code generation method provided in the embodiments of the present application can of course also be other forms not listed here. The training method of the code generation model and the code generation method provided in the embodiments of the present application are not specifically limited to this.

[0065] In addition, the present application can also be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, personal computers (PCs), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0066] For ease of understanding and explanation, the following text takes the training method and code generation method of the code generation model provided by the embodiment of the present application as an example to describe in detail the various specific embodiments of the present application. The implementation of the training method and code generation method of the code generation model provided by the embodiment of the present application in any of the above-mentioned forms of subject matter can refer to the process of the training method and code generation method of the code generation model applied by the electronic device described later.

[0067] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0068] Please refer to Figure 1 , Figure 1 The following is a flowchart of the steps of the code generation model training method provided in some embodiments of the present application. It should be understood that although Figure 1 The following flowcharts of other steps illustrate the execution order of some method steps, but based on the different design requirements of actual applications, the code generation model training method provided in the embodiment of the present application can certainly adopt an execution order different from the method steps shown in the figure. That is, Figure 1The order of the steps in the method shown does not constitute a limitation on the execution logic order of the training method of the code generation model provided in the embodiment of the present application. Figure 1 Reasonable changes in the order of the steps of the method shown should be included in the scope of protection of the code generation model training method provided in the embodiments of the present application.

[0069] like Figure 1 As shown, in some embodiments, the vehicle application code generation model training method provided by the embodiment of the present application may include step S101 and step S102.

[0070] Step S101: Acquire multimodal model training data; wherein the multimodal model training data includes first text data and first image data, and the first text data is data for describing features of the first image data.

[0071] When training a large model, the electronic device first collects multimodal model training data. When acquiring the model training data, the electronic device not only acquires the first text data but also acquires first image data having features described by the first text data. In this way, the electronic device constructs multimodal model training data based on the first text data and the first image data.

[0072] It should be noted that the multimodal training data may be large model training data including multiple text-image pairs, wherein each text-image pair is a first image data and first text data for describing features of the first image data.

[0073] Step S102: Based on the first text data and the first image data, a preset artificial intelligence model is trained to obtain a code generation model; wherein, the artificial intelligence model is constructed in combination with a multimodal large language model and a diffusion model; the code generation model is used to learn at least the second image data and output a code corresponding to the second image data.

[0074] After obtaining the multimodal model training data, the electronic device calls the artificial intelligence model pre-constructed by combining the multimodal large language model and the diffusion model, and then uses the multimodal model training data as the input of the artificial intelligence model, thereby training the artificial intelligence model based on the first text data and the first image data in the multimodal model training data. In this way, after the artificial intelligence model is trained based on the first text data and the first image data, the electronic device can obtain a code generation model for subsequent automated programming based on the code generation model to generate code, that is, using the code generation model to learn at least the second image data, thereby outputting code corresponding to the second image data.

[0075] In an embodiment of the present application, an artificial intelligence model is constructed by combining an electronic device with a multimodal large language model and a diffusion model, and by obtaining multimodal model training data (including first text data and first image data, where the first text data is data that describes the features of the first image data), a preset artificial intelligence model is trained based on the first text data and the first image data to obtain a code generation model. That is, the embodiment of the present application applies the diffusion model to the field of code generation, and obtains a code generation model based on multimodality and diffusion by combining the multimodal large language model and the diffusion model training, thereby using the code generation model to learn the second image data and output the code corresponding to the second image data.

[0076] In this way, compared with the traditional method of manually writing code data corresponding to complex website interfaces, the embodiments of the present application can improve the ability of the artificial intelligence-based code generation model to realize automated programming based on images or text. The code generation model obtained by the training method of the code generation model provided by the embodiments of the present application can automatically generate code for website interfaces with complex structures, thereby improving the efficiency of code writing.

[0077] It should be noted that multimodal pre-training models pre-train large models on large-scale data from multiple modalities. This allows them to comprehensively leverage information from different modalities and perform various cross-modal tasks. To achieve this, electronic devices first need to obtain multimodal model training data, then pre-train the model based on this model training data to generate a code generation model.

[0078] In some embodiments, the above-mentioned step S101: obtaining multimodal model training data includes at least one of the following:

[0079] First: obtaining website source code from a preset code hosting platform as first text data, and obtaining a website interface corresponding to the website source code as first image data, so as to construct multimodal model training data.

[0080] It should be noted that the preset code hosting platform can be GitHub.

[0081] The electronic device can obtain a repository on GitHub that has more than 1k star counts and provides a website link, and first obtain the website source code as the first text data, and obtain the website interface corresponding to the website source code as the first image data, thereby constructing multimodal model training data for subsequent model training.

[0082] Second: determine a target website that meets the preset website ranking conditions, capture the code of the target website as the first text data, and capture the website screenshot corresponding to the code of the target website as the first image data to construct multimodal model training data.

[0083] It should be noted that the preset website ranking condition may be that the website is ranked in the top 1 million, etc.

[0084] The electronic device determines the top 1 million target websites through the rankings on Alexa, a global website ranking list, and then captures the code of the target website as the first text data, and simultaneously captures the website screenshot corresponding to the code of the target website as the first image data, thereby also constructing multimodal model training data for subsequent model training.

[0085] Third: obtaining component source code from a preset component library as first text data, and obtaining a component image corresponding to the component source code as first image data, so as to construct multimodal model training data.

[0086] It should be noted that the preset component library can be Ant Design.

[0087] The electronic device can also obtain component source code from the industry-wide component library Ant Design as first text data, and component images corresponding to the component source code as first image data, thereby constructing multimodal model training data for subsequent model training. Components in the pre-set component library Ant Design are likely the smallest particles of a website page, and their source code is high-quality. Therefore, constructing model training data from component source code and component images is highly effective in improving the model.

[0088] Fourth, multimodal representation learning is performed on the model training data based on momentum distillation to remove original noise data in the model training data.

[0089] After obtaining the code as the first text data and the corresponding image page as the first image data, the electronic device can further adopt a momentum distillation self-training method to learn from the network image-text pair data (the first text data and the first image data corresponding to the first text data), thereby removing the original noise data in the model training data.

[0090] In addition, the electronic device can also construct multimodal model training data based on the first text data and the first image data corresponding to the first text data, and then perform multimodal representation learning on the model training data based on momentum distillation, thereby alleviating the noise problem in the original data.

[0091] In some embodiments, the embodiment of the present application is based on the first text data and the first image data to train the artificial intelligence model to obtain a code generation model, and its model architecture is as follows: Figure 2 The code generation model training method provided in the embodiment of the present application is composed of three transformer components: an encoder (E), a denoiser (N) and a decoder (D), plus a classification head (H). Based on this, before the above-mentioned step S102: training a preset artificial intelligence model based on the first text data and the first image data to obtain the code generation model, it can also include the following steps:

[0092] Get the preset AI model.

[0093] Before performing large model pre-training based on the first text data and the first image data, the electronic device first constructs an artificial intelligence model based on the multimodal large language model and the diffusion model.

[0094] It should be noted that the AI ​​model includes an encoder component, a denoiser component, a decoder component, and a classification head. The encoder component is created based on a multimodal large language model, while the denoiser component is created based on a diffusion model.

[0095] In some embodiments, the encoder (E) is a multimodal large language model based on contrastive learning. Unlike some contrastive learning methods in computer vision (CV) such as moco and simclr, the multimodal model training data constructed by the electronic device is a text-image pair: an image (first image data) and its corresponding text description (first text data). Therefore, the electronic device creates an encoder (E) through a multimodal large language model based on contrastive learning, and the encoder (E) can learn the matching relationship between text-image pairs through contrastive learning. For example, Figure 2 As shown, the encoder (E) mainly includes two models: Text Encoder and Image Encoder. The Text Encoder is used to extract the features of the text (first text data) and can adopt the text transformer model commonly used in NLP; while the Image Encoder is used to extract the features of the image (first image data) and can adopt the common CNN model or vision transformer.

[0096] In addition, the denoiser (N) can be a Transformer block with a self-attention mechanism, which can target the Gaussian noise x based on the self-attention mechanism. t and the resultant vector E s Perform cross attention processing and deal with noise xt Perform full self-attention processing. Compared with the traditional text diffusion method that generates tags independently, the electronic device uses full self-attention to make each hidden dimension (d i ) all have complete information about the other dimensions.

[0097] Please refer to Figure 3 , Figure 3 for Figure 1 Schematic diagram of the detailed process flow of step S102.

[0098] like Figure 3 As shown, in some embodiments, the above-mentioned step S102 of "performing model training on a preset artificial intelligence model based on the first text data and the first image data" may include steps S301 to S303 as shown below.

[0099] Step S301: performing comparative learning on the first text data and the first image data based on the encoder component to obtain a synthetic vector representing a matching relationship between the first text data and the first image data.

[0100] When the electronic device inputs multimodal model training data into the artificial intelligence model, thereby training the artificial intelligence model based on the first text data and the first image data to obtain a code generation model, the electronic device first performs comparative learning on the first text data and the first image data corresponding to the first text data based on the encoder component in the artificial intelligence model, thereby obtaining a synthetic vector for characterizing the matching relationship between the first text data and the first image data.

[0101] In some embodiments, the electronic device performs comparative learning on the text features extracted from the first text data and the image features extracted from the first image data through the encoder (E). For a training batch (multimodal model training data) containing N text-image pairs (a first text data and a first image data described by the first text data are a text-image pair), the N text features and the N image features are combined in pairs, and the model predicts N 2 The similarity of possible text-image pairs is calculated directly from the cosine similarity of text features and image features. Figure 2 The matrix shown in . Thus, if there are N positive samples, that is, text and images that truly belong to a pair (diagonal elements in the matrix), and the remaining N 2 —N text-image pairs are negative samples, so the training goal is to maximize the similarity of N positive samples while minimizing N 2—N negative samples’ similarity. The encoder (E) trains a multimodal pre-training neural network (Contrastive Language-Image Pre-Training, CLIP) model, and then multiplies the encoding results of the first text data and the first image data by the weight Wt=0.5 to generate a synthetic vector E n , E n ={e1,e2,e3,...,e n}.

[0102] Step S302: performing diffusion learning on the synthetic vector based on the denoiser component to obtain a diffusion target.

[0103] The electronic device trains the encoder component to generate a synthetic vector E based on the encoder component. n Afterwards, the synthesized vector E is further processed based on the denoiser component. n Perform diffusion learning to obtain the diffusion target.

[0104] In some embodiments, the electronic device, during the process of training the artificial intelligence model, converts the synthesized vector E generated by the encoder component into n Input to the denoiser component, which is then replaced by the denoiser component by superimposing Gaussian noise x t To synthesize vector E n Perform diffusion learning to obtain diffusion targets

[0105] Step S303: generating a hidden representation corresponding to the diffusion target based on the decoder component, and calculating a code label distribution corresponding to the hidden representation based on the classification head.

[0106] The electronic device is training the denoiser component to obtain a diffusion target based on the denoiser component. Afterwards, the diffusion target Input to the decoder component, thereby generating and diffusing the target based on the decoder component The corresponding hidden representation Ds={d1,d2,...,dn}=D(x 0 ,Es), finally, the electronic device projects the hidden representation Ds onto the actual code mark based on the classification head H and calculates the distribution p(y|d i ).

[0107] Please refer to Figure 4 , Figure 4 for Figure 3 Schematic diagram of the detailed process flow of step S302.

[0108] like Figure 4As shown, in some embodiments, the above step S302: performing diffusion learning on the synthetic vector based on the denoiser component to obtain a diffusion target may include steps S401 and S402 as shown below.

[0109] Step S401: gradually superimposing Gaussian noise on the synthetic vector based on the denoiser component.

[0110] During the training of the denoiser component, the denoiser component gradually adds Gaussian noise x to the synthetic vector En generated by the encoder component according to a preset time step t. t .

[0111] It should be noted that, based on different design requirements of actual applications, in different feasible embodiments, the time step t used by the electronic device for the noise denoiser component to superimpose noise simulation expansion process may of course be different. The training method of the code generation model provided in the embodiment of the present application is not limited to the specific size of the time t.

[0112] Step S402: performing self-attention processing on the synthesized vector, the predicted noise corresponding to the synthesized vector, and the Gaussian noise to obtain a diffusion target corresponding to the synthesized vector.

[0113] Since the denoiser (N) can be a Gaussian noise x t and the resultant vector E s Cross attention and noise x t The fully self-attention Transformer block, in the process of training the denoiser component, the electronic device superimposes Gaussian noise x on the synthetic vector En at each time step in the denoiser component. t Afterwards, the denoiser component performs a denoising operation on the synthetic vector En and the predicted noise corresponding to the synthetic vector En. Perform cross attention processing and deal with Gaussian noise x t Perform full self-attention processing to obtain the diffusion target corresponding to the synthetic vector En In this way, the electronic device can calculate the final hidden representation Ds={d1,d2,...,dn}=D(x 0 ,Es).

[0114] In some embodiments, during the training of the denoiser component, the electronic device may use the decoder (D) in the denoiser component to predict the noise based on the self-attention mechanism before the denoiser component maps the denoised embeddings to the discrete code Toekn. and the resultant vector E s Perform cross attention and add Gaussian noise x 0Full self-attention is performed on the hidden representation to calculate the final hidden representation Ds = {d1, d2, ..., dn} = D(x 0 ,Es). Finally, the electronic device uses the classification head H to transform the hidden representation D s Projected onto the actual code mark, calculate the distribution of the code mark p(y|d i ). The electronic device does not search for these tags and selects

[0115] In this embodiment, the first text data and the first image data are obtained by the electronic device to construct multimodal model training data, and then the encoder component in the pre-designed artificial intelligence model is trained using the text-image pair (the first text data and the first image data corresponding to the first text data) to generate a synthetic vector E based on the encoder component. n , then, based on the denoiser component, the synthetic vector E n Perform diffusion learning to obtain the diffusion target Then, the diffusion target is further Input to the decoder component, thereby generating and diffusing the target based on the decoder component The corresponding hidden representation Ds={d1,d2,...,dn}=D(x 0 ,Es), finally, the electronic device projects the hidden representation Ds onto the actual code mark based on the classification head H and calculates the distribution p(y|d i In this way, the electronic device can train a code generation model capable of implementing automated programming based on images and / or text based on model training data including N text-image pairs. This code generation model can then be used to automatically generate code for complex website interfaces, effectively improving code writing efficiency.

[0116] Please refer to Figure 5 , Figure 5 A flowchart of the steps involved in the loss function-based training model of the code generation model training method provided in some embodiments of the present application.

[0117] like Figure 5 As shown, in some embodiments, the training method of the code generation model provided in the embodiments of the present application may also include steps S501 and S502 as shown below.

[0118] Step S501: Obtain a preset model training loss function; wherein the model training loss function is calculated based on the error between the predicted noise and the Gaussian noise, the error between the hidden representation and the embedded code, and a standard cross-entropy loss function; wherein the embedded code is the source code in the first text data.

[0119] In the process of training the artificial intelligence model based on the first text data and the first image data to obtain the code generation model, the electronic device also pre-trains the artificial intelligence model based on the predicted noise The Gaussian noise ∈ t The error between the hidden representation Ds and the embedded code and the standard cross entropy loss function are used to calculate the model training loss function. The embedded code is the source code in the first text data.

[0120] Step S502: training the denoiser component, the decoder component and the classification head based on the model training loss function.

[0121] After obtaining the model training loss function, the electronic device can use the diffusion target based on the model training loss function The three main components are trained: the denoiser component, the decoder component and the classification head.

[0122] It should be noted that the electronic device trains the artificial intelligence model in two stages, namely, an unsupervised pre-training stage of the code snippet and a supervised fine-tuning stage of the encoder component, the denoiser component and the decoder component. The electronic device uses the encoder component to embed the first image data and the first text data into a continuous space, and then adds (and deletes) Gaussian noise ∈ t in the space at time step t. t In addition, the electronic device adapts the loss from GENIE by combining the hidden representation Ds of the decoder component. This allows the model to calculate the loss at time step t using the model training loss function as shown below.

[0123]

[0124] The electronics minimize the noise With the actual noise ∈ t The denoiser component is trained by minimizing the error between the hidden representation D s The error between the and embedding codes is used to train the decoder D and the function L, and the standard cross entropy loss function is applied to the output of the classification head H, which is given by the hidden representation D s and the true code snippet y, generating predicted code tokens.

[0125] In this embodiment, the electronic device first predicts the noise The Gaussian noise ∈ tThe error between the hidden representation Ds and the embedded code and the standard cross entropy loss function are used to calculate the model training loss function. In this way, the electronic device can use the diffusion target to train the three main components of the artificial intelligence model (denoiser, decoder and classification head) based on the model training loss function, and finally obtain a code generation model that can realize automatic programming based on images and / or text.

[0126] In some embodiments, the code generation model training method provided in the embodiments of the present application may further include at least one of the following:

[0127] Pre-training the denoiser component and the decoder component for unsupervised code generation based on preset Gaussian noise;

[0128] Denoiser input data is obtained by calculating a preset code segment based on the encoder component, and pre-training the denoiser component and the decoder component for denoising continuous code segments is performed based on the denoiser input data.

[0129] In the process of training an artificial intelligence model to obtain a code generation model, the electronic device uses two tasks: unsupervised code generation and improvement of continuous paragraph denoising (CPD) of the code, and pre-trains the denoiser (N) and decoder (D) on a code snippet corpus (multimodal model training data). Among them, the code-specific CPD task only masks tokens associated with identifiers or built-in keywords of the target language. The electronic device randomly samples from these two tasks during the pre-training process. Specifically, both the pre-training and fine-tuning tasks use Lt for sampling.

[0130] It should be noted that there is no natural language expression in the pre-training of the denoiser (N) and decoder (D), that is, the denoiser (N) does not have an input synthesis vector E s , so in the unsupervised code generation task, E s is replaced by Gaussian noise sampled at each denoising time step t. In the CPD task, E s It is obtained by passing the masked code y through the encoder (R).

[0131] In some embodiments, after the electronic device completes basic multimodal large model training for the artificial intelligence model to obtain a code generation model, it can also fine-tune the code generation model through multimodal instructions, and further fine-tune the instructions on the basis of modal alignment, so that the code generation model can better complete specific code generation tasks.

[0132] It should be noted that electronic devices use multimodal instruction tuning (M-IT) to tune the code generation model. Instructions refer to the description of a specific code generation task. Multimodal instruction tuning (M-IT) is a technology that fine-tunes a pre-trained MLLM using instruction-formatted data. Through this technology, electronic devices can enable the code generation model to generalize to unseen tasks based on new instructions, thereby improving zero-shot learning performance.

[0133] In some embodiments, the basic form of multimodal instruction data can be summarized as a triplet of (instruction, multimodal input, answer). The specific design of instructions can be divided into two methods: manual design and GPT-assisted design. Manual design refers to manually designing a series of instruction templates for each task, such as for traditional visual question answering tasks, while GPT-assisted design is to manually design a small number of examples to prompt GPT to generate richer instructions.

[0134] In some embodiments, the electronic device can also use Multimodal In-Context Learning (M-ICL) to fine-tune the code generation model. M-ICL involves using a small number of examples as prompt input to stimulate the potential of the code generation model and normalize the model's output.

[0135] In other embodiments, the electronic device may also tune the code generation model by simultaneously adopting multimodal instruction fine-tuning M-IT and multimodal context learning M-ICL, so that the code generation model can better complete specific code generation tasks.

[0136] In this embodiment, an electronic device trains an artificial intelligence model based on the first text data and the first image data to obtain a code generation model, and further fine-tunes the code generation model. Combined with experiments, it is found that the effect obtained by the code generation model with 100M parameters is equivalent to the effect of many models with billions or tens of billions of parameters, such as GPT-4Vision.

[0137] Please refer to Figure 6 , Figure 6 A schematic diagram of the overall process involved in some embodiments of the code model training method provided in the embodiments of the present application.

[0138] like Figure 6As shown, in some embodiments, an electronic device uses large-scale multimodal data to perform model training, resulting in an automatic code generation model based on a multimodal diffusion model, also referred to as a multimodal code diffusion model. In this embodiment, the multimodal code diffusion model is the aforementioned code generation model. The code generation model can comprehensively utilize information from different modalities to perform various cross-modal tasks.

[0139] When the electronic device generates a model for the training code, it first designs a large model architecture according to the scenario. The model architecture can be as described above. Figure 2 As shown, the code generation model includes three core components: encoder (E), denoiser (N), decoder (D), and a classification head (H). Among them, the encoder (E) is a multimodal large language model based on contrastive learning. Unlike some contrastive learning methods in computer vision (CV) such as moco and simclr, the multimodal model training data constructed by electronic devices is a text-image pair: an image (image data) and its corresponding text description (text data). Therefore, the electronic device creates an encoder (E) through a multimodal large language model based on contrastive learning, and the encoder (E) can learn the matching relationship of text-image pairs through contrastive learning. For example, as Figure 2 As shown in the figure, the encoder (E) mainly includes two models: Text Encoder and Image Encoder. The Text Encoder is used to extract the features of text (text data) and can adopt the text transformer model commonly used in NLP; while the Image Encoder is used to extract the features of image (image data) and can adopt the common CNN model or vision transformer.

[0140] In addition, the denoiser (N) can be a Transformer block that can denoise the Gaussian noise x based on the self-attention mechanism. t and the resultant vector E s Perform cross attention processing and noise x t Perform full self-attention processing. Compared with the traditional text diffusion method that generates tags independently, the electronic device uses full self-attention to make each hidden dimension (d i ) all have complete information about the other dimensions.

[0141] After designing the model architecture, the electronic device constructs model training data by collecting data and performing data cleaning operations. The methods for electronic devices to obtain model training data include: 1. Obtaining repositories with more than 1k star counts and providing website links from GitHub, first obtaining the source code of the website for subsequent model training. 2. Using the rankings on Alexa, a global ranking of well-known websites, to capture screenshots and code of the top 1 million websites. In terms of data cleaning, electronic devices use a self-training method called momentum distillation to learn from online graph-text data to alleviate the noise problem in the original data. 3. Find a common component library in the industry and obtain each component and its corresponding source code. These components may be the smallest particles of a page, and the code quality is high, which has a significant effect on improving the model.

[0142] After the electronic device further obtains the model training data, it pre-trains the model by setting tasks. Among them, the electronic device compares and learns the text features extracted from the text data and the image features extracted from the image data through the encoder (E). For a training batch (multimodal model training data) containing N text-image pairs (a text data and an image data described by the text data are a text-image pair), the N text features and N image features are combined in pairs, and the model will predict N 2 The similarity of possible text-image pairs is calculated directly from the cosine similarity of text features and image features. Figure 2 The matrix shown in . Thus, if there are N positive samples, that is, text and images that truly belong to a pair (diagonal elements in the matrix), and the remaining N 2 —N text-image pairs are negative samples, so the training goal is to maximize the similarity of N positive samples while minimizing N 2 —N negative samples’ similarity. The encoder (E) trains a multimodal pre-trained neural network (Contrastive Language-Image Pre-Training, CLIP) model, and then multiplies the encoding results of the text data and image data by a weight of 0.5 to generate a synthetic vector E n , E n ={e1,e2,e3,...,e n}. Afterwards, the denoiser (N) gradually superimposes Gaussian noise xt on the synthetic vector En according to a preset time step t. Since the denoiser (N) can be a Transformer block with cross attention of Gaussian noise xt and synthetic vector Es and full self-attention of noise xt, the electronic device can use the decoder (D) in the denoiser (N) to predict the noise based on the self-attention mechanism before the denoiser (N) maps the denoised embeddings to the discrete code Toekn. and the resultant vector E s Perform cross attention and add Gaussian noise x 0 Full self-attention is performed on the hidden representation to calculate the final hidden representation Ds = {d1, d2, ..., dn} = D(x 0 ,Es). Finally, the electronic device uses the classification head H to transform the hidden representation D s Projected onto the actual code mark, calculate the distribution of the code mark p(y|d i ). The electronic device does not search for these tags and selects

[0143] It is important to note that the electronic device trains the AI ​​model in two phases: an unsupervised pre-training phase for the code snippet and a supervised fine-tuning phase for the encoder component, denoiser component and decoder component. The electronic device uses the encoder (E) to embed the image data and text data into a continuous space, and then adds (and removes) Gaussian noise ∈ t in this space at time step t. t In addition, the electronic device adapts the loss from GENIE by combining the hidden representation Ds of the decoder component. This allows the model to calculate the loss at time step t using the model training loss function as shown below.

[0144]

[0145] The electronics minimize the noise With the actual noise ∈ t The denoiser component is trained by minimizing the error between the hidden representation D s The error between the and embedding codes is used to train the decoders D and L, and the standard cross entropy loss function is applied to the output of the classification head H, which is given by the hidden representation D s and the true code snippet y, generating predicted code tokens.

[0146] In addition, in the process of training the code generation model, the electronic device uses two tasks: unsupervised code generation and improvement of continuous paragraph denoising (CPD) of code, and pre-trains the denoiser (N) and decoder (D) on the code snippet corpus (multimodal model training data). Among them, the code-specific CPD task only masks tokens associated with identifiers or built-in keywords of the target language. The electronic device randomly samples from these two tasks during the pre-training process. Specifically, both the pre-training and fine-tuning tasks are sampled using Lt.

[0147] Since there is no natural language expression in the pre-training of the denoiser (N) and decoder (D), i.e., the denoiser (N) does not have an input synthesis vector E s , so in the unsupervised code generation task, E s is replaced by Gaussian noise sampled at each denoising time step t. In the CPD task, E s It is obtained by passing the masked code y through the encoder (R).

[0148] After pre-training the model, the electronic device further fine-tunes the model through instructions to further improve the model effect. That is, the electronic device tunes the code generation model through multimodal instruction tuning (M-IT) and multimodal in-context learning (M-ICL). Among them, instruction refers to the description of a specific code generation task. Multimodal instruction fine-tuning M-IT is a technology that fine-tunes the pre-trained MLLM through instruction-formatted data. Through this technology, the electronic device can make the code generation model follow new instructions and generalize to unseen tasks, thereby improving zero-shot learning performance. The basic form of multimodal instruction data can be summarized as a triplet of (instruction, multimodal input, answer). The specific design of instructions can be divided into two methods: manual design and GPT-assisted design. Manual design involves manually designing a series of instruction templates for each task, such as traditional visual question answering tasks. GPT-assisted design involves manually designing a small number of examples to prompt GPT to generate richer instructions. Multimodal contextual learning (M-ICL) involves using electronic devices to provide a small number of examples as prompt input, stimulating the potential of the code generation model and normalizing the model's output.

[0149] Finally, the electronic device evaluated the model's experimental results using a test dataset. During this process, the electronic device combined experiments and found that the code generation model, with 100M parameters, achieved performance comparable to models with billions or even tens of billions of parameters, such as GPT-4Vision.

[0150] Please refer to Figure 7 , Figure 7 A flowchart illustrating the steps of the code generation method provided in some embodiments of the present application.

[0151] like Figure 7 As shown, in some embodiments, the electronic device applies the code generation method provided by the embodiments of the present application, which may include step S701 and step S702.

[0152] Step S701: Acquire model input data; wherein the model input data at least includes second image data.

[0153] After the electronic device has trained and obtained the code generation model, it can use the code generation model to perform automated programming operations to generate code. That is, the electronic device first obtains the second image data for which code generation is required, and then uses the second image data as model input data.

[0154] Step S702: inputting the second image data into a code generation model, so as to learn the second image data based on the code generation model and output a code corresponding to the second image data.

[0155] It should be noted that the code generation model is a code generation model obtained by training the artificial intelligence model based on the first text data and the first image data using the training method of the code generation model provided in the above-mentioned embodiment of the present application.

[0156] After obtaining the model input data, the electronic device calls a pre-trained code generation model, thereby inputting the second image data in the model input data into the code generation model, so as to learn based on the code generation model using the encoder second image data, and denoise the synthetic vector representing the matching relationship between the text data (the text data generated by the model to describe the features of the second image data) and the second image data based on the denoiser, and then generates a hidden representation based on the decoder and maps the hidden representation to a specific code based on the classification head and outputs the code, which is the code corresponding to the second image data.

[0157] In some embodiments, after training the code generation model, the electronic device may also simultaneously obtain the second text data for which code generation is required and the second image data having the features described by the second text data, and then form a text-image pair with the second text data and the second image data as model input data. Afterwards, the electronic device calls the pre-trained code generation model, thereby inputting the second text data and the second image data in the model input data into the code generation model, using the encoder to learn the second text data and the second image data based on the code generation model, and using the denoiser to denoise the synthetic vector representing the matching relationship between the second text data and the second image data, and then generating a hidden representation based on the decoder and mapping the hidden representation to a specific code based on the classification head, and outputting the code corresponding to the second image data.

[0158] In an embodiment of the present application, by expanding the diffusion model to multimodality and applying it to the field of code generation, an electronic device combines a multimodal large language model with a diffusion model training to obtain a code generation model based on multimodality and diffusion, and then uses the code generation model to learn model input data including at least image data (such as a complex website interface, etc.), thereby automatically generating code corresponding to the image data. In this way, compared with the method of manually writing code data corresponding to a complex website interface, the embodiment of the present application effectively improves the ability of the code generation model to realize automated programming based on images or text, and broadens the scope of automated programming based on artificial intelligence.

[0159] The present application also provides a code generation model training device that can implement the above-mentioned code generation model training method. The device includes a first acquisition module and a model training module.

[0160] A first acquisition module is configured to acquire multimodal model training data; wherein the multimodal model training data includes first text data and first image data, and the first text data is data for describing features of the first image data;

[0161] A model training module, configured to perform model training on a preset artificial intelligence model based on the first text data and the first image data to obtain a code generation model;

[0162] Among them, the artificial intelligence model is constructed by combining a multimodal large language model and a diffusion model; the code generation model is used to learn at least the second image data and output a code corresponding to the second image data.

[0163] In some embodiments, the first acquisition module is further used to acquire a preset artificial intelligence model;

[0164] Among them, the artificial intelligence model includes an encoder component, a denoiser component, a decoder component and a classification head; the encoder component is created based on a multimodal large language model, and the denoiser component is created based on a diffusion model.

[0165] In some embodiments, the model training module is also used to perform comparative learning on the first text data and the first image data based on the encoder component to obtain a synthetic vector representing the matching relationship between the first text data and the first image data; perform diffusion learning on the synthetic vector based on the denoiser component to obtain a diffusion target; and generate a hidden representation corresponding to the diffusion target based on the decoder component, and calculate the code tag distribution corresponding to the hidden representation based on the classification head.

[0166] In some embodiments, the model training module is also used to gradually superimpose Gaussian noise on the synthetic vector based on the denoiser component; and to perform self-attention processing on the synthetic vector, the predicted noise corresponding to the synthetic vector and the Gaussian noise to obtain a diffusion target corresponding to the synthetic vector.

[0167] In some embodiments, the model training module is also used to obtain a preset model training loss function; wherein, the model training loss function is calculated based on the error between the predicted noise and the Gaussian noise, the error between the hidden representation and the embedded code, and the standard cross-entropy loss function; wherein, the embedded code is the source code in the first text data; and, the denoiser component, the decoder component and the classification head are trained based on the model training loss function.

[0168] In some embodiments, the model training module is also used to pre-train the denoiser component and the decoder component for unsupervised code generation based on preset Gaussian noise; and to calculate a preset code fragment based on the encoder component to obtain denoiser input data, and pre-train the denoiser component and the decoder component for denoising continuous code paragraphs based on the denoiser input data.

[0169] In some embodiments, the first acquisition module is further used to obtain website source code from a preset code hosting platform as first text data, and obtain a website interface corresponding to the website source code as first image data, so as to construct multimodal model training data; determine a target website that meets the preset website ranking conditions, crawl the code of the target website as the first text data, and crawl the website screenshot corresponding to the code of the target website as the first image data, so as to construct multimodal model training data; obtain component source code from a preset component library as the first text data, and obtain a component image corresponding to the component source code as the first image data, so as to construct multimodal model training data; and perform multimodal representation learning on the model training data based on momentum distillation to remove original noise data in the model training data.

[0170] The specific implementation of the code generation model training device provided in the embodiment of the present application is basically the same as the specific implementation of the above-mentioned code generation model training method, and will not be repeated here.

[0171] The embodiment of the present application also provides a code generation device that can implement the above code generation method. The device includes a second acquisition module and a code generation module.

[0172] A second acquisition module is used to acquire model input data; wherein the model input data at least includes second image data;

[0173] A code generation module is used to input the second image data into the code generation model as described in the first aspect above, so as to learn the second image data based on the code generation model and output a code corresponding to the second image data.

[0174] The specific implementation of the code generation device provided in the embodiment of the present application is basically the same as the specific embodiment of the above-mentioned code generation method, and will not be repeated here.

[0175] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-mentioned code generation model training method. The electronic device can be any smart terminal including a tablet computer, an in-vehicle computer, or the like.

[0176] See also Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0177] The processor 801 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0178] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the training method of the code generation model of the embodiment of this application;

[0179] Input / output interface 803, used to implement information input and output;

[0180] Communication interface 804, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0181] Bus 805 , which transmits information between various components of the device (e.g., processor 801 , memory 802 , input / output interface 803 , and communication interface 804 );

[0182] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0183] An embodiment of the present application also provides a vehicle, which is equipped with an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned code generation model training method when executing the computer program.

[0184] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, implements the training method of the above-mentioned code generation model.

[0185] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0186] An embodiment of the present application also provides a computer program product, including a computer program. The steps implemented when the computer program is executed by a processor are basically the same as the specific embodiments of the training method of the above-mentioned code generation model, and will not be repeated here.

[0187] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0188] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0189] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0190] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0191] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0192] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0193] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0194] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0196] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0197] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for training a code generation model, characterized in that: The method comprises: Acquire multimodal model training data; wherein the multimodal model training data includes first text data and first image data, the first text data is data for describing features of the first image data, the first text data is a code, and the first image data is an interface image corresponding to the code; Performing model training on a preset artificial intelligence model based on the first text data and the first image data to obtain a trained code generation model; Among them, the artificial intelligence model is constructed by combining a multimodal large language model and a diffusion model; the trained code generation model is used to process the input second image data and output a code corresponding to the second image data.

2. The method according to claim 1, characterized in that Before performing model training on a preset artificial intelligence model based on the first text data and the first image data to obtain a trained code generation model, the method further includes: Get the preset artificial intelligence model; Among them, the artificial intelligence model includes an encoder component, a denoiser component, a decoder component and a classification head; the encoder component is created based on a multimodal large language model, and the denoiser component is created based on a diffusion model.

3. The method according to claim 2, characterized in that The performing model training on a preset artificial intelligence model based on the first text data and the first image data includes: performing comparative learning on the first text data and the first image data based on the encoder component to obtain a synthetic vector representing a matching relationship between the first text data and the first image data; Performing diffusion learning on the synthetic vector based on the denoiser component to obtain a diffusion target; A hidden representation corresponding to the diffusion target is generated based on the decoder component, and a code label distribution corresponding to the hidden representation is calculated based on the classification head.

4. The method according to claim 3, characterized in that The performing diffusion learning on the synthetic vector based on the denoiser component to obtain a diffusion target includes: gradually superimposing Gaussian noise on the synthetic vector based on the denoiser component; Self-attention processing is performed on the synthetic vector, the predicted noise corresponding to the synthetic vector, and the Gaussian noise to obtain a diffusion target corresponding to the synthetic vector.

5. The method according to claim 4, characterized in that The method further comprises: Obtaining a preset model training loss function; wherein the model training loss function is calculated based on the error between the predicted noise and the Gaussian noise, the error between the hidden representation and the embedded code, and a standard cross entropy loss function; wherein the embedded code is the source code in the first text data; The denoiser component, the decoder component and the classification head are trained based on the model training loss function.

6. The method according to claim 2, characterized in that The method further comprises at least one of the following: Pre-training the denoiser component and the decoder component for unsupervised code generation based on preset Gaussian noise; Denoiser input data is obtained by calculating a preset code segment based on the encoder component, and pre-training the denoiser component and the decoder component for denoising continuous code segments is performed based on the denoiser input data.

7. The method according to any one of claims 1 to 6, characterized in that The obtaining of multimodal model training data includes at least one of the following: Obtaining website source code from a preset code hosting platform as first text data, and obtaining a website interface corresponding to the website source code as first image data, so as to construct multimodal model training data; Determining a target website that meets a preset website ranking condition, crawling the code of the target website as first text data, and crawling a website screenshot corresponding to the code of the target website as first image data, so as to construct multimodal model training data; Obtaining component source code from a preset component library as first text data, and obtaining a component image corresponding to the component source code as first image data, so as to construct multimodal model training data; Multimodal representation learning is performed on the model training data based on momentum distillation to remove original noise data in the model training data.

8. A code generation method, characterized in that: The method comprises: Acquiring model input data; wherein the model input data includes second image data; The second image data is input into the trained code generation model according to any one of claims 1 to 7, so as to process the second image data based on the trained code generation model and output a code corresponding to the second image data.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the training method of the code generation model described in any one of claims 1 to 7, and / or implements the code generation method described in claim 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the code generation model training method described in any one of claims 1 to 7, and / or implements the code generation method described in claim 8.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the code generation model training method described in any one of claims 1 to 7, and / or implements the code generation method described in claim 8.

Images