Training method of code generation model, code generation method, equipment and medium

Abstract

The invention provides a code generation model training method, a code generation method, equipment and a medium, and belongs to the technical field of artificial intelligence. Wherein the multi-modal model training data comprises first text data and first image data, and the first text data is data for performing feature description on the first image data; performing model training on a preset artificial intelligence model based on the first text data and the first image data to obtain a code generation model; wherein the artificial intelligence model is constructed by combining a multi-modal large language model and a diffusion model; the code generation model is used for at least learning the second image data and outputting codes corresponding to the second image data. According to the method and the device, the automatic programming capability of the code generation model based on artificial intelligence on the basis of images or characters can be improved, so that the code generation model can automatically generate codes of a website interface with a complex structure, and the code writing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a code generation model training method, a code generation method, a device and a medium. Background Art

[0002] With the continuous development and progress of artificial intelligence technology, automated programming with the help of large language models (LLM) has become a reality. In related technologies, large language model automated programming can only automatically write code at the level of simple functions according to text prompts. For website interfaces with complex structures that are difficult to describe clearly in words, it still takes a lot of time and human resources to manually write code data corresponding to the website interface, which results in very low efficiency in writing code for image data similar to website interfaces. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a training method, a code generation method, a device and a medium for a code generation model, aiming to enhance the automated programming capability of the code generation model so that the code generation model can automatically generate code for a website interface with a complex structure, thereby improving the efficiency of code writing.

[0004] To achieve the above object, a first aspect of an embodiment of the present application proposes a method for training a code generation model, the method comprising:

[0005] Acquire multimodal model training data; wherein the multimodal model training data includes first text data and first image data, and the first text data is data for describing features of the first image data;

[0006] Performing model training on a preset artificial intelligence model based on the first text data and the first image data to obtain a code generation model;

[0007] Among them, the artificial intelligence model is constructed by combining a multimodal large language model and a diffusion model; the code generation model is used to learn at least the second image data and output a code corresponding to the second image data.

[0008] In some embodiments, before performing model training on a preset artificial intelligence model based on the first text data and the first image data to obtain a code generation model, the method further includes:

[0009] Get the preset artificial intelligence model;

[0010] Among them, the artificial intelligence model includes an encoder component, a denoiser component, a decoder component and a classification head; the encoder component is created based on a multimodal large language model, and the denoiser component is created based on a diffusion model.

[0011] In some embodiments, the performing model training on a preset artificial intelligence model based on the first text data and the first image data includes:

[0012] Performing comparative learning on the first text data and the first image data based on the encoder component to obtain a synthetic vector representing a matching relationship between the first text data and the first image data;

[0013] Performing diffusion learning on the synthetic vector based on the denoiser component to obtain a diffusion target;

[0014] A hidden representation corresponding to the diffusion target is generated based on the decoder component, and a code label distribution corresponding to the hidden representation is calculated based on the classification head.

[0015] In some embodiments, the performing diffusion learning on the synthetic vector based on the denoiser component to obtain a diffusion target comprises:

[0016] gradually superimposing Gaussian noise on the synthetic vector based on the denoiser component;

[0017] Self-attention processing is performed on the synthetic vector, the predicted noise corresponding to the synthetic vector, and the Gaussian noise to obtain a diffusion target corresponding to the synthetic vector.

[0018] In some embodiments, the method further comprises:

[0019] Obtaining a preset model training loss function; wherein the model training loss function is calculated based on the error between the predicted noise and the Gaussian noise, the error between the hidden representation and the embedded code, and a standard cross entropy loss function; wherein the embedded code is the source code in the first text data;

[0020] The denoiser component, the decoder component and the classification head are trained based on the model training loss function.

[0021] In some embodiments, the method further comprises at least one of the following:

[0022] Pre-training the denoiser component and the decoder component for unsupervised code generation based on preset Gaussian noise;

[0023] The encoder component calculates a preset code segment to obtain denoiser input data, and the denoiser component and the decoder component are pre-trained to denoise continuous code paragraphs based on the denoiser input data.

[0024] In some embodiments, the acquiring of multimodal model training data includes at least one of the following:

[0025] Acquire website source code from a preset code hosting platform as first text data, and acquire a website interface corresponding to the website source code as first image data, so as to construct multimodal model training data;

[0026] Determine a target website that meets a preset website ranking condition, capture the code of the target website as first text data, and capture a website screenshot corresponding to the code of the target website as first image data, so as to construct multimodal model training data;

[0027] Acquire component source code from a preset component library as first text data, and acquire a component image corresponding to the component source code as first image data, so as to construct multimodal model training data;

[0028] Multimodal representation learning is performed on the model training data based on momentum distillation to remove original noise data in the model training data.

[0029] To achieve the above object, a second aspect of an embodiment of the present application proposes a code generation method, the method comprising:

[0030] Acquire model input data; wherein the model input data at least includes second image data;

[0031] The second image data is input into the code generation model as described in the first aspect above, so as to learn the second image data based on the code generation model and output a code corresponding to the second image data.

[0032] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application proposes a training device for a code generation model, the device comprising:

[0033] A first acquisition module, used to acquire multimodal model training data; wherein the multimodal model training data includes first text data and first image data, and the first text data is data for describing features of the first image data;

[0034] A model training module is used to perform model training on a preset artificial intelligence model based on the first text data and the first image data to obtain a code generation model; wherein the artificial intelligence model is constructed in combination with a multimodal large language model and a diffusion model; the code generation model is used to learn at least the second image data and output a code corresponding to the second image data.

[0035] To achieve the above-mentioned purpose, a fourth aspect of the embodiments of the present application provides a code generation device, the device comprising:

[0036] A second acquisition module, used to acquire model input data; wherein the model input data at least includes second image data;

[0037] A code generation module is used to input the second image data into the code generation model as described in the first aspect above, so as to learn the second image data based on the code generation model and output a code corresponding to the second image data.

[0038] To achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, and the memory stores a computer program. When the electronic device is applied to a vehicle, the processor implements the method described in the first aspect when executing the computer program, and / or the processor implements the method described in the second aspect when executing the computer program.

[0039] To achieve the above-mentioned purpose, the sixth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect above, and / or implements the method described in the second aspect above.

[0040] To achieve the above-mentioned purpose, the seventh aspect of an embodiment of the present application proposes a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the method provided in the first aspect above is implemented, and / or the method provided in the second aspect above is implemented.

[0041] The embodiment of the present application obtains an artificial intelligence model by combining a multimodal large language model and a diffusion model, and then obtains multimodal model training data (including first text data and first image data, the first text data is data for describing the characteristics of the first image data), so as to train the preset artificial intelligence model based on the first text data and the first image data to obtain a code generation model. That is, the embodiment of the present application combines the multimodal large language model and the diffusion model training to obtain a code generation model based on multimodality and diffusion, so as to use the code generation model to learn the second image data and output the code corresponding to the second image data. In this way, compared with the traditional method of manually writing code data corresponding to a complex website interface, the embodiment of the present application can enhance the ability of the code generation model based on artificial intelligence to realize automated programming based on images or text. The code generation model obtained by adopting the training method of the code generation model provided by the embodiment of the present application can automatically generate code for a website interface with a complex structure, thereby improving the efficiency of code writing. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of the steps of the code generation model training method provided in some embodiments of the present application;

[0043] Figure 2 A schematic diagram of a model architecture involved in a training method for a code generation model provided in an embodiment of the present application;

[0044] Figure 3 for Figure 1 Schematic diagram of detailed process flow of step S102;

[0045] Figure 4 for Figure 3 Schematic diagram of detailed process flow of step S302;

[0046] Figure 5 A schematic flow chart of the steps of training a model based on a loss function involved in some embodiments of the code generation model training method provided in the embodiments of the present application;

[0047] Figure 6 A schematic diagram of the overall process involved in some embodiments of the code model training method provided in the embodiments of the present application;

[0048] Figure 7 A schematic diagram of the steps of the code generation method provided in some embodiments of the present application;

[0049] Figure 8 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0053] First, the professional and technical terms involved in the embodiments of the present application are briefly explained.

[0054] Multi-modal Large Language Model (MLLM).

[0055] MLLM is usually based on the Large Language Model (LLM) and incorporates other non-text modal information to complete various multimodal tasks. MLLM can create stories based on images without the need for mathematical reasoning such as optical character recognition (OCR), which provides a potential path to achieve artificial general intelligence (AGI).

[0056] Compared with unimodal LLM, MLLM is more in line with the human habit of understanding the world. Because humans have multiple senses to receive multimodal information, this information is usually complementary and synergistic. Therefore, using multimodal information can generally better understand and complete tasks. In addition, MLLM has a more powerful and user-friendly interface than unimodal LLM. Since MLLM supports multimodal input, users can input and convey information to MLLM in a more flexible way. Furthermore, MLLM has a wider range of task support. Unimodal LLM can usually only complete tasks related to pure text, while MLLM can complete more additional tasks through multimodality, such as image description and visual knowledge question and answer.

[0057] Diffusion Models.

[0058] The diffusion model has shown excellent performance in image generation, and its application areas are still expanding. The diffusion model mainly uses an embedding layer to convert discrete tokens into continuous embeddings, and adds Gaussian noise in the process to imitate the expansion process, and then selects the vocabulary token closest to the embedding to map the denoised embeddings back to discrete text.

[0059] Next, the overall concept of the embodiment of the present application is described.

[0060] With the continuous development and progress of artificial intelligence technology, automated programming with the help of LLM has become a reality. In related technologies, large language model automated programming can only provide intelligent prompts and write simple functions according to the correct prompt word prompt, while complex website interfaces are difficult to describe clearly with simple text. Therefore, for such complex website interfaces that are difficult to describe clearly with text, it still takes a lot of time and human resources to manually write the code data corresponding to the website interface, which leads to very low efficiency in writing code for image data of similar website interfaces.

[0061] Based on this, the embodiments of the present application provide a training method for a code generation model, a code generation method, an electronic device, a computer-readable storage medium, and a computer program product, which aim to enhance the automated programming capability of the code generation model, so that the code generation model can automatically generate code for a website interface with a complex structure, thereby improving the efficiency of code writing.

[0062] The embodiment of the present application obtains an artificial intelligence model by combining a multimodal large language model and a diffusion model, and then obtains multimodal model training data (including first text data and first image data, the first text data is data for describing the characteristics of the first image data), so as to perform model training on the preset artificial intelligence model based on the first text data and the first image data, and obtain a code generation model. That is, the embodiment of the present application applies the diffusion model to the field of code generation, and obtains a code generation model based on multimodality and diffusion by combining a multimodal large language model and a diffusion model training, so as to use the code generation model to learn the second image data and output the code corresponding to the second image data. In this way, the ability of the code generation model based on artificial intelligence to realize automated programming based on images or text can be improved. Compared with the traditional method of manually writing code data corresponding to a complex website interface, the code generation model obtained by adopting the model training method of the embodiment of the present application can automatically generate code for a website interface with a complex structure, thereby improving the efficiency of code writing.

[0063] Next, the training method of the code generation model, the code generation method, the device, the electronic device, the computer-readable storage medium and the computer program product provided in the embodiments of the present application are specifically described through the following embodiments, and the training method of the code generation model and the code generation method provided in the embodiments of the present application are first described in detail.

[0064] It should be noted that the training method and code generation method of the code generation model provided in the embodiment of the present application relate to the field of artificial intelligence technology. The training method and code generation method of the code generation model provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be an electronic device such as a smart phone, a tablet computer, a laptop computer, a desktop computer, etc. The server side can be a server terminal device, can be configured as an independent physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms and other basic cloud computing services. The software can be an application, a computer program, and a storage medium carrying the computer program, etc., to implement the training method and / or code generation method of the code generation model. It should be understood that, based on different design requirements of actual applications, in different feasible embodiments, the training method of the code generation model and / or the terminal, server side, and software of the code generation method provided in the embodiments of the present application may of course also be in other forms not listed here. The training method of the code generation model and the code generation method provided in the embodiments of the present application are not specifically limited to this.

[0065] In addition, the present application can also be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, personal computers (PC), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0066] For ease of understanding and explanation, the following text takes the training method and code generation method of the code generation model provided by the electronic device application embodiment of the present application as an example to describe the various specific embodiments of the present application in detail. The implementation of the training method and code generation method of the code generation model provided by the embodiment of the present application in any of the above-mentioned forms of the subject application can refer to the process of the training method and code generation method of the code generation model applied by the electronic device described later.

[0067] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0068] Please refer to Figure 1 , Figure 1 A flowchart of the steps of the code generation model training method provided in some embodiments of the present application. It should be understood that although Figure 1 The subsequent flowcharts of other steps show the execution order of some method steps, but based on the different design requirements of actual applications, the training method of the code generation model provided in the embodiment of the present application can certainly adopt an execution order different from the method steps shown in the figure. That is, Figure 1The order of the steps of the method shown does not constitute a limitation on the execution logic order of the training method of the code generation model provided in the embodiment of the present application. Figure 1 Reasonable changes in the order of the method steps shown should be included in the protection scope of the code generation model training method provided in the embodiments of the present application.

[0069] like Figure 1 As shown, in some embodiments, the vehicle application code generation model training method provided by the embodiment of the present application may include step S101 and step S102.

[0070] Step S101: Acquire multimodal model training data; wherein the multimodal model training data includes first text data and first image data, and the first text data is data for describing features of the first image data.

[0071] When the electronic device is training a large model, it first collects multimodal model training data. When the electronic device acquires the model training data, in addition to acquiring the first text data, it also acquires the first image data having the features described by the first text data. In this way, the electronic device can construct the multimodal model training data based on the first text data and the first image data.

[0072] It should be noted that the multimodal training data may be large model training data including a plurality of text-image pairs, wherein each text-image pair is a first image data and a first text data for describing features of the first image data.

[0073] Step S102: Based on the first text data and the first image data, a preset artificial intelligence model is trained to obtain a code generation model; wherein the artificial intelligence model is constructed in combination with a multimodal large language model and a diffusion model; the code generation model is used to learn at least the second image data and output a code corresponding to the second image data.

[0074] After obtaining the multimodal model training data, the electronic device calls the artificial intelligence model pre-constructed by combining the multimodal large language model and the diffusion model, and then uses the multimodal model training data as the input of the artificial intelligence model, thereby training the artificial intelligence model based on the first text data and the first image data in the multimodal model training data. In this way, after the artificial intelligence model is trained based on the first text data and the first image data, the electronic device can obtain a code generation model for subsequent automated programming based on the code generation model to generate code, that is, using the code generation model to learn at least the second image data, thereby outputting code corresponding to the second image data.

[0075] In the embodiment of the present application, an artificial intelligence model is constructed by combining an electronic device with a multimodal large language model and a diffusion model, and by obtaining multimodal model training data (including first text data and first image data, the first text data is data for describing the features of the first image data), a preset artificial intelligence model is trained based on the first text data and the first image data to obtain a code generation model. That is, the embodiment of the present application applies the diffusion model to the field of code generation, and obtains a code generation model based on multimodality and diffusion by combining the multimodal large language model and the diffusion model training, so as to use the code generation model to learn the second image data and output the code corresponding to the second image data.

[0076] In this way, compared with the traditional method of manually writing code data corresponding to complex website interfaces, the embodiments of the present application can improve the ability of the artificial intelligence-based code generation model to realize automated programming based on images or text. The code generation model obtained by the training method of the code generation model provided by the embodiments of the present application can automatically generate code for website interfaces with complex structures, thereby improving the efficiency of code writing.

[0077] It should be noted that the multimodal pre-training model is a large model pre-trained on large-scale data of multiple modalities, which can comprehensively utilize information from different modalities to perform various cross-modal tasks. Based on this, the electronic device first needs to obtain multimodal model training data, and then pre-train the model based on the model training data to obtain a code generation model.

[0078] In some embodiments, the above step S101: obtaining multimodal model training data includes at least one of the following:

[0079] First: obtaining website source code from a preset code hosting platform as first text data, and obtaining a website interface corresponding to the website source code as first image data, so as to construct multimodal model training data.

[0080] It should be noted that the preset code hosting platform can be GitHub.

[0081] The electronic device can obtain a repository with more than 1k star numbers and a website link from GitHub, and first obtain the website source code as the first text data, and obtain the website interface corresponding to the website source code as the first image data, thereby constructing multimodal model training data for subsequent model training.

[0082] Second: determine a target website that meets the preset website ranking conditions, capture the code of the target website as the first text data, and capture the website screenshot corresponding to the code of the target website as the first image data, so as to construct multimodal model training data.

[0083] It should be noted that the preset website ranking condition may be that the website is ranked in the top 1 million and so on.

[0084] The electronic device determines the top 1 million target websites through the rankings on Alexa, a world-renowned website ranking list, and then captures the code of the target website as the first text data, and simultaneously captures the website screenshot corresponding to the code of the target website as the first image data, thereby constructing multimodal model training data for subsequent model training.

[0085] Third: obtaining a component source code from a preset component library as first text data, and obtaining a component image corresponding to the component source code as first image data, so as to construct multimodal model training data.

[0086] It should be noted that the preset component library may be Ant Design.

[0087] The electronic device can also obtain component source code as the first text data from the industry-wide component library Ant Design, and obtain component images corresponding to the component source code as the first image data, thereby constructing multimodal model training data for subsequent model training. Among them, the components in the preset component library Ant Design may be the smallest particles of the website page, and the quality of their source code is high, so constructing model training data with component source code and component images is very effective in improving the model.

[0088] Fourth, multimodal representation learning is performed on the model training data based on momentum distillation to remove original noise data in the model training data.

[0089] After obtaining the code as the first text data and the corresponding image page as the first image data, the electronic device can further adopt a momentum distillation self-training method to learn from the network image-text pair data (the first text data and the first image data corresponding to the first text data), thereby removing the original noise data in the model training data.

[0090] In addition, after constructing multimodal model training data based on the first text data and the first image data corresponding to the first text data, the electronic device can also perform multimodal representation learning on the model training data based on momentum distillation, thereby alleviating the noise problem in the original data.

[0091] In some embodiments, the embodiment of the present application trains an artificial intelligence model based on the first text data and the first image data to obtain a code generation model, and its model architecture is as follows: Figure 2 The code generation model training method provided in the embodiment of the present application may further include the following steps before the above-mentioned step S102: training a preset artificial intelligence model based on the first text data and the first image data to obtain the code generation model:

[0092] Get a preset AI model.

[0093] Before performing large model pre-training based on the first text data and the first image data, the electronic device first constructs an artificial intelligence model based on the multimodal large language model and the diffusion model.

[0094] It should be noted that the artificial intelligence model includes an encoder component, a denoiser component, a decoder component and a classification head. Among them, the encoder component is created based on a multimodal large language model, and the denoiser component is created based on a diffusion model.

[0095] In some embodiments, the encoder (E) is a multimodal large language model based on contrastive learning. Unlike some contrastive learning methods in computer vision (CV) such as moco and simclr, the multimodal model training data constructed by the electronic device is a text-image pair: an image (first image data) and its corresponding text description (first text data). Therefore, the electronic device creates an encoder (E) through a multimodal large language model based on contrastive learning, and the encoder (E) can learn the matching relationship between text-image pairs through contrastive learning. For example, Figure 2 As shown, the encoder (E) mainly includes two models: Text Encoder and Image Encoder, wherein the Text Encoder is used to extract the features of the text (first text data), and the text transformer model commonly used in NLP can be used; and the Image Encoder is used to extract the features of the image (first image data), and the commonly used CNN model or vision transformer can be used.

[0096] In addition, the denoiser (N) can be a Transformer block with a self-attention mechanism. The denoiser (N) can target the Gaussian noise x based on the self-attention mechanism. t and the resultant vector E s Perform cross attention processing and deal with noise xt Compared with the traditional text diffusion method that generates tags independently, the electronic device uses full self-attention to make each hidden dimension (d i ) have complete information about the other dimensions.

[0097] Please refer to Figure 3 , Figure 3 for Figure 1 Schematic diagram of the detailed step flow of step S102.

[0098] like Figure 3 As shown, in some embodiments, the above-mentioned step S102 of "performing model training on a preset artificial intelligence model based on the first text data and the first image data" may include steps S301 to S303 as shown below.

[0099] Step S301: performing comparative learning on the first text data and the first image data based on the encoder component to obtain a synthetic vector representing a matching relationship between the first text data and the first image data.

[0100] When the electronic device inputs multimodal model training data into the artificial intelligence model, thereby training the artificial intelligence model based on first text data and first image data to obtain a code generation model, the electronic device first performs comparative learning on the first text data and the first image data corresponding to the first text data based on the encoder component in the artificial intelligence model, thereby obtaining a synthetic vector for characterizing the matching relationship between the first text data and the first image data.

[0101] In some embodiments, the electronic device performs comparative learning on the text features extracted from the first text data and the image features extracted from the first image data through the encoder (E). For a training batch (multimodal model training data) containing N text-image pairs (a first text data and a first image data described by the first text data are a text-image pair), the N text features and the N image features are combined in pairs, and the model predicts N 2 The similarity of possible text-image pairs is calculated by directly calculating the cosine similarity of text features and image features. Figure 2 In this way, if there are N positive samples, that is, text and images that truly belong to a pair (diagonal elements in the matrix), and the remaining N 2 —N text-image pairs are negative samples, so the training goal is to maximize the similarity of N positive samples while minimizing N 2—Similarity of N negative samples. The encoder (E) trains a multimodal pre-trained neural network (Contrastive Language-Image Pre-Training, CLIP) model, and then multiplies the encoding results of the first text data and the first image data by the weight Wt=0.5 to generate a synthetic vector E n , E n ={e1,e2,e3,...,e n}.

[0102] Step S302: performing diffusion learning on the synthetic vector based on the denoiser component to obtain a diffusion target.

[0103] The electronic device trains the encoder component to generate a synthetic vector E based on the encoder component n Afterwards, the synthesized vector E is further processed based on the denoiser component. n Perform diffusion learning to obtain the diffusion target.

[0104] In some embodiments, the electronic device, during the training of the artificial intelligence model, converts the synthesized vector E generated by the encoder component into n Input to the denoiser component, which is then replaced by the denoiser component by superimposing Gaussian noise x t To synthesize the vector E n Perform diffusion learning to obtain diffusion targets

[0105] Step S303: generating a hidden representation corresponding to the diffusion target based on the decoder component, and calculating a code tag distribution corresponding to the hidden representation based on the classification head.

[0106] The electronic device is training the denoiser component to obtain a diffusion target based on the denoiser component. After that, the diffusion target Input to the decoder component, thereby generating and diffusing the target based on the decoder component The corresponding hidden representation Ds = {d1, d2, ..., dn} = D(x 0 ,Es), and finally, the electronic device projects the hidden representation Ds onto the actual code mark based on the classification head H and calculates the distribution p(y|d i ).

[0107] Please refer to Figure 4 , Figure 4 for Figure 3 Schematic diagram of the detailed step flow of step S302.

[0108] like Figure 4As shown, in some embodiments, the above step S302: performing diffusion learning on the synthetic vector based on the denoiser component to obtain a diffusion target may include steps S401 and S402 as shown below.

[0109] Step S401: gradually superimposing Gaussian noise on the synthetic vector based on the denoiser component.

[0110] When the electronic device is training the denoiser component, the denoiser component gradually superimposes Gaussian noise x on the synthetic vector En generated by the encoder component according to a preset time step t. t .

[0111] It should be noted that, based on different design requirements of actual applications, in different feasible embodiments, the time step t used by the electronic device for the denoiser component to superimpose noise simulation expansion process may of course be different. The training method of the code generation model provided in the embodiment of the present application is not limited to the specific size of the time t.

[0112] Step S402: performing self-attention processing on the synthetic vector, the predicted noise corresponding to the synthetic vector, and the Gaussian noise to obtain a diffusion target corresponding to the synthetic vector.

[0113] Since the denoiser (N) can be a Gaussian noise x t and the resultant vector E s Cross-Attention and Noise x t The fully self-attention Transformer block, the electronic device, in the process of training the denoiser component, superimposes Gaussian noise x on the synthetic vector En at each time step in the denoiser component t Afterwards, the denoiser component performs a denoising operation on the synthetic vector En and the predicted noise corresponding to the synthetic vector En. Perform cross attention processing and deal with Gaussian noise x t Perform full self-attention processing to obtain the diffusion target corresponding to the synthetic vector En In this way, the electronic device can calculate the final hidden representation Ds={d1,d2,...,dn}=D(x 0 ,Es).

[0114] In some embodiments, during the training of the denoiser component, the electronic device may use a decoder (D) in the denoiser component to predict the noise based on a self-attention mechanism before the denoiser component maps the denoised Embeddings to a discrete code Toekn. and the resultant vector E s Perform cross attention and add Gaussian noise x 0Full self-attention is performed on the hidden representation Ds = {d1, d2, ..., dn} = D(x 0 ,Es). Finally, the electronic device uses the classification head H to transform the hidden representation D s Project it onto the actual code mark and calculate the distribution of the code mark p(y|d i ). The electronic device does not search for these tags and selects

[0115] In this embodiment, the first text data and the first image data are acquired by the electronic device to construct multimodal model training data, and then the encoder component in the pre-designed artificial intelligence model is trained using the text-image pair (the first text data and the first image data corresponding to the first text data) to generate a synthetic vector E based on the encoder component. n , then, based on the denoiser component, the synthetic vector E n Perform diffusion learning to obtain the diffusion target The diffusion target is then further Input to the decoder component, thereby generating and diffusing the target based on the decoder component The corresponding hidden representation Ds = {d1, d2, ..., dn} = D(x 0 ,Es), and finally, the electronic device projects the hidden representation Ds onto the actual code mark based on the classification head H and calculates the distribution p(y|d i ). In this way, the electronic device can be trained based on the model training data including N text-image pairs to obtain a code generation model that can realize automatic programming based on images and / or texts, so that the code generation model can be used to automatically generate codes for website interfaces with complex structures, effectively improving the efficiency of code writing.

[0116] Please refer to Figure 5 , Figure 5 A flowchart of the steps of training a model based on a loss function involved in some embodiments of the code generation model training method provided in the embodiments of the present application.

[0117] like Figure 5 As shown, in some embodiments, the code generation model training method provided in the embodiments of the present application may also include step S501 and step S502 as shown below.

[0118] Step S501: Obtain a preset model training loss function; wherein the model training loss function is calculated based on the error between the predicted noise and the Gaussian noise, the error between the hidden representation and the embedded code, and a standard cross entropy loss function; wherein the embedded code is the source code in the first text data.

[0119] In the process of training the artificial intelligence model based on the first text data and the first image data to obtain the code generation model, the electronic device also pre-trains the artificial intelligence model based on the predicted noise The Gaussian noise ∈ t The error between the hidden representation Ds and the embedded code and the standard cross entropy loss function are used to calculate the model training loss function. The embedded code is the source code in the first text data.

[0120] Step S502: training the denoiser component, the decoder component and the classification head based on the model training loss function.

[0121] After obtaining the model training loss function, the electronic device can use the diffusion target based on the model training loss function The three main components are trained: the denoiser component, the decoder component and the classification head.

[0122] It should be noted that the electronic device trains the artificial intelligence model in two stages, wherein the two stages are: an unsupervised pre-training stage of the code snippet and a supervised fine-tuning stage of the encoder component, the denoiser component and the decoder component. The electronic device uses the encoder component to embed the first image data and the first text data into a continuous space, and then in the space, adds (and deletes) Gaussian noise ∈ t In addition, the electronic device adapts the loss from GENIE by combining the hidden representation Ds of the decoder component. The model calculates the loss at time step t using the model training loss function as shown below.

[0123]

[0124] The electronics minimize the noise With the actual noise ∈ t The denoiser component is trained by minimizing the error between the hidden representation D s The decoder D and the function L are trained by the error between the and embedded codes, and the standard cross entropy loss function is applied to the output of the classification head H, which gives the hidden representation D s and the actual code snippet y, generating predicted code tokens.

[0125] In this embodiment, the electronic device first predicts the noise The Gaussian noise ∈ tThe error between the hidden representation Ds and the embedded code, and the standard cross entropy loss function are used to calculate the model training loss function. In this way, the electronic device can use the diffusion target to train the three main components of the artificial intelligence model (denoiser, decoder and classification head) based on the model training loss function, so as to finally obtain a code generation model that can realize automated programming based on images and / or text.

[0126] In some embodiments, the code generation model training method provided in the embodiments of the present application may further include at least one of the following:

[0127] Pre-training the denoiser component and the decoder component for unsupervised code generation based on preset Gaussian noise;

[0128] The encoder component calculates a preset code segment to obtain denoiser input data, and the denoiser component and the decoder component are pre-trained to denoise continuous code paragraphs based on the denoiser input data.

[0129] In the process of model training for the artificial intelligence model to obtain the code generation model, the electronic device uses the two tasks of unsupervised code generation and improvement of continuous paragraph denoising (CPD) of the code to pre-train the denoiser (N) and decoder (D) on the code snippet corpus (multimodal model training data). Among them, the code-specific CPD task only masks the tokens associated with the identifiers or built-in keywords of the target language. The electronic device randomly samples from these two tasks during the pre-training process. Specifically, both the pre-training and fine-tuning tasks are sampled using Lt.

[0130] It should be noted that there is no natural language expression in the pre-training of the denoiser (N) and the decoder (D), that is, the denoiser (N) does not have an input synthesis vector E s , so in the unsupervised code generation task, E s is replaced by Gaussian noise sampled at each denoising time step t. In the CPD task, E s It is obtained by calculating the masked code y through the encoder (R).

[0131] In some embodiments, after the electronic device completes basic multimodal large model training for the artificial intelligence model to obtain the code generation model, it can also fine-tune the code generation model through multimodal instruction fine-tuning, based on modal alignment, and further fine-tune the instruction training to enable the code generation model to better complete specific code generation tasks.

[0132] It should be noted that the electronic device tunes the code generation model through multimodal instruction tuning (M-IT), where instruction refers to the description of a specific code generation task. Multimodal instruction tuning (M-IT) is a technology that fine-tunes the pre-trained MLLM through instruction-formatted data. Through this technology, electronic devices can make the code generation model generalize to unseen tasks following new instructions, thereby improving zero-shot learning performance.

[0133] In some embodiments, the basic form of multimodal instruction data can be summarized as a triplet of (instruction, multimodal input, answer). The specific design of instructions can be divided into two methods: manual design and GPT-assisted design. Among them, manual design refers to manually designing a series of instruction templates for each task, such as for traditional visual question-answering tasks, while GPT-assisted design is to manually design a small number of samples to prompt GPT to generate richer instructions.

[0134] In some embodiments, the electronic device may also use multimodal in-context learning (M-ICL) to tune the code generation model. Multimodal in-context learning (M-ICL) refers to the electronic device giving a small number of samples as prompt input to stimulate the potential capabilities of the code generation model and normalize the output of the model.

[0135] In other embodiments, the electronic device may also tune the code generation model by simultaneously adopting multimodal instruction fine-tuning M-IT and multimodal context learning M-ICL, so that the code generation model can better complete specific code generation tasks.

[0136] In this embodiment, an electronic device trains an artificial intelligence model based on the first text data and the first image data to obtain a code generation model, and further fine-tunes the code generation model. Combined with experiments, it is found that the effect obtained by the code generation model with 100M parameters is equivalent to the effect of many models with billions or tens of billions of parameters such as GPT-4Vision.

[0137] Please refer to Figure 6 , Figure 6 A schematic diagram of the overall process involved in some embodiments of the code model training method provided in the embodiments of the present application.

[0138] like Figure 6As shown, in some embodiments, the electronic device performs model training through multimodal large-scale data to obtain an automatic code generation model based on a multimodal diffusion model, also known as a multimodal code diffusion model. In this embodiment, the multimodal code diffusion model is the above-mentioned code generation model. The code generation model can comprehensively utilize information from different modalities to perform various cross-modal tasks.

[0139] When the electronic device generates a model for the training code, it first designs a large model architecture according to the scenario. The model architecture can be as described above. Figure 2 As shown, the code generation model includes three core components: an encoder (E), a denoiser (N), and a decoder (D), plus a classification head (H). Among them, the encoder (E) is a multimodal large language model based on contrastive learning. Unlike some contrastive learning methods in computer vision (CV) such as moco and simclr, the multimodal model training data constructed by electronic devices is text-image pairs: an image (image data) and its corresponding text description (text data). Therefore, the electronic device creates an encoder (E) through a multimodal large language model based on contrastive learning, and the encoder (E) can learn the matching relationship between text-image pairs through contrastive learning. For example, as Figure 2 As shown, the encoder (E) mainly includes two models: Text Encoder and Image Encoder. The Text Encoder is used to extract the features of text (text data) and can adopt the text transformer model commonly used in NLP; while the Image Encoder is used to extract the features of images (image data) and can adopt the commonly used CNN model or vision transformer.

[0140] In addition, the denoiser (N) can be a Transformer block that can denoise the Gaussian noise x based on the self-attention mechanism. t and the resultant vector E s Perform cross attention processing and deal with the noise x t Compared with the traditional text diffusion method that generates tags independently, the electronic device uses full self-attention to make each hidden dimension (d i ) have complete information about the other dimensions.

[0141] After designing the model architecture, the electronic device constructs model training data by collecting data and performing data cleaning operations. Among them, the method for the electronic device to obtain model training data includes: 1. Obtaining a repository with more than 1k collection marks (star) and providing a website link from github, first get the source code of the website for subsequent model training. 2. Through the ranking on Alexa, a world-renowned website ranking, grab the screenshots and codes of the top 1 million websites. In terms of data cleaning, the electronic device uses the self-training method of momentum distillation to learn from network image and text data to alleviate the noise problem in the original data. 3. Find a common component library in the industry and obtain each component and its corresponding source code. These components may be the smallest particles of the page, and the code quality is high, which is very effective in improving the model.

[0142] After the electronic device further obtains the model training data, it pre-trains the model by setting tasks. The electronic device compares and learns the text features extracted from the text data and the image features extracted from the image data through the encoder (E). For a training batch (multimodal model training data) containing N text-image pairs (a text data and an image data described by the text data are a text-image pair), the N text features and N image features are combined in pairs, and the model predicts N 2 The similarity of possible text-image pairs is calculated by directly calculating the cosine similarity of text features and image features. Figure 2 In this way, if there are N positive samples, that is, text and images that truly belong to a pair (diagonal elements in the matrix), and the remaining N 2 —N text-image pairs are negative samples, so the training goal is to maximize the similarity of N positive samples while minimizing N 2 —The similarity of N negative samples. The encoder (E) will train a multimodal pre-trained neural network (Contrastive Language-Image Pre-Training, CLIP) model, and then multiply the encoding results of the text data and image data by a weight of 0.5 to generate a synthetic vector E n , E n ={e1,e2,e3,...,e n}. Afterwards, the denoiser (N) gradually superimposes Gaussian noise xt on the synthetic vector En according to a preset time step t. Since the denoiser (N) can be a Transformer block with cross-attention of Gaussian noise xt and synthetic vector Es and full self-attention of noise xt, the electronic device can use the decoder (D) in the denoiser (N) to predict the noise based on the self-attention mechanism before the denoiser (N) maps the denoised embeddings to the discrete code Toekn. and the resultant vector E s Perform cross attention and add Gaussian noise x 0 Full self-attention is performed on the hidden representation Ds = {d1, d2, ..., dn} = D(x 0 ,Es). Finally, the electronic device uses the classification head H to transform the hidden representation D s Project it onto the actual code mark and calculate the distribution of the code mark p(y|d i ). The electronic device does not search for these tags and selects

[0143] It is important to note that the electronic device trains the AI ​​model in two stages: an unsupervised pre-training stage for the code snippet and a supervised fine-tuning stage for the encoder component, the denoiser component and the decoder component. The electronic device uses the encoder (E) to embed the image data and text data into a continuous space, and then adds (and removes) Gaussian noise ∈ t in this space at time step t. t In addition, the electronic device adapts the loss from GENIE by combining the hidden representation Ds of the decoder component. The model calculates the loss at time step t using the model training loss function as shown below.

[0144]

[0145] The electronics minimize the noise With the actual noise ∈ t The denoiser component is trained by minimizing the error between the hidden representation D s The decoders D and L are trained by the error between the and embedded codes, and the standard cross entropy loss function is applied to the output of the classification head H, which gives the hidden representation D s and the actual code snippet y, generating predicted code tokens.

[0146] In addition, the electronic device uses two tasks, unsupervised code generation and improvement of continuous paragraph denoising (CPD) of code, to pre-train the denoiser (N) and decoder (D) on the code snippet corpus (multimodal model training data) in the process of training the code generation model. Among them, the code-specific CPD task only masks tokens associated with identifiers or built-in keywords of the target language. The electronic device randomly samples from these two tasks during the pre-training process. Specifically, both the pre-training and fine-tuning tasks are sampled using Lt.

[0147] Since there is no natural language expression in the pre-training of the denoiser (N) and the decoder (D), i.e., the denoiser (N) does not have an input synthetic vector E s , so in the unsupervised code generation task, E s is replaced by Gaussian noise sampled at each denoising time step t. In the CPD task, E s It is obtained by calculating the masked code y through the encoder (R).

[0148] After pre-training the model, the electronic device further fine-tunes the model by instructions to further improve the model effect. That is, the electronic device tunes the code generation model by multimodal instruction tuning (M-IT) and multimodal in-context learning (M-ICL). Among them, instruction refers to the description of a specific code generation task. Multimodal instruction fine-tuning M-IT is a technology that fine-tunes the pre-trained MLLM through instruction-formatted data. Through this technology, electronic devices can make the code generation model follow new instructions to generalize to unseen tasks, thereby improving zero-shot learning performance. The basic form of multimodal instruction data can be summarized as a triplet of (instruction, multimodal input, answer). The specific design of instructions can be divided into two methods: manual design and GPT-assisted design. Among them, manual design refers to manually designing a series of instruction templates for each task, such as for traditional visual question-answering tasks, while GPT-assisted design is to manually design a small number of samples to prompt GPT to generate richer instructions. Multimodal contextual learning M-ICL refers to electronic devices giving a small number of samples as prompt input to stimulate the potential capabilities of the code generation model and normalize the output of the model.

[0149] Finally, the electronic device evaluates the experimental effect of the model through the test data set. In this process, the electronic device combines experiments and finds that the effect of the code generation model with 100M parameters is equivalent to the effect of many models with billions or tens of billions of parameters such as GPT-4Vision.

[0150] Please refer to Figure 7 , Figure 7 A flowchart of the steps of the code generation method provided in some embodiments of the present application.

[0151] like Figure 7 As shown, in some embodiments, the electronic device applies the code generation method provided by the embodiments of the present application, which may include step S701 and step S702.

[0152] Step S701: Acquire model input data; wherein the model input data at least includes second image data.

[0153] After the electronic device has trained and obtained the code generation model, it can use the code generation model to perform an automated programming operation to generate code. That is, the electronic device first obtains the second image data for which code needs to be generated, and then uses the second image data as model input data.

[0154] Step S702: input the second image data into a code generation model to learn the second image data based on the code generation model and output a code corresponding to the second image data.

[0155] It should be noted that the code generation model is a code generation model obtained by training the artificial intelligence model based on the first text data and the first image data using the training method of the code generation model provided in the above-mentioned embodiment of the present application.

[0156] After obtaining the model input data, the electronic device calls a pre-trained code generation model, thereby inputting the second image data in the model input data into the code generation model, so as to learn based on the code generation model using the encoder second image data, and denoise the synthetic vector representing the matching relationship between the text data (the text data generated by the model to describe the features of the second image data) and the second image data based on the denoiser, and then generates a hidden representation based on the decoder and maps the hidden representation to a specific code based on the classification head and outputs the code, which is the code corresponding to the second image data.

[0157] In some embodiments, after training the code generation model, the electronic device can also simultaneously obtain the second text data for which the code needs to be generated and the second image data having the features described by the second text data, and then use the second text data and the second image data to form a text-image pair as model input data. Afterwards, the electronic device calls the pre-trained code generation model, thereby inputting the second text data and the second image data in the model input data into the code generation model, so as to learn the second text data and the second image data based on the code generation model using an encoder, and denoise the synthetic vector representing the matching relationship between the second text data and the second image data based on a denoiser, and then generate a hidden representation based on a decoder and map the hidden representation to a specific code based on a classification head and output the code corresponding to the second image data.

[0158] In the embodiment of the present application, by expanding the diffusion model to multimodality and applying it to the field of code generation, the electronic device combines the multimodal large language model and the diffusion model training to obtain a code generation model based on multimodality and diffusion, and then uses the code generation model to learn the model input data including at least image data (such as complex website interfaces, etc.), thereby automatically generating code corresponding to the image data. In this way, compared with the method of manually writing code data corresponding to complex website interfaces, the embodiment of the present application effectively improves the ability of the code generation model to realize automated programming based on images or text, and broadens the scope of automated programming based on artificial intelligence.

[0159] The embodiment of the present application also provides a training device for a code generation model, which can implement the training method for the code generation model. The device includes a first acquisition module and a model training module.

[0160] A first acquisition module, used to acquire multimodal model training data; wherein the multimodal model training data includes first text data and first image data, and the first text data is data for describing features of the first image data;

[0161] A model training module, used to perform model training on a preset artificial intelligence model based on the first text data and the first image data to obtain a code generation model;

[0162] Among them, the artificial intelligence model is constructed by combining a multimodal large language model and a diffusion model; the code generation model is used to learn at least the second image data and output a code corresponding to the second image data.

[0163] In some embodiments, the first acquisition module is further used to acquire a preset artificial intelligence model;

[0164] Among them, the artificial intelligence model includes an encoder component, a denoiser component, a decoder component and a classification head; the encoder component is created based on a multimodal large language model, and the denoiser component is created based on a diffusion model.

[0165] In some embodiments, the model training module is also used to perform comparative learning on the first text data and the first image data based on the encoder component to obtain a synthetic vector that represents the matching relationship between the first text data and the first image data; perform diffusion learning on the synthetic vector based on the denoiser component to obtain a diffusion target; and generate a hidden representation corresponding to the diffusion target based on the decoder component, and calculate the code tag distribution corresponding to the hidden representation based on the classification head.

[0166] In some embodiments, the model training module is also used to gradually superimpose Gaussian noise on the synthetic vector based on the denoiser component; and, perform self-attention processing on the synthetic vector, the predicted noise corresponding to the synthetic vector, and the Gaussian noise to obtain a diffusion target corresponding to the synthetic vector.

[0167] In some embodiments, the model training module is also used to obtain a preset model training loss function; wherein the model training loss function is calculated based on the error between the predicted noise and the Gaussian noise, the error between the hidden representation and the embedded code, and a standard cross entropy loss function; wherein the embedded code is the source code in the first text data; and, based on the model training loss function, the denoiser component, the decoder component and the classification head are trained.

[0168] In some embodiments, the model training module is also used to pre-train the denoiser component and the decoder component for unsupervised code generation based on preset Gaussian noise; and to calculate a preset code fragment based on the encoder component to obtain denoiser input data, and pre-train the denoiser component and the decoder component for denoising continuous code paragraphs based on the denoiser input data.

[0169] In some embodiments, the first acquisition module is further used to obtain website source code from a preset code hosting platform as first text data, and obtain a website interface corresponding to the website source code as first image data to construct multimodal model training data; determine a target website that meets preset website ranking conditions, capture the code of the target website as the first text data, and capture a website screenshot corresponding to the code of the target website as the first image data to construct multimodal model training data; obtain component source code from a preset component library as the first text data, and obtain a component image corresponding to the component source code as the first image data to construct multimodal model training data; and perform multimodal representation learning on the model training data based on momentum distillation to remove original noise data in the model training data.

[0170] The specific implementation of the code generation model training device provided in the embodiment of the present application is basically the same as the specific implementation of the code generation model training method described above, and will not be repeated here.

[0171] The embodiment of the present application also provides a code generation device, which can implement the above code generation method, and the device includes a second acquisition module and a code generation module.

[0172] A second acquisition module, used to acquire model input data; wherein the model input data at least includes second image data;

[0173] A code generation module is used to input the second image data into the code generation model as described in the first aspect above, so as to learn the second image data based on the code generation model and output a code corresponding to the second image data.

[0174] The specific implementation of the code generation device provided in the embodiment of the present application is basically the same as the specific implementation of the above-mentioned code generation method, and will not be repeated here.

[0175] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the training method of the code generation model when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0176] See also Figure 8 , Figure 8 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0177] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0178] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the training method of the code generation model of the embodiment of this application;

[0179] Input / output interface 803, used to implement information input and output;

[0180] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0181] A bus 805 that transmits information between the various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);

[0182] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0183] An embodiment of the present application also provides a vehicle, which is equipped with an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the training method of the above-mentioned code generation model when executing the computer program.

[0184] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the training method of the above-mentioned code generation model is implemented.

[0185] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0186] An embodiment of the present application also provides a computer program product, including a computer program. The steps implemented when the computer program is executed by a processor are basically the same as the specific embodiment of the training method of the above-mentioned code generation model, and will not be repeated here.

[0187] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0188] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0189] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0191] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0192] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0193] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0194] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0196] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0197] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for training a code generation model, characterized in that: The method comprises: Acquire multimodal model training data; wherein the multimodal model training data includes first text data and first image data, and the first text data is data for describing features of the first image data; Performing model training on a preset artificial intelligence model based on the first text data and the first image data to obtain a code generation model; Among them, the artificial intelligence model is constructed by combining a multimodal large language model and a diffusion model; the code generation model is used to learn at least the second image data and output a code corresponding to the second image data.

2. The method according to claim 1, characterized in that Before performing model training on a preset artificial intelligence model based on the first text data and the first image data to obtain a code generation model, the method further includes: Get the preset artificial intelligence model; Among them, the artificial intelligence model includes an encoder component, a denoiser component, a decoder component and a classification head; the encoder component is created based on a multimodal large language model, and the denoiser component is created based on a diffusion model.

3. The method according to claim 2, characterized in that The performing model training on a preset artificial intelligence model based on the first text data and the first image data includes: Performing comparative learning on the first text data and the first image data based on the encoder component to obtain a synthetic vector representing a matching relationship between the first text data and the first image data; Performing diffusion learning on the synthetic vector based on the denoiser component to obtain a diffusion target; A hidden representation corresponding to the diffusion target is generated based on the decoder component, and a code label distribution corresponding to the hidden representation is calculated based on the classification head.

4. The method according to claim 3, characterized in that The step of performing diffusion learning on the synthetic vector based on the denoiser component to obtain a diffusion target includes: gradually superimposing Gaussian noise on the synthetic vector based on the denoiser component; Self-attention processing is performed on the synthetic vector, the predicted noise corresponding to the synthetic vector, and the Gaussian noise to obtain a diffusion target corresponding to the synthetic vector.

5. The method according to claim 4, characterized in that The method further comprises: Obtaining a preset model training loss function; wherein the model training loss function is calculated based on the error between the predicted noise and the Gaussian noise, the error between the hidden representation and the embedded code, and a standard cross entropy loss function; wherein the embedded code is the source code in the first text data; The denoiser component, the decoder component and the classification head are trained based on the model training loss function.

6. The method according to claim 2, characterized in that The method further comprises at least one of the following: Pre-training the denoiser component and the decoder component for unsupervised code generation based on preset Gaussian noise; The encoder component calculates a preset code segment to obtain denoiser input data, and the denoiser component and the decoder component are pre-trained to denoise continuous code paragraphs based on the denoiser input data.

7. The method according to any one of claims 1 to 6, characterized in that The obtaining of multimodal model training data includes at least one of the following: Acquire website source code from a preset code hosting platform as first text data, and acquire a website interface corresponding to the website source code as first image data, so as to construct multimodal model training data; Determine a target website that meets a preset website ranking condition, capture the code of the target website as first text data, and capture a website screenshot corresponding to the code of the target website as first image data, so as to construct multimodal model training data; Acquire component source code from a preset component library as first text data, and acquire a component image corresponding to the component source code as first image data, so as to construct multimodal model training data; Multimodal representation learning is performed on the model training data based on momentum distillation to remove original noise data in the model training data.

8. A code generation method, characterized in that: The method comprises: Acquire model input data; wherein the model input data at least includes second image data; The second image data is input to the code generation model according to any one of claims 1 to 7, so that the second image data is learned based on the code generation model and a code corresponding to the second image data is output.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the training method of the code generation model described in any one of claims 1 to 7, and / or implements the code generation method described in claim 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the code generation model training method described in any one of claims 1 to 7, and / or implements the code generation method described in claim 8.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the code generation model training method described in any one of claims 1 to 7, and / or implements the code generation method described in claim 8.

Images