Training method and device of multi-modal hybrid generative model, storage medium and equipment

Abstract

The application discloses a training method and device of a multimodal hybrid generation model, a storage medium and equipment, and belongs to the technical field of deep learning. A first text is encoded by using a text tokenizer, a first image is encoded by using an image tokenizer, and a MixIt model is trained according to an encoding result and a true value; a noisy image vector is generated according to a second training sample, the noisy image vector being generated by adding random noise to an image vector; a prompt word in the second training sample is encoded by using a Flux model to obtain a text vector; an image vector and the text vector are processed by using a ControlNet model, and a control vector is output to the Flux model; the noisy image vector, the text vector and the control vector are processed by using the Flux model, and a Flux-ControlNet model is trained according to predicted noise and random noise. The application can generate a graphic-text hybrid content containing high-quality images.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a training method, apparatus, storage medium and device for a multimodal hybrid generative model. Background Technology

[0002] With technological advancements and hardware improvements, multimodal generation technology is booming:

[0003] First, the Vision-Language Model (VLM), represented by the Large Language and Vision Assistant (LLaVA), maps images to a text encoding space through an image encoder and a text-image adapter. Using corresponding text instructions, the large model can analyze, understand, and summarize image content and output corresponding text content.

[0004] Secondly, Chameleon and Janus-Pro, based on autoregressive techniques, infer alternating multimodal tokens by fusing multiple modal tokens early on. This enables them to understand and generate arbitrary sequences of images and text, pushing multimodal models towards unified generation and understanding modeling. However, limitations such as low resolution and reconstruction loss introduced by visual word segmentation result in a lack of fine detail despite the rich semantic content of the generated images. For example, small facial regions occupying limited image space lack detail.

[0005] Then, multimodal hybrid generation models, such as the StableDiffusion Diffusion Model and the RectifiedFlow Model Flux, can output stunning results in seconds with appropriate prompts, without the need for complex post-processing. Although models like StableDiffusion and Flux can only output a single modality of the image, the generated image quality is high.

[0006] However, existing multimodal hybrid generative models can only generate single-modal data, such as text or images, and cannot generate multimodal data. Summary of the Invention

[0007] This application provides a training method, apparatus, storage medium, and device for a multimodal hybrid generative model, to solve the problem that multimodal hybrid generative models can only generate single-modal data and cannot generate multimodal data. The technical solution is as follows:

[0008] According to a first aspect of this application, a method for training a multimodal hybrid generative model is provided, wherein the multimodal hybrid generative model includes a MixIT model and a Flux-ControlNet model, and the Flux-ControlNet model includes a Flux model and a ControlNet model, the method comprising:

[0009] Multiple first training samples are obtained, the first text in the first training samples is encoded using a text segmenter, the first image in the first training samples is encoded using an image segmenter, and the Janus model is trained based on the encoding results and the ground truth in the first training samples to obtain the MixIt model.

[0010] Multiple second training samples are obtained, and a noisy image vector is generated based on the second training samples. The noisy image vector is generated by adding random noise to the image vector.

[0011] The text processing module in the Flux model is used to encode the prompt words in the second training sample to obtain a text vector;

[0012] The image vector and the text vector are processed using the ControlNet model, and the resulting control vector is output to the generation module in the Flux model;

[0013] The generation module processes the noisy image vector, the text vector, and the control vector to obtain predicted noise;

[0014] A first loss value is calculated based on the predicted noise and the random noise, and the Flux-ControlNet model is trained based on the first loss value.

[0015] In one possible implementation, generating the noisy image vector based on the second training sample includes:

[0016] When the second training sample includes a second image, a prompt word, and random noise, the second image is encoded using the Vector Quantization VQ-Variational Autoencoder (VAE) in the Flux model to obtain an image vector; the random noise is then added to the image vector to obtain a noisy image vector.

[0017] When the second training sample includes the image vector predicted by the MixIT model, the prompt word, and random noise, the random noise is added to the image vector to obtain a noisy image vector.

[0018] In one possible implementation, the step of encoding the prompt words in the second training sample using the text processing module in the Flux model to obtain a text vector includes:

[0019] When the text processing module includes a contrastive language-image pre-trained CLIP and a text-to-text transfer converter T5, the text segmenter is used to encode and embed the prompt words to obtain a text encoding vector.

[0020] The text encoding vector is obtained by encoding the text using CLIP and T5 respectively.

[0021] In one possible implementation, the step of processing the image vector and the text vector using the ControlNet model and outputting the resulting control vector to the generation module in the Flux model includes:

[0022] When the ControlNet model includes a convolutional network and a DiT spread transformer, the image vector is convolved using the convolutional network to obtain a convolutional vector.

[0023] The convolution vector and the text vector are processed using the DiT method, and the resulting control vector is output to the generation module in the Flux model.

[0024] In one possible implementation, the step of processing the noisy image vector, the text vector, and the control vector using the generation module to obtain predicted noise includes:

[0025] When the generation module includes DiT and Multilayer Perceptron (MLP), the DiT is used to process the noisy image vector, the text vector, and the control vector to obtain an intermediate vector;

[0026] The intermediate vector is processed using the MLP to obtain the prediction noise.

[0027] In one possible implementation, the step of encoding the first text in the first training sample using a text segmenter, encoding the first image in the first training sample using an image segmenter, and training the Janus model based on the encoding results and ground truth values ​​in the first training sample to obtain the MixIt model includes:

[0028] The first text in the first training sample is encoded using a text segmenter to obtain a text index sequence;

[0029] The image segmenter is used to encode the first image in the first training sample to obtain an image index sequence;

[0030] The text index sequence and the image index sequence are mixed to obtain a mixed vector;

[0031] The sliding window in the Janus model is used to extract vector data from the mixed vector, and the predicted value of the next vector data is generated based on the vector data.

[0032] The second loss value is calculated based on the predicted value and the true value of the next vector data in the first training sample. The Janus model is then trained based on the second loss value to obtain the MixIt model.

[0033] In one possible implementation, the method further includes:

[0034] The second text and the third image are obtained. The second text is encoded using the text segmenter, and the third image is encoded using the image segmenter. The encoding results are processed using the MixIt model to obtain the text index sequence and the image index sequence.

[0035] The text index sequence is decoded using the text segmenter to obtain a third text, which includes the generated text and image location markers.

[0036] Generate prompt words based on the second text and the generated text;

[0037] Generate an image vector based on the image index sequence;

[0038] The Flux-ControlNet model is used to process the preset image resolution, the prompt words, random noise, and the image vector to obtain the fourth image;

[0039] The fourth image is inserted into the image position mark in the third text to obtain the final image and text generation result.

[0040] According to a second aspect of this application, a training apparatus for a multimodal hybrid generative model is provided, the multimodal hybrid generative model including a MixIT model and a Flux-ControlNet model, the Flux-ControlNet model including a Flux model and a ControlNet model, the apparatus comprising:

[0041] The first acquisition module is used to acquire multiple first training samples, encode the first text in the first training samples using a text segmenter, encode the first image in the first training samples using an image segmenter, and train the Janus model based on the encoding results and the ground truth in the first training samples to obtain the MixIt model.

[0042] The second acquisition module is used to acquire multiple second training samples and generate a noisy image vector based on the second training samples. The noisy image vector is generated by adding random noise to the image vector.

[0043] The encoding module is used to encode the prompt words in the second training sample using the text processing module in the Flux model to obtain a text vector;

[0044] The control module is used to process the image vector and the text vector using the ControlNet model, and output the resulting control vector to the generation module in the Flux model;

[0045] The prediction module is used to process the noisy image vector, the text vector, and the control vector using the generation module to obtain predicted noise;

[0046] The training module is used to calculate a first loss value based on the predicted noise and the random noise, and to train the Flux-ControlNet model based on the first loss value.

[0047] According to a third aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the training method of the multimodal hybrid generative model as described above.

[0048] According to a fourth aspect of this application, a computer device is provided, the computer device including the training apparatus for the above-described multimodal hybrid generative model.

[0049] The beneficial effects of the technical solution provided in this application include at least the following:

[0050] The multimodal hybrid generative model includes the MixIT model and the Flux-ControlNet model. The MixIT model is used to generate multimodal data containing text and images, while the Flux-ControlNet model is used to generate high-quality images. Through a two-stage approach, text and image generation and image quality are decoupled, maintaining the contextual consistency of text and image generation while ensuring the quality of generated images.

[0051] Based on the Flux model of corrective flow generation, combined with ControlNet model fine-tuning technology, it can not only enhance the image quality of image-text generation based on small datasets with short training cycles and low resource consumption, but also control the high-quality image generation according to the set image resolution, and adjust the image details individually in combination with prompt words, thus achieving fine adjustment of image details.

[0052] The Flux-ControlNet model uses some of the image vectors predicted by the MixIT model during training, thus bridging the gap between the training and inference scenarios of the Flux-ControlNet model. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart of a training method for a multimodal hybrid generative model provided in one embodiment of this application;

[0055] Figure 2 This is a schematic diagram of the structure of an image word segmenter provided in one embodiment of this application;

[0056] Figure 3 This is a schematic diagram illustrating the training of the MixIT model provided in one embodiment of this application;

[0057] Figure 4 This is a flowchart of a training method for a multimodal hybrid generative model provided in one embodiment of this application;

[0058] Figure 5 This is a schematic diagram of training a Flux-ControlNet model provided in one embodiment of this application;

[0059] Figure 6 This is a flowchart of a method for generating images and text in a multimodal hybrid generative model according to an embodiment of this application;

[0060] Figure 7 This is a schematic diagram of a graphic generation process provided in one embodiment of this application;

[0061] Figure 8 This is a structural block diagram of a training device for a multimodal hybrid generative model provided in one embodiment of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0063] The multimodal hybrid generation model in this embodiment includes the MixIT model and the Flux-ControlNet model. The MixIT model is trained on the Janus model, which is a multimodal hybrid generation model. Therefore, the MixIT model can generate text and image content, but the image quality in the text and image content is not high. The Flux-ControlNet model includes the Flux model and the ControlNet model. The Flux model is a corrected flow model, and the ControlNet model is a fine-tuning model. The ControlNet model can assist the Flux model in generating high-quality images. In this application, the text and image are first generated by the MixIT model, and a high-quality image is generated by the Flux-ControlNet model. Then, the high-quality image is inserted into the text and image to obtain high-quality text and image content.

[0064] The training process of the multimodal hybrid generative model is divided into two stages. The first stage is to generate the MixIT model based on the Janus model using multimodal image and text data. The second stage is to train the Flux-ControlNet model using image and text data. The training process is described below.

[0065] like Figure 1 The diagram illustrates a flowchart of a training method for a multimodal hybrid generative model according to an embodiment of this application. This training method can be applied to a computer device. The training method for the multimodal hybrid generative model may include:

[0066] Step 101: Obtain multiple first training samples, encode the first text in the first training samples using a text segmenter, encode the first image in the first training samples using an image segmenter, and train the Janus model based on the encoding results and the ground truth in the first training samples to obtain the MixIt model.

[0067] In this embodiment, the first training sample set can be generated solely from text, in which case the first training sample includes the first text and text ground truth values; alternatively, the first training sample set can be generated solely from images, in which case different text instructions can be set according to different tasks, and the corresponding first training sample includes the first text, the first image (single or multiple images), and ground truth values; alternatively, the first training sample set can be generated from both text and images, in which case the first training sample includes the first text, the first image (single or multiple images), and ground truth values. The mixing ratio of the above-mentioned first training samples can be set according to business requirements.

[0068] In this embodiment, the text segmenter and image segmenter can be trained first, and then the trained text segmenter and image segmenter can be used to train the MixIt model. Specifically, the text segmenter can be trained on the text corpus using the Byte Pair Encoding (BPE) method, and the image segmenter can be trained on the image dataset using the Vector Quantized (VQ)-Variational Autoencoder (VAE) algorithm. The weights of the trained text segmenter and image segmenter will be frozen and no longer updated in subsequent stages.

[0069] Figure 2 The model structure of an image segmenter is shown. The image segmenter includes an encoder, a discrete vector set, and a decoder. First, the encoder encodes the image to obtain continuous feature vectors. Then, the continuous feature vectors are discretized by using the nearest neighbor method to find the nearest feature vector in the discrete vector set. The index of the found feature vector is used to generate an image sequence index. The discrete vector set is then searched to restore the image sequence index into continuous feature vectors. Finally, the decoder is used to restore the continuous feature vectors into an image.

[0070] like Figure 3 As shown, Figure 3 The autoregressive large language model in the model is the MixIt model. When training the MixIt model, a text segmenter is used to encode the first text in the first training sample, obtaining a text encoding vector; an image segmenter is used to encode the first image in the first training sample, obtaining an image encoding vector; after aligning the image encoding vector and the text encoding vector using an image-text adapter, the Janus model is trained to obtain the MixIt model.

[0071] Step 102: Obtain multiple second training samples and generate a noisy image vector based on the second training samples. The noisy image vector is generated by adding random noise to the image vector.

[0072] The second training samples include two types. The first type includes a second image, a prompt word, and random noise. The second type includes an image vector predicted by the MixIT model, a prompt word, and random noise. Regardless of the type of second training sample, a noisy image vector can be generated.

[0073] Step 103: Use the text processing module in the Flux model to encode the prompt words in the second training sample to obtain text vectors.

[0074] Step 104: Use the ControlNet model to process the image vectors and text vectors, and output the resulting control vectors to the generation module in the Flux model.

[0075] Step 105: The generation module is used to process the noisy image vector, text vector, and control vector to obtain the predicted noise.

[0076] The output of the Flux model is prediction noise.

[0077] Step 106: Calculate the first loss value based on the predicted noise and random noise, and train the Flux-ControlNet model based on the first loss value.

[0078] In summary, the training method for the multimodal hybrid generative model provided in this application includes a MixIT model and a Flux-ControlNet model. The MixIT model is used to generate multimodal data containing text and images, while the Flux-ControlNet model is used to generate high-quality images. Through a two-stage approach, text and image generation and image quality are decoupled, maintaining the contextual consistency of text and image generation while ensuring the quality of image generation.

[0079] Based on the Flux model of corrective flow generation, combined with ControlNet model fine-tuning technology, it can not only enhance the image quality of image-text generation based on small datasets with short training cycles and low resource consumption, but also adjust image details individually in combination with prompt words, achieving fine adjustment of image details.

[0080] like Figure 4 The diagram illustrates a flowchart of a training method for a multimodal generative model according to an embodiment of this application. This training method can be applied to a computer device. The training method for the multimodal generative model may include:

[0081] Step 401: Obtain multiple first training samples, encode the first text in the first training samples using a text segmenter, encode the first image in the first training samples using an image segmenter, and train the Janus model based on the encoding results and the ground truth in the first training samples to obtain the MixIt model.

[0082] The explanations of the first training sample, the text segmenter, and the image segmenter are detailed in the descriptions above and will not be repeated here.

[0083] Specifically, a text segmenter is used to encode the first text in the first training sample, and an image segmenter is used to encode the first image in the first training sample. Based on the encoding results and the ground truth values ​​in the first training sample, the Janus model is trained to obtain the MixIt model, which may include:

[0084] (1) Use a text segmenter to encode the first text in the first training sample to obtain the text index sequence.

[0085] The text segmenter encodes the first text to obtain a continuous text vector; the continuous text vector is then discretized to obtain a text index sequence.

[0086] (2) Use an image segmenter to encode the first image in the first training sample to obtain the image index sequence.

[0087] The image segmenter encodes the first image to obtain a continuous image vector; the continuous image vector is then discretized to obtain an image index sequence.

[0088] (3) Mix the text index sequence and the image index sequence to obtain a mixed vector.

[0089] The text index sequence and the image index sequence may be interleaved, requiring different start and end markers to be set for the text index sequence and the image index sequence to identify the start and end positions of each text segment and each image segment.

[0090] (4) Use the sliding window in the Janus model to extract vector data from the mixed vector and generate the predicted value of the next vector data based on the vector data.

[0091] The Janus model takes the true value of a vector data point captured by a sliding window as input and outputs the predicted value of the next vector data point.

[0092] (5) Calculate the second loss value based on the predicted value and the true value of the next vector data in the first training sample, and train the Janus model based on the second loss value to obtain the MixIt model.

[0093] Step 402: Obtain multiple second training samples.

[0094] The second training samples include two types. The first type includes a second image, prompt words, and random noise. The second type includes image vectors predicted by the MixIT model, prompt words, and random noise. The mixing ratio of these two types of second training samples can be set according to business needs, such as 7:3. This embodiment does not impose any limitations.

[0095] The Flux-ControlNet model uses some of the image vectors predicted by the MixIT model during training, thus bridging the gap between the training and inference scenarios of the Flux-ControlNet model.

[0096] Step 403: When the second training sample includes the second image, prompt words and random noise, the second image is encoded using VQ-VAE in the Flux model to obtain an image vector; random noise is added to the image vector to obtain a noisy image vector, and step 405 is executed.

[0097] The VQ-VAE mentioned here has the same structure as the image word segmenter mentioned above, but the parameters are different.

[0098] like Figure 5 As shown, after the image is encoded into an image vector by VQ-VAE, noise is added to the image to obtain a noisy image vector.

[0099] Step 404: When the second training sample includes the image vector predicted by the MixIT model, the prompt word, and random noise, add random noise to the image vector to obtain a noisy image vector, and then proceed to step 405.

[0100] The image vector predicted by the MixIT model is... Figure 3 The image encoding vector output by the MixIT model.

[0101] Step 405: Use the text processing module in the Flux model to encode the prompt words in the second training sample to obtain text vectors.

[0102] like Figure 5 As shown, the Flux model includes a VQ-VAE, a text processing module, and a generation module. The text processing module includes a Contrastive Language-Image Pre-Training (CLIP) and a Text-to-Text Transfer Transformer (T5). The generation module includes a Diffusion Transformer (DiT) and a Multi-Layer Perceptron (MLP). The inputs of VQ-VAE, CLIP, and T5 are the inputs of the Flux model. The outputs of VQ-VAE, CLIP, and T5 are connected to the inputs of DiT, and the output of DiT is connected to the input of the MLP. The output of the MLP is the output of the Flux model.

[0103] Specifically, the text processing module in the Flux model is used to encode the prompt words in the second training sample to obtain text vectors. This can include: using a text segmenter to encode and embed the prompt words to obtain text encoded vectors; and using CLIP and T5 to encode the text encoded vectors to obtain text vectors.

[0104] Figure 5 The text in the code is the prompt word. After the text segmenter generates a text encoding vector for the prompt word, the text encoding vector is input into CLIP and T5 respectively. CLIP and T5 are used to encode the text encoding vector respectively to obtain two text vectors.

[0105] Step 406: The image vector and text vector are processed using the ControlNet model, and the resulting control vector is output to the generation module in the Flux model.

[0106] like Figure 5 As shown, the ControlNet model includes a convolutional network and DiT. The input of the convolutional network is the input of the ControlNet model, and the output of the convolutional network is connected to the input of DiT. The output of DiT is the output of the ControlNet model. The input of DiT is also connected to the outputs of CLIP and T5.

[0107] Specifically, the ControlNet model is used to process image vectors and text vectors, and the resulting control vectors are output to the generation module in the Flux model. This can include: using a convolutional network to convolve the image vectors to obtain convolutional vectors; and using DiT to process the convolutional vectors and text vectors, and outputting the resulting control vectors to the generation module in the Flux model.

[0108] Figure 5 The vector sequence in the image is the image vector.

[0109] The ControlNet model is essentially a method for fine-tuning large models on small datasets. It can achieve the effect of controlled condition-guided generation while maintaining the original general generation capabilities.

[0110] Step 407: The generation module processes the noisy image vector, text vector, and control vector to obtain the predicted noise.

[0111] Specifically, the generation module processes the noisy image vector, text vector, and control vector to obtain the predicted noise. This can include: processing the noisy image vector, text vector, and control vector using DiT to obtain an intermediate vector; and processing the intermediate vector using MLP to obtain the predicted noise.

[0112] Step 408: Calculate the first loss value based on the predicted noise and random noise, and train the Flux-ControlNet model based on the first loss value.

[0113] In summary, the training method for the multimodal hybrid generative model provided in this application includes a MixIT model and a Flux-ControlNet model. The MixIT model is used to generate multimodal data containing text and images, while the Flux-ControlNet model is used to generate high-quality images. Through a two-stage approach, text and image generation and image quality are decoupled, maintaining the contextual consistency of text and image generation while ensuring the quality of image generation.

[0114] Based on the Flux model of corrective flow generation, combined with ControlNet model fine-tuning technology, it can not only enhance the image quality of image-text generation based on small datasets with short training cycles and low resource consumption, but also adjust image details individually in combination with prompt words, achieving fine adjustment of image details.

[0115] The Flux-ControlNet model uses some of the image vectors predicted by the MixIT model during training, thus bridging the gap between the training and inference scenarios of the Flux-ControlNet model.

[0116] like Figure 6 As shown below, the usage of the multimodal hybrid generative model is explained, specifically including the following steps:

[0117] Step 601: Obtain the second text and the third image, encode the second text using a text segmenter, encode the third image using an image segmenter, and process the encoding results using the MixIt model to obtain the text index sequence and the image index sequence.

[0118] like Figure 7 As shown, the second text is "Write an article with illustrations about pandas", and the third image is a picture of a panda.

[0119] Step 602: Use a text segmenter to decode the text index sequence to obtain the third text, which includes the generated text and image location markers.

[0120] like Figure 7 As shown, the third text is: "Title: Why are giant pandas considered national treasures? Do you know? Body: xxxx" <ins>xxxx <ins>"xxxx". Among them, <ins>Image location markers are used to mark the locations where high-quality images generated by the Flux-ControNet model are inserted.

[0121] Step 603: Generate prompt words based on the second text and the generated text.

[0122] like Figure 7 As shown, the prompt is "Please output an image description based on the writing requirements and the manuscript".

[0123] Step 604: Generate an image vector based on the image index sequence.

[0124] The image vector mentioned in this step is Figure 7 The vector sequence in.

[0125] Step 605: The Flux-ControlNet model is used to process the preset image resolution, prompt words, random noise, and image vectors to obtain the fourth image.

[0126] In this embodiment, the image resolution can also be preset, and the specific value can be set according to business needs. For example... Figure 7 As shown, the image resolution is 960×480.

[0127] Step 606: Insert the fourth image into the image position mark in the third text to obtain the final image and text generation result.

[0128] like Figure 7 As shown, the Flux model generates a panda image, which is then inserted into the third text. <ins>You can get an article containing high-quality images by clicking here.

[0129] like Figure 8 The diagram illustrates a structural block diagram of a training apparatus for a multimodal hybrid generative model according to an embodiment of this application. This training apparatus can be applied to a computer device. The multimodal hybrid generative model includes a MixIT model and a Flux-ControlNet model. The Flux-ControlNet model includes a Flux model and a ControlNet model. The apparatus includes:

[0130] The first acquisition module 810 is used to acquire multiple first training samples, encode the first text in the first training samples using a text segmenter, encode the first image in the first training samples using an image segmenter, and train the Janus model based on the encoding results and the ground truth in the first training samples to obtain the MixIt model.

[0131] The second acquisition module 820 is used to acquire multiple second training samples and generate a noisy image vector based on the second training samples. The noisy image vector is generated by adding random noise to the image vector.

[0132] The encoding module 830 is used to encode the prompt words in the second training sample using the text processing module in the Flux model to obtain text vectors;

[0133] The control module 840 is used to process image vectors and text vectors using the ControlNet model and output the resulting control vectors to the generation module in the Flux model.

[0134] The prediction module 850 is used to process the noisy image vector, text vector and control vector using the generation module to obtain the predicted noise;

[0135] Training module 860 is used to calculate a first loss value based on predicted noise and random noise, and to train the Flux-ControlNet model based on the first loss value.

[0136] In an optional embodiment, the second acquisition module 820 is further configured to:

[0137] When the second training sample includes the second image, prompt words, and random noise, the second image is encoded using VQ-VAE in the Flux model to obtain an image vector; random noise is added to the image vector to obtain a noisy image vector.

[0138] When the second training sample includes image vectors predicted by the MixIT model, prompt words, and random noise, the random noise is added to the image vector to obtain a noisy image vector.

[0139] In an optional embodiment, the encoding module 830 is further configured to:

[0140] When the text processing module includes CLIP and T5, the text segmenter is used to encode and embed the prompt words to obtain the text encoding vector.

[0141] The text vector is obtained by encoding the text using CLIP and T5 respectively.

[0142] In an optional embodiment, the control module 840 is further configured to:

[0143] When the ControlNet model includes a convolutional network and DiT, the convolutional network is used to convolve the image vector to obtain a convolutional vector.

[0144] DiT is used to process the convolution vector and text vector, and the resulting control vector is output to the generation module in the Flux model.

[0145] In an optional embodiment, the prediction module 850 is further configured to:

[0146] When the generation module includes DiT and MLP, DiT is used to process the noisy image vector, text vector and control vector to obtain the intermediate vector;

[0147] The intermediate vector is processed using MLP to obtain the prediction noise.

[0148] In an optional embodiment, the first acquisition module 810 is further configured to:

[0149] The first text in the first training sample is encoded using a text segmenter to obtain a text index sequence;

[0150] The first image in the first training sample is encoded using an image segmenter to obtain an image index sequence;

[0151] The text index sequence and the image index sequence are mixed to obtain a mixed vector;

[0152] The sliding window in the Janus model is used to extract vector data from the mixture vector, and the predicted value of the next vector data is generated based on the vector data.

[0153] The second loss value is calculated based on the predicted value and the true value of the next vector data in the first training sample. The Janus model is then trained based on the second loss value to obtain the MixIt model.

[0154] In an optional embodiment, the device further includes:

[0155] The third acquisition module is used to acquire the second text and the third image. It uses a text segmenter to encode the second text and an image segmenter to encode the third image. It then uses the MixIt model to process the encoding results to obtain the text index sequence and the image index sequence.

[0156] The decoding module is used to decode the text index sequence using a text segmenter to obtain the third text, which includes the generated text and image location markers.

[0157] The prompt word generation module is used to generate prompt words based on the second text and the generated text.

[0158] The vector generation module is used to generate image vectors based on the image index sequence;

[0159] The image generation module is used to process the preset image resolution, prompt words, random noise, and image vectors using the Flux-ControlNet model to obtain the fourth image;

[0160] The image and text generation module is used to insert the fourth image into the image position mark in the third text to obtain the final image and text generation result.

[0161] In summary, the training device for the multimodal hybrid generative model provided in this application embodiment includes a MixIT model and a Flux-ControlNet model. The MixIT model is used to generate multimodal data containing text and images, while the Flux-ControlNet model is used to generate high-quality images. Through a two-stage approach, text and image generation and image quality are decoupled, maintaining the contextual consistency of text and image generation while ensuring the quality of image generation.

[0162] Based on the Flux model of corrective flow generation, combined with ControlNet model fine-tuning technology, it can not only enhance the image quality of image-text generation based on small datasets with short training cycles and low resource consumption, but also control the high-quality image generation according to the set image resolution, and adjust the image details individually in combination with prompt words, thus achieving fine adjustment of image details.

[0163] The Flux-ControlNet model uses some of the image vectors predicted by the MixIT model during training, thus bridging the gap between the training and inference scenarios of the Flux-ControlNet model.

[0164] One embodiment of this application provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the training method for the multimodal hybrid generative model as described above.

[0165] One embodiment of this application provides a computer device, which includes a training apparatus for the above-described arbitrary multimodal hybrid generative model.

[0166] It should be noted that the multimodal hybrid generative model training device provided in the above embodiments is only illustrated by the division of the above functional modules when training the multimodal hybrid generative model. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the multimodal hybrid generative model training device can be divided into different functional modules to complete all or part of the functions described above. In addition, the multimodal hybrid generative model training device and the multimodal hybrid generative model training method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0167] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0168] The above description is not intended to limit the embodiments of this application. Any adjustments, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of the embodiments of this application.< / ins> < / ins> < / ins> < / ins>

Claims

1. A training method for a multimodal hybrid generative model, characterized in that, The multimodal hybrid generative model includes a MixIT model and a Flux-ControlNet model, wherein the Flux-ControlNet model includes a Flux model and a ControlNet model, and the method includes: Multiple first training samples are obtained, the first text in the first training samples is encoded using a text segmenter, the first image in the first training samples is encoded using an image segmenter, and the Janus model is trained based on the encoding results and the ground truth in the first training samples to obtain the MixIT model. Multiple second training samples are obtained, and a noisy image vector is generated based on the second training samples. The noisy image vector is generated by adding random noise to the original image vector. The text processing module in the Flux model is used to encode the prompt words in the second training sample to obtain a text vector; The ControlNet model is used to process the original image vector and the text vector, and the resulting control vector is output to the generation module in the Flux model. The generation module processes the noisy image vector, the text vector, and the control vector to obtain predicted noise; A first loss value is calculated based on the predicted noise and the random noise, and the Flux-ControlNet model is trained based on the first loss value.

2. The training method for the multimodal hybrid generative model according to claim 1, characterized in that, The step of generating a noisy image vector based on the second training sample includes: When the second training sample includes a second image, a prompt word, and random noise, the second image is encoded using the Vector Quantization VQ-Variational Autoencoder (VAE) in the Flux model to obtain the original image vector; the random noise is then added to the original image vector to obtain a noisy image vector. When the second training sample includes the original image vector predicted by the MixIT model, the prompt word, and random noise, the random noise is added to the original image vector to obtain a noisy image vector.

3. The training method for the multimodal hybrid generative model according to claim 1, characterized in that, The text processing module in the Flux model is used to encode the prompt words in the second training sample to obtain a text vector, including: When the text processing module includes a contrastive language-image pre-trained CLIP and a text-to-text transfer converter T5, the text segmenter is used to encode and embed the prompt words to obtain a text encoding vector. The text encoding vector is obtained by encoding the text using CLIP and T5 respectively.

4. The training method for the multimodal hybrid generative model according to claim 1, characterized in that, The process of using the ControlNet model to process the original image vector and the text vector, and outputting the resulting control vector to the generation module in the Flux model, includes: When the ControlNet model includes a convolutional network and a DiT spread transformer, the original image vector is convolved using the convolutional network to obtain a convolutional vector. The convolution vector and the text vector are processed using the DiT method, and the resulting control vector is output to the generation module in the Flux model.

5. The training method for the multimodal hybrid generative model according to claim 1, characterized in that, The step of processing the noisy image vector, the text vector, and the control vector using the generation module to obtain predicted noise includes: When the generation module includes DiT and Multilayer Perceptron (MLP), the DiT is used to process the noisy image vector, the text vector, and the control vector to obtain an intermediate vector; The intermediate vector is processed using the MLP to obtain the prediction noise.

6. The training method for the multimodal hybrid generative model according to claim 1, characterized in that, The process of encoding the first text in the first training sample using a text segmenter, encoding the first image in the first training sample using an image segmenter, and training the Janus model based on the encoding results and ground truth values ​​in the first training sample to obtain the MixIT model includes: The first text in the first training sample is encoded using a text segmenter to obtain a text index sequence; The image segmenter is used to encode the first image in the first training sample to obtain an image index sequence; The text index sequence and the image index sequence are mixed to obtain a mixed vector; The sliding window in the Janus model is used to extract vector data from the mixed vector, and the predicted value of the next vector data is generated based on the vector data. The second loss value is calculated based on the predicted value and the true value of the next vector data in the first training sample. The Janus model is then trained based on the second loss value to obtain the MixIT model.

7. The training method for the multimodal hybrid generative model according to any one of claims 1 to 6, characterized in that, The method further includes: The second text and the third image are obtained. The second text is encoded using the text segmenter, and the third image is encoded using the image segmenter. The encoding results are processed using the MixIT model to obtain a text index sequence and an image index sequence. The text index sequence is decoded using the text segmenter to obtain a third text, which includes the generated text and image location markers. Generate prompt words based on the second text and the generated text; Generate the original image vector based on the image index sequence; The Flux-ControlNet model is used to process the preset image resolution, the prompt words, random noise, and the original image vector to obtain the fourth image; The fourth image is inserted into the image position mark in the third text to obtain the final image and text generation result.

8. A training device for a multimodal hybrid generative model, characterized in that, The multimodal hybrid generative model includes a MixIT model and a Flux-ControlNet model, the Flux-ControlNet model includes a Flux model and a ControlNet model, and the device includes: The first acquisition module is used to acquire multiple first training samples, encode the first text in the first training samples using a text segmenter, encode the first image in the first training samples using an image segmenter, and train the Janus model based on the encoding results and the ground truth in the first training samples to obtain the MixIT model. The second acquisition module is used to acquire multiple second training samples and generate a noisy image vector based on the second training samples. The noisy image vector is generated by adding random noise to the original image vector. The encoding module is used to encode the prompt words in the second training sample using the text processing module in the Flux model to obtain a text vector; The control module is used to process the original image vector and the text vector using the ControlNet model, and output the resulting control vector to the generation module in the Flux model; The prediction module is used to process the noisy image vector, the text vector, and the control vector using the generation module to obtain predicted noise; The training module is used to calculate a first loss value based on the predicted noise and the random noise, and to train the Flux-ControlNet model based on the first loss value.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to implement the training method for the multimodal hybrid generative model as described in any one of claims 1 to 7.

10. A computer device, characterized in that, The computer device includes: a training apparatus for the multimodal hybrid generative model as described in claim 8.

Images