Text-to-image model, model training method and apparatus, image generation method and apparatus, and electronic device

Abstract

The present application relates to the field of artificial intelligence. Disclosed are a text-to-image model, a model training method and apparatus, an image generation method and apparatus, and an electronic device. The text-to-image model comprises: a public text encoder network, which is used for converting image description information into a public text feature vector, the image description information being used for describing image content contained in an image to be generated by the text-to-image model; a user preference text encoder network, which is used for outputting a user preference text feature vector on the basis of the common text feature vector, the user preference text feature vector being used for indicating the image content contained in the image to be generated by the text-to-image model, and a drawing style corresponding to the image content; a diffusion model, which is used for performing noise prediction processing on the public text feature vector and the user preference text feature vector to obtain a first predicted noise, the first predicted noise being used for denoising the user preference text feature vector; and an image decoder, which is used for performing decoding processing on the denoised user preference text feature vector and outputting an image.

Description

Image generation models, model training, image generation methods, devices and electronic equipment

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202410809420.8, filed in China on June 21, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application belongs to the field of artificial intelligence technology, specifically relating to a text-to-image model, model training, image generation method, device, and electronic device. Background Technology

[0004] Currently, Artificial Intelligence (AI) is widely used in the multimedia field. For example, AI-generated image algorithms are widely used in wallpaper, avatar, game, and animation design, greatly improving the efficiency of image creation and design. However, in related technologies, when using models to generate images, without specifying a style based on user input, the style of the images output by the model is relatively random, making it difficult to obtain the image style desired by the user. Summary of the Invention

[0005] The purpose of this application is to provide a text-based image model, model training, image generation method, apparatus, and electronic device that can achieve the image style output by the text-based image model to the image style required by the user.

[0006] In a first aspect, embodiments of this application provide a text-based image model, including:

[0007] A public text encoder network is used to convert image description information into public text feature vectors, wherein the image description information is used to describe the image content contained in the image generated by the text-generated image model.

[0008] A user preference text encoder network is used to output a user preference text feature vector based on the public text feature vector. The user preference text feature vector is used to indicate the image content contained in the image generated by the text-to-image model and the painting style corresponding to the image content. The user preference text encoder network is connected to the public text encoder network.

[0009] A diffusion model is used to perform noise prediction processing on the public text feature vector and the user preference text feature vector to obtain a first predicted noise, which is used to denoise the user preference text feature vector; the diffusion model is connected to the public text encoder network and the user preference text encoder network respectively.

[0010] The image decoder is used to decode the denoised user preference text feature vector and output an image.

[0011] Secondly, a model training method is provided for training the text-based graph model as described in the first aspect, the method comprising:

[0012] At least one training sample is obtained, each training sample including image description information, image sample and painting style sample, the training sample being obtained based on user preference information;

[0013] The initial model is trained using the at least one training sample to obtain the text-to-image model; the text-to-image model is used to generate M images corresponding to the user preference information based on the image description information, where M is an integer greater than 1.

[0014] Thirdly, an image generation method is provided, including:

[0015] The image description information is input into the common text encoder network of the text-generated image model as described in the first aspect to obtain a common text feature vector, wherein the image description information is used to indicate the image content contained in the image generated by the text-generated image model.

[0016] The public text feature vector is input into the user preference text encoder network of the text-generated image model, and the user preference text feature vector is output; wherein, the user preference text feature vector is used to indicate the image content contained in the image generated by the text-generated image model and the painting style corresponding to the image content;

[0017] The user preference text feature vector is denoised using the diffusion model of the text graph model to obtain the denoised user preference text feature vector.

[0018] The image decoder of the text-based graph model decodes the denoised user preference text feature vector to obtain the image.

[0019] Fourthly, embodiments of this application provide a model training apparatus for training the text-based image model as described in the first aspect, comprising:

[0020] The first acquisition module is used to acquire at least one training sample, each training sample including image description information, image sample and painting style sample, the training sample being obtained based on user preference information;

[0021] The training module is used to train the initial model using the at least one training sample to obtain the text-to-image model; the text-to-image model is used to generate M images corresponding to the user preference information based on the image description information, where M is an integer greater than 1.

[0022] Fifthly, an image generation apparatus is provided, comprising:

[0023] The second acquisition module is used to input image description information into the public text encoder network of the text-generated image model as described in the first aspect to obtain a public text feature vector. The image description information is used to indicate the image content contained in the image generated by the text-generated image model.

[0024] The third acquisition module is used to input the public text feature vector into the user preference text encoder network of the text-generated image model and output the user preference text feature vector; wherein, the user preference text feature vector is used to indicate the image content contained in the image generated by the text-generated image model and the painting style corresponding to the image content;

[0025] The fourth acquisition module is used to perform denoising processing on the user preference text feature vector through the diffusion model of the text-generated image model to obtain the denoised user preference text feature vector.

[0026] The fifth acquisition module is used to decode the denoised user preference text feature vector through the image decoder of the text-to-image model to obtain an image.

[0027] In a sixth aspect, embodiments of this application provide an electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the method described in the second aspect.

[0028] In a seventh aspect, embodiments of this application provide an electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the method described in the third aspect.

[0029] Eighthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the second or third aspect.

[0030] Ninthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the methods described in the second or third aspect.

[0031] In a tenth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the second or third aspect.

[0032] In this embodiment, the training samples of the text-based graph model are obtained based on user preference information. The user preference text encoder network in the text-based graph model can learn user preference information, so that the text-based graph model obtained based on the training samples can generate an image style corresponding to the user preference information, thereby ensuring that the image style output by the text-based graph model is the image style required by the user. Attached Figure Description

[0033] Figure 1 is a schematic diagram of the structure of a text image model provided in some embodiments of this application;

[0034] Figure 2 is a schematic diagram of a common text encoder provided in some embodiments of this application;

[0035] Figure 3 is a schematic diagram of an image decoder provided in some embodiments of this application;

[0036] Figure 4 is a schematic diagram of an image encoder provided in some embodiments of this application;

[0037] Figure 5 is a schematic diagram of a user preference text encoder provided in some embodiments of this application;

[0038] Figure 6 is a flowchart illustrating the model training method provided in some embodiments of this application;

[0039] Figure 7 is a schematic diagram of a user preference-based data flywheel provided in some embodiments of this application;

[0040] Figure 8 is a schematic diagram of inputting drawing instructions to a text image model according to some embodiments of this application;

[0041] Figure 9 is a schematic diagram of image generation based on a text-based image model provided in some embodiments of this application;

[0042] Figure 10 is a schematic diagram of the training process of a user preference text encoder network provided in some embodiments of this application;

[0043] Figure 11 is a schematic diagram of a first diffusion model provided by some embodiments of this application;

[0044] Figure 12 is a schematic diagram of the training process of a first diffusion model or a second diffusion model provided in some embodiments of this application;

[0045] Figure 13 is a schematic diagram of an image generation method provided in some embodiments of this application;

[0046] Figure 14 is a schematic diagram of image generation based on a text-based image model provided in some embodiments of this application;

[0047] Figure 15 is a schematic diagram of the modules of a model training device provided in some embodiments of this application;

[0048] Figure 16 is a schematic diagram of the modules of an image generation apparatus provided in some embodiments of this application;

[0049] Figure 17 is a structural block diagram of an electronic device provided in some embodiments of this application;

[0050] Figure 18 is a structural block diagram of an electronic device provided in some embodiments of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0052] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0053] The following description, in conjunction with the accompanying drawings, details the text-to-image model, model training method, and image generation method provided in this application through specific embodiments and application scenarios.

[0054] It should be noted that the model training method and image generation method provided in this application can be executed by electronic devices such as mobile phones, tablets, laptops, PDAs, and in-vehicle electronic devices. Some embodiments of this application use electronic devices as the executing entity to illustrate the model training method and image generation method provided in this application.

[0055] As shown in Figure 1, this application embodiment provides a text-based image model, including:

[0056] (1) A public text encoder network is used to convert image description information into public text feature vectors, wherein the image description information is used to describe the image content contained in the image generated by the text-generated image model.

[0057] The image description information mentioned above can also be described as drawing instructions; for example, the image description information could be to draw a female figure.

[0058] This common text encoder network converts image description information into common text feature vectors, enabling subsequent interaction between these text feature vectors and image feature vectors to achieve text-controlled image generation. Since the text feature vectors extracted from image description information in this common text encoder network have no style preference, the text feature vectors output by this network are called common text feature vectors.

[0059] Optionally, one network structure of the public text encoder network, as shown in Figure 2, includes a tokenizer module and 12 self-attention modules (Transformer Blocks). Each self-attention module includes a self-attention operator, a vector addition operator, a normalization operator, and a feed forward operator. QKV are the three input vectors of the attention mechanism, representing the query, key, and value, respectively. In the self-attention mechanism, these three vectors come from the same input mapping; in the cross-attention mechanism, the query comes from an independent input mapping, while the key and value come from the same input mapping.

[0060] Specifically, the image description information first passes through the Tokenizer module, which converts the text into a numerical vector with dimensions [128, 768], where 128 represents the maximum number of character tokens and 768 is the representation vector for each character. Then, the [128, 768]-dimensional numerical vector is input into the self-attention module for self-attention calculation, resulting in a [128, 768]-dimensional vector, denoted as the Text Encoder Vector (TEV), which is the aforementioned common text feature vector.

[0061] (2) A user preference text encoder network is used to output a user preference text feature vector based on the public text feature vector. The user preference text feature vector is used to indicate the image content contained in the image generated by the text-generated image model and the painting style corresponding to the image content. The user preference text encoder network is connected to the public text encoder network.

[0062] This user preference text encoder network is mainly used to convert public text feature vectors into user preference text feature vectors with user style preferences, so as to facilitate subsequent interaction with image feature vectors and realize the generation of images with style preferences.

[0063] (3) A diffusion model is used to perform noise prediction processing on the public text feature vector and the user preference text feature vector to obtain a first prediction noise, which is used to denoise the user preference text feature vector; the diffusion model is connected to the public text encoder network and the user preference text encoder network respectively.

[0064] Optionally, the diffusion model can implement the one-stage noise prediction processing and the two-stage noise prediction processing shown in Figure 1.

[0065] The first-stage noise prediction process includes performing noise prediction processing on the public text feature vector to obtain a second predicted noise, which is used to denoise the public text feature vector. The second-stage noise prediction process includes performing noise prediction processing on the user preference text feature vector based on the denoised public text feature vector to obtain a first predicted noise.

[0066] (4) Image decoder, used to decode the user preference text feature vector after noise reduction and output the image.

[0067] This image decoder is used to convert user preference text feature vectors into images, thereby generating images.

[0068] Optionally, the structure of this image decoder, as shown in Figure 3, includes three deconvolution layers: DeconvBlock1, DeconvBlock2, and DeconvBlock3. The image reconstruction feature LatentB (dimension [32, 64, 64]) undergoes a series of deconvolution operations to finally generate an image of [3, 512, 512]. The calculation processes of DeconvBlock1, DeconvBlock2, and DeconvBlock3 are the same, only the deconvolution kernel, stride, and number of channels differ. Here, we will use DeconvBlock2 as an example to explain the calculation process:

[0069] (1): The image [32,64,64] output by DeconvBlock1 is first processed by the deconvolution layer Deconv2_1 to obtain the feature matrix [16,128,128]. The deconvolution kernel size is 3x3, the stride is 2, and the number of output channels is 16.

[0070] (2): The feature matrix of [16,128,128] dimensions remains the same after passing through the Relu2_1 layer;

[0071] (3): Conv2_2, Relu2_2, Conv2_3, and Relu2_3 are the usual convolution and Relu operations. The final output feature matrix of DeconvBlock2 has a dimension of [16, 128, 128].

[0072] Optionally, the text image model in this embodiment of the application further includes an encoder, which is used to encode an image into an image feature vector, and a subsequent image decoder is used to decode the image feature vector into an image. This process is the inverse process performed by the image encoder.

[0073] The image encoder is described below.

[0074] As shown in Figure 4, the 3D representation matrix [3, 512, 512] of the image is first transformed into a [32, 64, 64]-dimensional image feature vector LatentA through a series of convolutional neural network (CNN) convolution operations. The calculation processes of convolution module 1 (ConvBlock1), convolution module 2 (ConvBlock2), and convolution module 3 (ConvBlock3) are the same, except that the stride of the convolution kernel is different. Here, we will take ConvBlock1 as an example to explain the calculation process:

[0075] (1): The image [3,512,512] first passes through the convolutional layer Conv1_1 to calculate the feature matrix [8,256,256]. The convolutional kernel size is 3x3, the stride is 2, and the number of output channels is 8.

[0076] (2): The feature matrix of [8,256,256] dimensions remains the same after passing through the Relu1_1 layer;

[0077] (3): Conv1_2 and Relu1_2 repeat the process of Step1 and Step2, except that the convolution stride of Conv1_2 is changed to 1, so that the dimension of the output feature matrix is ​​still [8,256,256].

[0078] In the text-based image model of this application embodiment, the user preference text encoder network can learn user preference information. By training the text-based image model with training samples containing user preference information, the text-based image model can generate image styles corresponding to user preference information, thereby ensuring that the image style output by the text-based image model can meet the user's preferences.

[0079] Optionally, the user preference text encoder network includes:

[0080] There are N self-attention modules, which are used to generate text feature vectors based on a common text feature vector and a corresponding painting style; where each self-attention module corresponds to a painting style, and N is an integer greater than 1.

[0081] A gating network is used to output the gating weights corresponding to each of the self-attention modules, and the gating weights are used to indicate the probability that the user selects the drawing style corresponding to the self-attention module; wherein, the gating network is connected to each of the N self-attention modules respectively;

[0082] The processing module is used to determine the text feature vector generated by the self-attention module with the largest gating weight as the user preference text feature vector.

[0083] As an optional implementation, the network structure of this user preference text encoder network is shown in Figure 5, including a preset number of expert networks and gate networks. Each expert network includes one self-attention module (OneTransformerBlock), and the preset number can be represented by N. The architecture of each expert network is the same as the architecture of one self-attention module in Figure 4. Here, the expert network only stacks one self-attention module (in the common text feature vector encoder network, 12 self-attention modules are stacked), so the vector dimension remains unchanged, still [128, 768].

[0084] The aforementioned gating network consists of N feedforward operators, and each feedforward operator is processed by a sigmoid layer to calculate a gating weight g. i (i = 1, 2, ..., N), representing the probability of selecting this style. The processor is based on gating weights g1, g2, ..., g N Select the style with the highest score, let's say it's style k (1≤k≤N). Then the feature vector output by Expertk is used as the user preference text feature vector, denoted as StyleTextEmbedding.

[0085] The aforementioned user preference text encoder network can learn user preference information. The gating network calculates the probability of selecting each painting style based on the learned user preference information. The processor determines the text feature vector generated by the self-attention module with the highest probability as the user preference text feature vector, thereby obtaining a text feature vector with user preference information, namely the aforementioned user preference text feature vector, so as to facilitate subsequent interaction with the image feature vector and realize the generation of an image with style preference.

[0086] Optionally, the diffusion model includes:

[0087] A first diffusion model, connected to the public text encoder network, is used to perform noise prediction processing on the public text feature vector to obtain a second predicted noise, which is used to denoise the public text feature vector.

[0088] The second diffusion model, the second diffusion module is used to perform noise prediction processing on the user preference text feature vector based on the denoised public text feature vector, to obtain the first predicted noise.

[0089] The first diffusion model is used to implement one-stage noise prediction processing, and the second diffusion model is used to implement two-stage noise prediction processing.

[0090] The purpose of employing a two-stage text-controlled image feature generation is as follows: the first stage uses common text feature vectors, which aligns the generated text and image content. The second stage uses user-preferred text features, which further generate corresponding style-specific image features from the first stage's image features, achieving style alignment between text and image. Through these two core steps, alignment can be achieved both in image content and style; for example, if the image content is a female figure, the style could be ink painting.

[0091] In the text-to-image model of this application embodiment, the user preference text encoder network can learn user preference information. By training the text-to-image model with training samples containing user preference information, the text-to-image model can generate an image style corresponding to the user preference information, thereby ensuring that the image style output by the text-to-image model can meet the user's preference. Furthermore, through two-stage noise prediction processing, the aesthetics of the image can be improved while ensuring the consistency between the text and the image.

[0092] As shown in Figure 6, this application embodiment also provides a model training method for training the text-based image model as described above, including:

[0093] Step 601: Obtain at least one training sample, each training sample including image description information, image sample and painting style sample, the training sample being obtained based on user preference information.

[0094] In this embodiment of the application, training samples can be obtained based on images output by the Chinese raw image model within a preset time period and images selected by the user from the images output by the model. For example, the preset time period can be one month.

[0095] Step 602: Train the initial model using the at least one training sample to obtain the text-based image model; the text-based image model is used to generate M images corresponding to the user preference information based on the image description information, where M is an integer greater than 1.

[0096] Optionally, the initial model is a model that generates M images based on image description information, and the image style of the M images is an image style randomly selected by the model from preset image styles.

[0097] For example, the text-based image model is used to generate M images corresponding to a target painting style based on painting instructions. The target painting style is the painting style corresponding to user preference information.

[0098] For example, as shown in Figure 7, the initial model generates an image based on the user's drawing instructions. The user makes a preference selection in the generated image, and then the user's behavior data is recorded and collected. This behavior data is the user's behavior of selecting an image in the image output by the model. Then, the model is iteratively trained based on the user's behavior data to produce a trained text-to-image model. The text-to-image service is updated based on the produced model. In this way, the style of the image generated by the text-to-image model can be more similar to the image style required by the user.

[0099] The model training method in this application uses training samples obtained based on user preference information to train the initial model, so that the user preference text encoder network in the text-to-image model can learn user preference information. Thus, the text-to-image model obtained based on the training samples can generate image styles corresponding to user preference information, thereby ensuring that the image style output by the text-to-image model can meet the user's preferences.

[0100] Optionally, obtaining at least one training sample includes:

[0101] Obtain image description information input by the user;

[0102] Input image description information into the Wensheng image model and output M images, each image corresponding to a painting style, and different images corresponding to different painting styles;

[0103] The image selected by the user from the M images is used as an image sample, and the painting style corresponding to the image selected by the user is used as a painting style sample;

[0104] The training samples are obtained based on the image description information, image samples, and painting style samples.

[0105] Optionally, the above image description information is referred to as a drawing instruction. This drawing instruction is used to instruct the text-based image model to generate a target object, which can be a person, an animal, or a landscape, etc. For example, as shown in Figure 8, the user inputs a drawing instruction according to the prompt information 801, which is "Input drawing instruction". The user inputs a drawing instruction 802, which is "Draw a female figure". Assuming that the text-based image model supports 20 candidate styles as shown in Table 1, as shown in Figure 9, the text-based image model randomly outputs images of four drawing styles based on this drawing instruction. For example, the images of the four drawing styles are ink painting style image 901, watercolor style image 902, oil painting style image 903, and photographic style image 904.

[0106] The image generated by the Wensheng Image Model displays a selection button below each painting style, allowing users to choose an image of that style, as shown in Figure 9. Each style image offers two buttons: "Draw a Similar Image" and "Download." Clicking either "Draw a Similar Image" or "Download" indicates that the user has selected the corresponding painting style. For example, if a user clicks the "Draw a Similar Image" button below a traditional Chinese ink painting image, four similar ink painting images will be generated. If the user clicks the "Download" button, the corresponding image will be directly downloaded to the phone's photo album. Based on the user's selection, a training sample is obtained: <a female figure, ink painting, and corresponding images of female figures in ink painting>. This training sample consists of a triplet of data. By collecting training samples within the target time period, the training samples for the Wensheng Image Model are obtained.

[0107] Table 1

[0108] In this embodiment, training samples are obtained based on the images output by the text-based image model within a preset time and the images selected by the user from the images output by the model. These training samples are designed to reflect the user's preferred choices. Subsequently, the text-based image model is trained using these training samples that contain user preference information. This enables the text-based image model to generate image styles that correspond to the user's preference information, thereby ensuring that the image style output by the text-based image model meets the user's preferences.

[0109] Optionally, the step of training the initial model using the at least one training sample to obtain the text-based image model includes:

[0110] Based on the image description information, painting style samples, and the first loss function in the training samples, at least one network parameter of the user preference text encoder network in the text-generated image model is updated.

[0111] Based on image description information in the training samples and a second loss function, update at least one network parameter of the first diffusion model in the text image model.

[0112] Based on the image description information, painting style samples, image samples, and third loss function in the training samples, at least one network parameter of the second diffusion model in the text-based image model is updated.

[0113] Based on the trained user preference text encoder network, the first diffusion model, and the second diffusion model, the text-generated graph model is obtained.

[0114] The training process in this embodiment mainly includes training the user preference text encoder network, the first diffusion model, and the second diffusion model. By training the user preference text encoder network, the first diffusion model, and the second diffusion model, the generated model can be optimized in the direction of user preferences.

[0115] Optionally, based on image description information, painting style samples, and a first loss function in the training samples, at least one network parameter of the user preference text encoder network in the text-based image model is updated, including:

[0116] Based on the image description information and the painting style samples, the probability of the user selecting each of the N painting styles is determined, wherein the selection probability of the first painting style is 1, the selection probability of the second painting style is 0, the first painting style is the painting style corresponding to the painting style sample, and the second painting style is the painting style other than the first painting style among the N painting styles.

[0117] Based on the selection probability, the gating weights corresponding to the N self-attention modules in the user preference text encoder network, and the first loss function, update at least one network parameter of the user preference text encoder network in the text graph model.

[0118] Assuming the first loss function is denoted as L1, as shown in Figure 10, the process includes:

[0119] (1): Gating weight value g i Normalization is performed using an activation function (softmax) to obtain the normalized gate weights p. i As shown in formula (1);

[0120] (2): y i (i = 1, 2, ..., N) represents the style selected by the user. When the user selects style i (e.g., ink painting style), the corresponding y i =1; for other styles, y is 0;

[0121] Step 3: For g i and y i The cross-entropy loss, L1, is calculated as shown in Equation (2).

[0122] Assuming N=3, and the user selects the second style, i.e., y2=1, g2=0.6; y1=0, g1=0.2, y3=0, g3=0.2, then based on the above formula (1), we can obtain... Based on formula (2), we can obtain that

[0123] Here, based on the image description information and painting style samples, the probability of the user selecting each of the N painting styles is determined. According to the selection probability, the gating weights corresponding to the N self-attention modules in the user preference text encoder network, and the first loss function, at least one network parameter of the user preference text encoder network in the text-based image model is updated, so that the user preference text encoder network can learn the user's preferences. This enables the text-based image model to generate image styles corresponding to the user's preference information, thereby ensuring that the image style output by the text-based image model can meet the user's preferences.

[0124] Optionally, updating at least one network parameter of the first diffusion model in the text image model based on image description information in the training samples and the second loss function includes:

[0125] The image description information is input into a common text encoder network, which outputs a common text feature vector.

[0126] The image corresponding to the image description information is input into the image encoder of the text image model, and the first image feature vector is output.

[0127] The first image feature vector with added random noise and the public text feature vector are input into the first diffusion model, and the third predicted noise is output.

[0128] Based on the third predicted noise, the first random noise, and the second loss function, update at least one network parameter of the first diffusion model in the Wensheng graph model.

[0129] In the training process described above, random noise is added to image features, and then the first diffusion model predicts this random noise, thereby achieving both noise addition and denoising of image features. During the image feature denoising process, a text feature vector is introduced, which interacts with the noisy image feature vector to predict the noise, thus realizing the image generation process with text as the control condition. The structure of this first diffusion model is shown in Figure 11.

[0130] (1) The process of constructing a Text Condition Vector (TCV):

[0131] (1.1) Based on the Common Text Embedding (CTE), a linear mapping layer (Project) is passed through to obtain a text encoding vector (TPV) with dimensions [320,768].

[0132] (1.2) Initialize a temporal embedding layer with dimensions [320, 1280], and then pass it through a linear mapping layer (Project) to obtain a temporal mapping embedding with dimensions [320, 768].

[0133] (1.3) The text encoding vector TPV and the time mapping Embedding are added together to obtain the text conditional control vector TCV, with dimensions [320, 768].

[0134] (2) The process of constructing the Image Project Vector (IPV):

[0135] (2.1) Add a random Gaussian noise matrix NoiseA to the image feature vector LatentA through the AddNoise operation to obtain the noisy image latent matrix NoiseLatentA;

[0136] Here, LatentA is the first image feature vector mentioned above, and the first random noise is NoiseA.

[0137] (2.2) Perform a Conv convolution operation on NoiseLatentA to obtain a matrix with dimensions [320, 64, 64]. The convolution kernel size of this matrix is ​​3x3, the stride is 1, and the number of output channels is 320. This matrix is ​​then reshaped to obtain an image project vector (IPV) with dimensions [320, 64x64].

[0138] (3) Computation process of the DiTBlock based on the Transformer architecture: The entire computation process consists of several stacked DiTBlocks. The following is an explanation of one DiTBlock, namely the small DiTBlock on the right side of Figure 11:

[0139] (3.1) The CrossAttention module uses the image encoding mapping vector IPV as the Query and the text conditional control vector TCV as the Key and Value to calculate the cross attention mechanism. The meanings of Add, Norm, and FeedForward are the same as those of the operators described above, and will not be repeated here.

[0140] (3.2): Repeat the process of (3.1), using the output of (3.1) as the new query, while keeping the key and value the same as in Step 1. Repeat this process for several DiTBlocks to complete the calculation process of the first diffusion model.

[0141] (4) The process of constructing the image reconstruction feature LatentA:

[0142] (4.1) The vector output by a series of DiTBlocks has a dimension of [320, 64, 64]. This vector is processed by a convolutional Conv_out layer to obtain a vector of dimension [32, 64, 64], which is denoted as the image prediction noise matrix PredNoiseA; PredNoiseA is the third prediction noise.

[0143] (4.2) Subtract the image prediction noise matrix PredNoiseA from the image feature vector NoiseLatentA to obtain the final image reconstruction feature LatentB (i.e., LatentB = NoiseLatentA - PredNoiseA), with dimensions [32, 64, 64], and output this matrix.

[0144] As shown in Figure 12, the output of the first-stage noise prediction network is PredNoiseA. A mean-square error (MSE) loss function, denoted as L2, is constructed using the initially added random noise matrix NoiseA and PredNoiseA. This L2 is the second loss function. By training this loss function, the network parameters of the first diffusion model can be adjusted through backpropagation, so that the PredNoiseA output by the first diffusion model can approximate the random noise matrix NoiseA. Finally, the formula LatentB = NoisedLatentA - PredNoiseA is obtained.

[0145] By training the first diffusion model, it is possible to control the generation of images through common text feature vectors. Since the training process of the first diffusion model uses common text feature vectors, the trained first diffusion model can achieve consistency between text content and image content.

[0146] Optionally, updating at least one network parameter of the second diffusion model in the text-based image model based on image description information, painting style samples, image samples, and a third loss function from the training samples includes:

[0147] The image description information is input into a common text encoder network, which outputs a common text feature vector.

[0148] The public text feature vector and the painting style sample are input into the user preference text encoder network, and the user preference text feature vector is output.

[0149] The image sample is input into the image encoder in the text image model, and the second image feature vector is output; the second image feature vector can correspond to LatentA in Figure 1;

[0150] The second image feature vector with added random noise and the common text feature vector are input into the first diffusion model, and the fourth predicted noise is output; this fourth predicted noise can correspond to PredNoiseA;

[0151] Based on the fourth prediction noise, the second image feature vector with added second random noise is denoised to obtain the third image feature vector; this third image feature vector can correspond to LatentB in Figure 1.

[0152] The third image feature vector with added third random noise and the user preference text feature vector are input into the second diffusion model to output the fifth predicted noise; the third random noise corresponds to NoiseB in Figure 1, the third image feature vector with added third random noise corresponds to Noise LatentB in Figure 1; the fifth predicted noise corresponds to PredNoiseB in Figure 1.

[0153] Based on the second random noise, the fifth predicted noise, and the third loss function, update at least one network parameter of the second diffusion model in the Wensheng graph model.

[0154] In some embodiments of this application, the training process of the second diffusion model is similar to that of the first diffusion model, except that the control condition is changed from the public text feature vector to the user-preferred text feature vector. The training process of the first diffusion model can be described as a one-stage noise prediction process, and the training process of the second diffusion model can be described as a two-stage noise prediction process, which is as follows:

[0155] (1) Change the common text feature vector (CommonTextEmbedding) in Figure 11 to the user preference text feature vector (StyleTextEmbedding), as shown in Figure 1. That is, use the text feature vector encoded by the user preference text encoder network as the control condition input to the two-stage diffusion model.

[0156] (2) Change the input LatentA to LatentB, the random noise matrix NoiseA to NoiseB, NoiseLatentA to NoiseLatentB, the prediction noise matrix PredNoiseA to PredNoiseB, and the final output LatentB to LatentC, as shown in Figure 1.

[0157] (3) In Figure 11, 15 DiTBLocks were stacked together. The two-stage diffusion model does not need so many network layers, so it was changed to 5 DiTBlocks stacked together.

[0158] Referring to Figure 12, the output of the two-stage noise prediction network is PredNoiseB. An MSE loss function, denoted as L3, is constructed using the initially added random noise matrix NoiseB and PredNoiseB. By training this loss function, the network parameters of the second diffusion model can be adjusted through backpropagation, so that the PredNoiseB output by the second diffusion model can approximate the random noise matrix NoiseB. Finally, the formula LatentC = NoisedLatentB - PredNoiseB is obtained.

[0159] By training the second diffusion model, it is possible to control the generation of images by using user-preferred text feature vectors. Since the training process of the second diffusion model uses user-preferred text feature vectors, the trained second diffusion model can achieve consistency between text content and image painting style.

[0160] By training the first and second diffusion models described above, the images output by the models can be made consistent with the public text feature vector in terms of content and consistent with the user-preferred text feature vector in terms of painting style.

[0161] To better understand the training process described above, the entire network structure is explained as follows.

[0162] The image encoder in the text-controlled image model converts the real image into an image feature vector; the image decoder converts the image feature vector back into the real image. These two processes are inverses of each other, achieving the conversion between the real image and the image feature vector. Since the real image is a matrix of size [3, 512, 512] and the image feature vector is a matrix of size [32, 64, 64], the image feature vector matrix is ​​smaller than the real image matrix, as can be seen from the calculation of the number of elements in these two matrices. Therefore, to achieve text-controlled image generation, it is only necessary to operate in the image feature space, treating the image feature vector as the real image. After text control generates the image feature vector, the image decoder can then convert the image feature vector back into the real image, thus saving computation. The process of text-controlled image feature generation involves one-stage noise prediction and two-stage noise prediction.

[0163] The entire model training process in the embodiments of this application is described below.

[0164] Suppose the training dataset is S, which consists of triples such as <a female figure, ink painting, and corresponding image of the female figure in the ink painting>. During training, the above inference process is performed on each triple in the training dataset S. Then, the three loss functions mentioned above are used to differentiate the network parameters, thereby achieving backpropagation to update the network parameters. When the training of each sample in the training dataset S is completed, the training is terminated, thus producing a version of the text-based image model.

[0165] For example, the specific training process is as follows:

[0166] First, based on the L1 loss function, a user preference text encoder is trained using user preference data such as "<a female figure, ink painting>" until the training set S is exhausted. By training the user preference text encoder network, user preference text features are extracted.

[0167] Then, based on the L2 loss function, the first diffusion model is trained using a method such as <a female figure, or one image randomly selected from four styles of images of a female figure>, until the training set S is exhausted. By training a one-stage noise prediction model, i.e., the first diffusion model, the common text features that control the generation of image content are learned, thereby producing the network parameters of the first diffusion model, which can be described as model parameters or network weights.

[0168] Next, based on the loss function L3, a two-stage noise prediction model, i.e., the second diffusion model, is trained using a format such as <a female figure, ink painting, and corresponding image of a female figure in ink painting>, until the training set S is fully traversed. By training the two-stage noise prediction model, the generation of image styles is controlled by the textual features of user preferences, thereby producing the network parameters of the second diffusion model.

[0169] Finally, based on the overall loss function L = a*L1 + b*L2 + c*L3 composed of three loss functions, the user preference text encoder, the first diffusion model, and the second diffusion model are jointly trained end-to-end. Here, a + b + c = 1, and a, b, and c are hyperparameters that can be set during training, for example, a = b = c = 1 / 3. The network weights of the common text encoder, image encoder, and image decoder are frozen and not trained, thus achieving end-to-end text-to-image training until the training set S is traversed. This process mainly implements end-to-end training, connecting the three trained network structures above to form a complete network structure.

[0170] As shown in Figure 13, this application embodiment also provides an image generation method, including:

[0171] Step 1301: Input the image description information into the common text encoder network of the above-mentioned text-generated image model to obtain the common text feature vector. The image description information is used to indicate the image content contained in the image generated by the text-generated image model.

[0172] The image description information mentioned above can also be described as drawing instructions; for example, the image description information could be to draw a female figure.

[0173] Step 1302: Input the public text feature vector into the user preference text encoder network of the text-generated image model, and output the user preference text feature vector; wherein, the user preference text feature vector is used to indicate the image content contained in the image generated by the text-generated image model and the painting style corresponding to the image content.

[0174] Step 1303: Denoise the user preference text feature vector by using the diffusion model of the text-generated graph model to obtain the denoised user preference text feature vector.

[0175] Step 1304: The image is obtained by decoding the denoised user preference text feature vector through the image decoder of the text-to-image model.

[0176] In this embodiment of the application, since the text-based image model is trained based on user preference information, after inputting image description information into the text-based image model, an image style corresponding to the user preference information can be obtained, thereby ensuring that the image style output by the text-based image model can meet the user's preferences.

[0177] Optionally, the user preference text feature vector is denoised using the diffusion model of the text-generated graph model to obtain a denoised user preference text feature vector, including:

[0178] The public text feature vector is input into the first diffusion model, and the second predicted noise is output.

[0179] Based on the second predicted noise, the public text feature vector is denoised to obtain the first text feature vector.

[0180] The first text feature vector with added fourth random noise and the user preference text feature vector are input into the second diffusion model, and the first predicted noise is output.

[0181] Based on the first predicted noise, the user preference text feature vector is denoised to obtain the denoised user preference text feature vector.

[0182] The specific working principles of the first and second diffusion models have been described in detail in the above embodiments and will not be repeated here.

[0183] After completing the entire training process described above, a new text-to-image model can be generated, thereby updating the text-to-image system's service. Then, based on this new model and user commands, images are generated. Compared to the previous text-to-image service, the model trained with user preferences can automatically calculate the four highest-scoring styles based on user commands, and then generate four images in the corresponding styles to push to the user. The calculation method for the four highest-scoring styles is as follows:

[0184] Based on the aforementioned gating network, the four largest gating weights, g_a, g_b, g_c, and g_d, are calculated. These gating weights are derived using the Sigmoid algorithm, and the four corresponding styles are used as the final styles to generate the images. The generated images thus reflect user preferences and are no longer random styles. For example, if a user inputs "draw a female figure," the gating network calculates scores for 20 styles as shown in Table 2, and selects the four styles with the highest scores: photography, 3D animation, ink painting, and illustration. The interface is then displayed as shown in Figure 14, which shows image 1401 in the photography style, image 1402 in the 3D animation style, image 1403 in the ink painting style, and image 1404 in the illustration style.

[0185] in,

[0186] The formula for calculating sigmoid is as follows:

[0187] Among them, w j These are the parameters of the network model used for training and learning.

[0188] Table 2

[0189] Based on the above process, and referring to the design in Figure 7, continuous training and iteration are performed. This process is repeated to form a data flywheel-driven text graph optimization scheme.

[0190] It should be noted that the training method of the above-mentioned text-based image model in the embodiments of this application can also be applied to text-based video models or text-based audio models, etc.

[0191] The model training method provided in this application can be executed by a model training device. This application uses an example of a model training device executing the model training method to illustrate the model training device provided in this application.

[0192] As shown in Figure 15, this application embodiment also provides a model training device 1500 for training the text-based image model as described above. The device includes:

[0193] The first acquisition module 1501 is used to acquire at least one training sample, each training sample including image description information, image sample and painting style sample, the training sample being obtained based on user preference information;

[0194] The training module 1502 is used to train the initial model using the at least one training sample to obtain the text-to-image model; the text-to-image model is used to generate M images corresponding to the user preference information based on the image description information, where M is an integer greater than 1.

[0195] Optionally, the first acquisition module 1501 is specifically used for:

[0196] Obtain image description information input by the user;

[0197] Input image description information into the Wensheng image model and output M images, each image corresponding to a painting style, and different images corresponding to different painting styles;

[0198] The image selected by the user from the M images is used as an image sample, and the painting style corresponding to the image selected by the user is used as a painting style sample;

[0199] The training samples are obtained based on the image description information, image samples, and painting style samples.

[0200] Optionally, the training module 1502 is specifically used for:

[0201] Based on the image description information, painting style samples, and first loss function in the training samples, at least one network parameter of the user preference text encoder network in the text-generated image model is updated.

[0202] Based on the image description information in the training samples and the second loss function, update at least one network parameter of the first diffusion model in the text image model;

[0203] Based on the image description information, painting style samples, image samples, and third loss function in the training samples, update at least one network parameter of the second diffusion model in the text image model;

[0204] Based on the trained user preference text encoder network, the first diffusion model, and the second diffusion model, the text-generated graph model is obtained.

[0205] Optionally, the training module 1502 is specifically used for:

[0206] Based on the image description information and the painting style samples, the probability of the user selecting each of the N painting styles is determined, wherein the selection probability of the first painting style is 1, the selection probability of the second painting style is 0, the first painting style is the painting style corresponding to the painting style sample, and the second painting style is the painting style other than the first painting style among the N painting styles.

[0207] Based on the selection probability, the gating weights corresponding to the N self-attention modules in the user preference text encoder network, and the first loss function, update at least one network parameter of the user preference text encoder network in the text graph model.

[0208] Optionally, the training module 1502 is specifically used for:

[0209] The image description information is input into a common text encoder network, which outputs a common text feature vector.

[0210] The image corresponding to the image description information is input into the image encoder of the text image model, and the first image feature vector is output.

[0211] The first image feature vector with added random noise and the public text feature vector are input into the first diffusion model, and the third predicted noise is output.

[0212] Based on the third predicted noise, the first random noise, and the second loss function, update at least one network parameter of the first diffusion model in the Wensheng graph model.

[0213] Optionally, the training module 1502 is specifically used for:

[0214] The image description information is input into a common text encoder network, which outputs a common text feature vector.

[0215] The public text feature vector and the painting style sample are input into the user preference text encoder network, and the user preference text feature vector is output.

[0216] The image sample is input into the image encoder in the text image model, and the second image feature vector is output.

[0217] The second image feature vector with added random noise and the common text feature vector are input into the first diffusion model, and the fourth predicted noise is output.

[0218] Based on the fourth predicted noise, the second image feature vector with added second random noise is denoised to obtain the third image feature vector;

[0219] The third image feature vector with added third random noise and the user preference text feature vector are input into the second diffusion model, and the fifth predicted noise is output.

[0220] Based on the second random noise, the fifth predicted noise, and the third loss function, update at least one network parameter of the second diffusion model in the Wensheng graph model.

[0221] In this embodiment, the initial model is trained using training samples obtained based on user preference information, so that the user preference text encoder network in the text-to-image model can learn user preference information. Thus, the text-to-image model obtained based on the training samples can generate image styles corresponding to user preference information, thereby ensuring that the image style output by the text-to-image model can meet the user's preferences.

[0222] The image generation method provided in this application can be executed by an image generation device. This application uses an image generation device executing the image generation method as an example to illustrate the image generation device provided in this application.

[0223] As shown in Figure 16, this application embodiment provides an image generation apparatus 1600, including:

[0224] The second acquisition module 1601 is used to input image description information into the common text encoder network of the text-generated image model to obtain a common text feature vector. The image description information is used to indicate the image content contained in the image generated by the text-generated image model.

[0225] The third acquisition module 1602 is used to input the public text feature vector into the user preference text encoder network of the text-generated image model and output the user preference text feature vector; wherein, the user preference text feature vector is used to indicate the image content contained in the image generated by the text-generated image model and the painting style corresponding to the image content;

[0226] The fourth acquisition module 1603 is used to perform denoising processing on the user preference text feature vector through the diffusion model of the text-generated image model to obtain the denoised user preference text feature vector.

[0227] The fifth acquisition module 1604 is used to decode the denoised user preference text feature vector through the image decoder of the text-generated image model to obtain an image.

[0228] Optionally, the fourth acquisition module 1603 is specifically used for:

[0229] The public text feature vector is input into the first diffusion model, and the second predicted noise is output.

[0230] Based on the second predicted noise, the public text feature vector is denoised to obtain the first text feature vector.

[0231] The first text feature vector with added fourth random noise and the user preference text feature vector are input into the second diffusion model, and the first predicted noise is output.

[0232] Based on the first predicted noise, the user preference text feature vector is denoised to obtain the denoised user preference text feature vector.

[0233] In this embodiment of the application, since the text-based image model is trained based on user preference information, after inputting image description information into the text-based image model, an image style corresponding to the user preference information can be obtained, thereby ensuring that the image style output by the text-based image model can meet the user's preferences.

[0234] The model training device and image generation device in the embodiments of this application can be electronic devices or components within electronic devices, such as integrated circuits or chips. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. The embodiments of this application do not specifically limit the scope of the electronic device.

[0235] The model training device and image generation device in the embodiments of this application can be devices with an operating system. The operating system can be Android, iOS, or other possible operating systems, and this application embodiment does not specifically limit them.

[0236] The model training apparatus provided in this application embodiment can implement the various processes implemented in the method embodiments of Figures 6 to 12. To avoid repetition, these processes will not be described again here.

[0237] The image generation apparatus provided in this application embodiment can implement the various processes implemented in the method embodiment of FIG13. To avoid repetition, it will not be described again here.

[0238] Optionally, as shown in FIG17, this application embodiment also provides an electronic device 1700, including a processor 1701 and a memory 1702. The memory 1702 stores a program or instructions that can run on the processor 1701. When the program or instructions are executed by the processor 1701, they implement the various steps of the above-described model training method or image generation method embodiments and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0239] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0240] Figure 18 is a schematic diagram of the hardware structure of an electronic device that implements an embodiment of this application.

[0241] The electronic device 1800 includes, but is not limited to, components such as: radio frequency unit 1801, network module 1802, audio output unit 1803, input unit 1804, sensor 1805, display unit 1806, user input unit 1807, interface unit 1808, memory 1809, and processor 1810.

[0242] Those skilled in the art will understand that the electronic device 1800 may also include a power supply (such as a battery) for powering various components. The power supply may be logically connected to the processor 1810 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The electronic device structure shown in Figure 18 does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0243] The processor 1810 is used to acquire at least one training sample, each training sample including image description information, image sample and painting style sample, the training sample being obtained based on user preference information; the initial model is trained using the at least one training sample to obtain a text-based image model; the text-based image model is used to generate M images corresponding to the user preference information based on the image description information, where M is an integer greater than 1.

[0244] In this embodiment, the initial model is trained using training samples obtained based on user preference information, so that the user preference text encoder network in the text-to-image model can learn user preference information. Thus, the text-to-image model obtained based on the training samples can generate image styles corresponding to user preference information, thereby ensuring that the image style output by the text-to-image model can meet the user's preferences.

[0245] Optionally, the processor 1810 is also used for:

[0246] Obtain image description information input by the user;

[0247] Input image description information into the Wensheng image model and output M images, each image corresponding to a painting style, and different images corresponding to different painting styles;

[0248] The image selected by the user from the M images is used as an image sample, and the painting style corresponding to the image selected by the user is used as a painting style sample;

[0249] The training samples are obtained based on the image description information, image samples, and painting style samples.

[0250] Optionally, the processor 1810 is also used for:

[0251] Based on the image description information, painting style samples, and first loss function in the training samples, at least one network parameter of the user preference text encoder network in the text-generated image model is updated.

[0252] Based on the image description information in the training samples and the second loss function, update at least one network parameter of the first diffusion model in the text image model;

[0253] Based on the image description information, painting style samples, image samples, and third loss function in the training samples, update at least one network parameter of the second diffusion model in the text image model;

[0254] Based on the trained user preference text encoder network, the first diffusion model, and the second diffusion model, the text-generated graph model is obtained.

[0255] Optionally, the processor 1810 is also used for:

[0256] Based on the image description information and the painting style samples, the probability of the user selecting each of the N painting styles is determined, wherein the selection probability of the first painting style is 1, the selection probability of the second painting style is 0, the first painting style is the painting style corresponding to the painting style sample, and the second painting style is the painting style other than the first painting style among the N painting styles.

[0257] Based on the selection probability, the gating weights corresponding to the N self-attention modules in the user preference text encoder network, and the first loss function, update at least one network parameter of the user preference text encoder network in the text graph model.

[0258] Optionally, the processor 1810 is also used for:

[0259] The image description information is input into a common text encoder network, which outputs a common text feature vector.

[0260] The image corresponding to the image description information is input into the image encoder of the text image model, and the first image feature vector is output.

[0261] The first image feature vector with added random noise and the public text feature vector are input into the first diffusion model, and the third predicted noise is output.

[0262] Based on the third predicted noise, the first random noise, and the second loss function, update at least one network parameter of the first diffusion model in the Wensheng graph model.

[0263] Optionally, the processor 1810 is also used for:

[0264] The image description information is input into a common text encoder network, which outputs a common text feature vector.

[0265] The public text feature vector and the painting style sample are input into the user preference text encoder network, and the user preference text feature vector is output.

[0266] The image sample is input into the image encoder in the text image model, and the second image feature vector is output.

[0267] The second image feature vector with added random noise and the common text feature vector are input into the first diffusion model, and the fourth predicted noise is output.

[0268] Based on the fourth predicted noise, the second image feature vector with added second random noise is denoised to obtain the third image feature vector;

[0269] The third image feature vector with added third random noise and the user preference text feature vector are input into the second diffusion model, and the fifth predicted noise is output.

[0270] Based on the second random noise, the fifth predicted noise, and the third loss function, update at least one network parameter of the second diffusion model in the Wensheng graph model.

[0271] The processor 1810 is used to input image description information into the common text encoder network of the text-generated image model as described above, and obtain a common text feature vector. The image description information is used to indicate the image content contained in the image generated by the text-generated image model.

[0272] The public text feature vector is input into the user preference text encoder network of the text-generated image model, and the user preference text feature vector is output; wherein, the user preference text feature vector is used to indicate the image content contained in the image generated by the text-generated image model and the painting style corresponding to the image content;

[0273] The user preference text feature vector is denoised using the diffusion model of the text graph model to obtain the denoised user preference text feature vector.

[0274] The image decoder of the text-based graph model decodes the denoised user preference text feature vector to obtain the image.

[0275] Optionally, the processor 1810 is also configured to: input the common text feature vector into the first diffusion model and output the second prediction noise;

[0276] Based on the second predicted noise, the public text feature vector is denoised to obtain the first text feature vector.

[0277] The first text feature vector with added fourth random noise and the user preference text feature vector are input into the second diffusion model, and the first predicted noise is output.

[0278] Based on the first predicted noise, the user preference text feature vector is denoised to obtain the denoised user preference text feature vector.

[0279] It should be understood that, in this embodiment, the input unit 1804 may include a graphics processing unit (GPU) 18041 and a microphone 18042. The GPU 18041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1806 may include a display panel 18061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1807 includes at least one of a touch panel 18071 and other input devices 18072. The touch panel 18071 is also called a touch screen. The touch panel 18071 may include a touch detection device and a touch controller. Other input devices 18072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0280] The memory 1809 can be used to store software programs and various data. The memory 1809 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1809 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1809 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0281] Processor 1810 may include one or more processing units; optionally, processor 1810 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 1810.

[0282] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described model training method or image generation method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0283] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0284] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described model training method or image generation method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0285] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0286] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described model training method or image generation method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0287] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0288] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0289] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A text-based image model, comprising: A public text encoder network is used to convert image description information into public text feature vectors, wherein the image description information is used to describe the image content contained in the image generated by the text-generated image model. A user preference text encoder network is used to output a user preference text feature vector based on the public text feature vector. The user preference text feature vector is used to indicate the image content contained in the image generated by the text-to-image model and the painting style corresponding to the image content. The user preference text encoder network is connected to the public text encoder network. A diffusion model is used to perform noise prediction processing on the public text feature vector and the user preference text feature vector to obtain a first predicted noise, which is used to denoise the user preference text feature vector; the diffusion model is connected to the public text encoder network and the user preference text encoder network respectively. The image decoder is used to decode the denoised user preference text feature vector and output an image.

2. The text-based image model according to claim 1, wherein, The user preference text encoder network includes: There are N self-attention modules, which are used to generate text feature vectors based on a common text feature vector and a corresponding painting style; where each self-attention module corresponds to a painting style, and N is an integer greater than 1. A gating network is used to output the gating weights corresponding to each of the self-attention modules, and the gating weights are used to indicate the probability that the user selects the drawing style corresponding to the self-attention module; wherein, the gating network is connected to each of the N self-attention modules respectively; The processing module is used to determine the text feature vector generated by the self-attention module with the largest gating weight as the user preference text feature vector.

3. The text-based image model according to claim 1, wherein, The diffusion model includes: A first diffusion model, connected to the public text encoder network, is used to perform noise prediction processing on the public text feature vector to obtain a second predicted noise, which is used to denoise the public text feature vector. The second diffusion model, the second diffusion module is used to perform noise prediction processing on the user preference text feature vector based on the denoised public text feature vector, to obtain the first predicted noise.

4. A model training method for training a text-based graph model as described in any one of claims 1 to 3, the method comprising: At least one training sample is obtained, each training sample including image description information, image sample and painting style sample, the training sample being obtained based on user preference information; The initial model is trained using the at least one training sample to obtain the text-to-image model; the text-to-image model is used to generate M images corresponding to the user preference information based on the image description information, where M is an integer greater than 1.

5. The method according to claim 4, wherein, The acquisition of at least one training sample includes: Obtain image description information input by the user; Input image description information into the Wensheng image model and output M images, each image corresponding to a painting style, and different images corresponding to different painting styles; The image selected by the user from the M images is used as an image sample, and the painting style corresponding to the image selected by the user is used as a painting style sample; The training samples are obtained based on the image description information, image samples, and painting style samples.

6. The method according to claim 4, wherein, The step of training the initial model using the at least one training sample to obtain the text-based image model includes: Based on the image description information, painting style samples, and first loss function in the training samples, at least one network parameter of the user preference text encoder network in the text-generated image model is updated. Based on the image description information in the training samples and the second loss function, update at least one network parameter of the first diffusion model in the text image model; Based on the image description information, painting style samples, image samples, and third loss function in the training samples, update at least one network parameter of the second diffusion model in the text image model; Based on the trained user preference text encoder network, the first diffusion model, and the second diffusion model, the text-generated graph model is obtained.

7. The method according to claim 6, wherein, Based on the image description information, painting style samples, and the first loss function in the training samples, at least one network parameter of the user preference text encoder network in the text-based image model is updated, including: Based on the image description information and the painting style samples, the selection probability of the user choosing each of the N painting styles is determined, wherein the selection probability of the first painting style is 1, the selection probability of the second painting style is 0, the first painting style is the painting style corresponding to the painting style sample, and the second painting style is the painting style other than the first painting style among the N painting styles. Based on the selection probability, the gating weights corresponding to the N self-attention modules in the user preference text encoder network, and the first loss function, update at least one network parameter of the user preference text encoder network in the text graph model.

8. The method according to claim 6, wherein, The step of updating at least one network parameter of the first diffusion model in the text image model based on image description information in the training samples and the second loss function includes: The image description information is input into a common text encoder network, which outputs a common text feature vector. The image corresponding to the image description information is input into the image encoder of the text image model, and the first image feature vector is output. The first image feature vector with added random noise and the public text feature vector are input into the first diffusion model, and the third predicted noise is output. Based on the third predicted noise, the first random noise, and the second loss function, update at least one network parameter of the first diffusion model in the Wensheng graph model.

9. The method according to claim 6, wherein, The step of updating at least one network parameter of the second diffusion model in the text-based image model based on image description information, painting style samples, image samples, and a third loss function from the training samples includes: The image description information is input into a common text encoder network, which outputs a common text feature vector. The public text feature vector and the painting style sample are input into the user preference text encoder network, and the user preference text feature vector is output. The image sample is input into the image encoder in the text image model, and the second image feature vector is output. The second image feature vector with added random noise and the common text feature vector are input into the first diffusion model, and the fourth predicted noise is output. Based on the fourth predicted noise, the second image feature vector with added second random noise is denoised to obtain the third image feature vector; The third image feature vector with added third random noise and the user preference text feature vector are input into the second diffusion model, and the fifth predicted noise is output. Based on the second random noise, the fifth predicted noise, and the third loss function, update at least one network parameter of the second diffusion model in the Wensheng graph model.

10. An image generation method, comprising: The image description information is input into the common text encoder network of the text-generated image model as described in any one of claims 1 to 3 to obtain a common text feature vector, wherein the image description information is used to indicate the image content contained in the image generated by the text-generated image model. The public text feature vector is input into the user preference text encoder network of the text-generated image model, and the user preference text feature vector is output; wherein, the user preference text feature vector is used to indicate the image content contained in the image generated by the text-generated image model and the painting style corresponding to the image content; The user preference text feature vector is denoised using the diffusion model of the text graph model to obtain the denoised user preference text feature vector. The image decoder of the text-based graph model decodes the denoised user preference text feature vector to obtain the image.

11. The method according to claim 10, wherein, The user preference text feature vector is denoised using the diffusion model of the text graph model to obtain a denoised user preference text feature vector, including: The public text feature vector is input into the first diffusion model, and the second predicted noise is output. Based on the second predicted noise, the public text feature vector is denoised to obtain the first text feature vector. The first text feature vector with added fourth random noise and the user preference text feature vector are input into the second diffusion model, and the first predicted noise is output. Based on the first predicted noise, the user preference text feature vector is denoised to obtain the denoised user preference text feature vector.

12. A model training apparatus for training a text-based graph model as described in any one of claims 1 to 3, the apparatus comprising: The first acquisition module is used to acquire at least one training sample, each training sample including image description information, image sample and painting style sample, the training sample being obtained based on user preference information; The training module is used to train the initial model using the at least one training sample to obtain the text-to-image model; the text-to-image model is used to generate M images corresponding to the user preference information based on the image description information, where M is an integer greater than 1.

13. The apparatus according to claim 12, wherein, The first acquisition module is specifically used for: Obtain image description information input by the user; Input image description information into the Wensheng image model and output M images, each image corresponding to a painting style, and different images corresponding to different painting styles; The image selected by the user from the M images is used as an image sample, and the painting style corresponding to the image selected by the user is used as a painting style sample; The training samples are obtained based on the image description information, image samples, and painting style samples.

14. The apparatus according to claim 12, wherein, The training module is specifically used for: Based on the image description information, painting style samples, and first loss function in the training samples, at least one network parameter of the user preference text encoder network in the text-generated image model is updated. Based on the image description information in the training samples and the second loss function, update at least one network parameter of the first diffusion model in the text image model; Based on the image description information, painting style samples, image samples, and third loss function in the training samples, update at least one network parameter of the second diffusion model in the text image model; Based on the trained user preference text encoder network, the first diffusion model, and the second diffusion model, the text-generated graph model is obtained.

15. The apparatus according to claim 14, wherein, The training module is specifically used for: Based on the image description information and the painting style samples, the selection probability of the user choosing each of the N painting styles is determined, wherein the selection probability of the first painting style is 1, the selection probability of the second painting style is 0, the first painting style is the painting style corresponding to the painting style sample, and the second painting style is the painting style other than the first painting style among the N painting styles. Based on the selection probability, the gating weights corresponding to the N self-attention modules in the user preference text encoder network, and the first loss function, update at least one network parameter of the user preference text encoder network in the text graph model.

16. The apparatus according to claim 14, wherein, The training module is specifically used for: The image description information is input into a common text encoder network, which outputs a common text feature vector. The image corresponding to the image description information is input into the image encoder of the text image model, and the first image feature vector is output. The first image feature vector with added random noise and the public text feature vector are input into the first diffusion model, and the third predicted noise is output. Based on the third predicted noise, the first random noise, and the second loss function, update at least one network parameter of the first diffusion model in the Wensheng graph model.

17. The apparatus according to claim 14, wherein, The training module is specifically used for: The image description information is input into a common text encoder network, which outputs a common text feature vector. The public text feature vector and the painting style sample are input into the user preference text encoder network, and the user preference text feature vector is output. The image sample is input into the image encoder in the text image model, and the second image feature vector is output. The second image feature vector with added random noise and the common text feature vector are input into the first diffusion model, and the fourth predicted noise is output. Based on the fourth predicted noise, the second image feature vector with added second random noise is denoised to obtain the third image feature vector; The third image feature vector with added third random noise and the user preference text feature vector are input into the second diffusion model, and the fifth predicted noise is output. Based on the second random noise, the fifth predicted noise, and the third loss function, update at least one network parameter of the second diffusion model in the Wensheng graph model.

18. An image generation apparatus, comprising: The second acquisition module is used to input image description information into the public text encoder network of the text-generated image model as described in any one of claims 1 to 3, and obtain a public text feature vector, wherein the image description information is used to indicate the image content contained in the image generated by the text-generated image model. The third acquisition module is used to input the public text feature vector into the user preference text encoder network of the text-generated image model and output the user preference text feature vector; wherein, the user preference text feature vector is used to indicate the image content contained in the image generated by the text-generated image model and the painting style corresponding to the image content; The fourth acquisition module is used to perform denoising processing on the user preference text feature vector through the diffusion model of the text-generated image model to obtain the denoised user preference text feature vector. The fifth acquisition module is used to decode the denoised user preference text feature vector through the image decoder of the text-to-image model to obtain an image.

19. The apparatus according to claim 18, wherein, The fourth acquisition module is specifically used for: The public text feature vector is input into the first diffusion model, and the second predicted noise is output. Based on the second predicted noise, the public text feature vector is denoised to obtain the first text feature vector. The first text feature vector with added fourth random noise and the user preference text feature vector are input into the second diffusion model, and the first predicted noise is output. Based on the first predicted noise, the user preference text feature vector is denoised to obtain the denoised user preference text feature vector.

20. An electronic device comprising a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the model training method as claimed in any one of claims 1 to 9.

21. An electronic device comprising a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the image generation method as claimed in any one of claims 10 to 11.

Images