Image generation method and device, electronic equipment and storage medium

Abstract

The invention provides an image generation method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a prompt text; based on a first text encoder, obtaining a first text encoding feature corresponding to the prompt text; based on a second text encoder, a second text encoding feature corresponding to the prompt text is obtained, and the first text encoder is different from the second text encoder; based on the first text coding feature and the second text coding feature, obtaining a joint text coding feature; and obtaining an output image corresponding to the prompt text through an image generation module based on preset noise and the joint text coding features. In this way, by combining the first text coding feature and the second text coding feature, a generated image with a high degree of alignment with the prompt text can be obtained in subsequent steps.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of multimodal generation technology, and in particular to an image generation method, device, electronic device, and storage medium. Background Art

[0002] Multimodal generation can accept a variety of different input methods (such as language, image, voice, video), and in some cases produce different output modes, such as input stimulus (Prompt) is a text prompt, the output text prompt guides the generated image. By adjusting the prompt, the multimodal generation model can better understand the intention of the text prompt and generate better images.

[0003] However, in the prior art, even if a sufficiently good prompt is provided, it is still difficult to improve the alignment between the text representation and the visual representation of the input prompt, and further it is difficult to generate a target image that meets the requirements.

[0004] Therefore, how to improve the alignment between the visual representation and text representation of the generated image is a technical problem that technicians in this field urgently need to solve. Summary of the invention

[0005] Embodiments of the present disclosure provide an image generating method, an apparatus, an electronic device, and a storage medium.

[0006] In a first aspect, an embodiment of the present disclosure provides an image generation method, the method comprising:

[0007] Get the prompt text;

[0008] Based on the first text encoder, obtain a first text encoding feature corresponding to the prompt text;

[0009] Based on a second text encoder, obtaining a second text encoding feature corresponding to the prompt text, wherein the first text encoder and the second text encoder are different;

[0010] Based on the first text encoding feature and the second text encoding feature, a joint text encoding feature is obtained;

[0011] Based on the preset noise and the above-mentioned joint text encoding features, an output image corresponding to the above-mentioned prompt text is obtained through an image generation module.

[0012] Optionally, the image generation module includes an encoder, a deep neural network and a decoder; and

[0013] The output image corresponding to the prompt text is obtained through the image generation module based on the preset noise and the combined text encoding feature, including:

[0014] Based on the above preset noise, a noise image coding feature is obtained through the above encoder;

[0015] Based on the above noise image coding features and the above joint text coding features, the image and text fusion coding features are obtained through the above deep neural network;

[0016] Based on the above image and text fusion coding features, the output image corresponding to the above prompt text is obtained through the above decoder.

[0017] Optionally, the second text encoder is pre-trained through the following training steps:

[0018] Determining the model structure and model parameter information of the initial second text encoder;

[0019] Respectively obtaining model structures and model parameter information of a pre-trained first text encoder, an image generation module, and a quality evaluation model, wherein the quality evaluation model is used to evaluate the image generation quality of a target image obtained by performing image generation based on a target text;

[0020] For a training sample in the training sample set, the following second text encoder parameter adjustment operation is performed until a preset training end condition is satisfied, wherein the training sample includes a training text and a corresponding matching training image:

[0021] Based on the training text in the training sample, obtaining a first training text encoding feature through the first text encoder;

[0022] Based on the training text in the training sample, obtaining a second training text encoding feature through the initial second text encoder;

[0023] Based on the first training text encoding feature and the second training text encoding feature, a joint training text encoding feature is obtained;

[0024] Based on the training image in the training sample and the above-mentioned joint training text encoding features, the training output image is obtained through the above-mentioned image generation module;

[0025] Based on the training text in the training sample and the training output image, the training output image generation quality is obtained through the image generation quality evaluation model;

[0026] The parameters of the initial second text encoder are adjusted with the maximization of the quality of the training output image generation as the optimization goal.

[0027] Optionally, the image generation module includes an encoder, a deep neural network and a decoder; and

[0028] The training output image is obtained by the image generation module based on the training image in the training sample and the joint training text encoding feature, including:

[0029] Based on the training images in the training samples, the encoding features of the training images are obtained by the encoder.

[0030] Based on the above training image coding features and the above joint training text coding features, the above deep neural network is used to obtain the training image and text fusion coding features;

[0031] Based on the above training image and text fusion coding features, the training output image is obtained through the above decoder.

[0032] Optionally, the training output image generation quality is obtained based on the training text in the training sample and the training output image by using the image generation quality evaluation model, including:

[0033] Based on the training text in the training sample and the training output image, a quality evaluation model is generated through the image to obtain an aesthetic score, a relevance score, and a clarity score;

[0034] Based on the above aesthetic score, relevance score and clarity score, the training output image generation quality is obtained.

[0035] Optionally, the image generation quality evaluation model includes an aesthetic scoring model, a picture-text relevance scoring model, and an image clarity scoring model; and

[0036] Based on the training text in the training sample and the training output image, the quality evaluation model is generated through the image to obtain the aesthetic score, relevance score and clarity score, including:

[0037] Based on the above training output image, an aesthetic score is obtained through the above aesthetic scoring model, where the above aesthetic scoring model is an aesthetic predictor model obtained by training based on the aesthetic scores of the image and the corresponding annotated image;

[0038] Based on the training text in the training sample and the training output image, a relevance score is obtained through the image-text relevance scoring model;

[0039] Based on the above training output image, a clarity score is obtained through the above image clarity scoring model;

[0040] The above-mentioned output image based on the training is used to obtain a clarity score through an image clarity scoring model, including:

[0041] Using a preset algorithm, respectively calculate the horizontal gradient and vertical gradient of the training output image in the horizontal direction and the vertical direction;

[0042] Searching for image clarity scores corresponding to both the horizontal gradient and the vertical gradient in a preset gradient clarity correspondence table, wherein the gradient clarity correspondence table is used to characterize the correspondence between the first direction gradient and the second direction gradient and the image clarity score;

[0043] The found image clarity score is determined as the above clarity score.

[0044] Optionally, the above image generation module is pre-trained through the following training steps:

[0045] Determine the model structure and model parameter information of the initial image generation module;

[0046] Respectively obtain the model structure and model parameter information of the pre-trained first text encoder and the second text encoder;

[0047] For a training sample in the training sample set, the following initial image generation module parameter adjustment operation is performed, and the training sample includes a training text and a corresponding matching training image:

[0048] Based on the training text in the training sample, obtaining a first training text encoding feature through the first text encoder;

[0049] Based on the training text in the training sample, obtaining a second training text encoding feature through the second text encoder;

[0050] Based on the first training text encoding feature and the second training text encoding feature, a joint training text encoding feature is obtained;

[0051] Based on the training image in the training sample and the joint training text encoding features, a training output image is obtained through the initial image generation module;

[0052] Based on the training images in the training sample and the training output images, a loss function is obtained;

[0053] Based on the above loss function, the parameters of the above initial image generation module are adjusted.

[0054] In a second aspect, an embodiment of the present disclosure provides an image generating device, the device comprising:

[0055] The acquisition module is used to obtain the prompt text;

[0056] A first encoding module, used for obtaining a first text encoding feature corresponding to the prompt text based on a first text encoder;

[0057] A second encoding module, used for obtaining a second text encoding feature corresponding to the prompt text based on a second text encoder, wherein the first text encoder and the second text encoder are different;

[0058] A combining module, configured to obtain a combined text encoding feature based on the first text encoding feature and the second text encoding feature;

[0059] A generation module is used to obtain an output image corresponding to the above-mentioned prompt text through an image generation module based on preset noise and the above-mentioned joint text encoding features.

[0060] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0061] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method described in any implementation manner in the first aspect.

[0062] In order to improve the alignment between the visual representation of the generated image and the text representation, the image generation method, device, electronic device and storage medium provided by the embodiments of the present disclosure obtain prompt text; based on the first text encoder, obtain the first text encoding feature corresponding to the prompt text; based on the second text encoder, obtain the second text encoding feature corresponding to the prompt text, and the first text encoder and the second text encoder are different; based on the first text encoding feature and the second text encoding feature, obtain the joint text encoding feature; based on the preset noise and the joint text encoding feature, obtain the output image corresponding to the prompt text through the image generation module. In this way, by combining the first text encoding feature and the second text encoding feature, the subsequent steps can obtain a generated image with a high degree of alignment with the prompt text. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are only for the purpose of illustrating specific embodiments and are not to be considered as limiting the present disclosure. In the drawings:

[0064] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0065] Figure 2is a flow chart of an embodiment of an image generation method according to the present disclosure;

[0066] Figure 3 A training flow chart of a second text encoder of an embodiment of the image generation method disclosed herein;

[0067] Figure 4 A training flowchart of a second text encoder of another embodiment of the image generation method disclosed herein;

[0068] Figure 5 A training flow chart of an image generation module of an embodiment of the image generation method disclosed herein;

[0069] Figure 6 is a structural schematic diagram of an embodiment of an image generating device according to the present disclosure;

[0070] Figure 7 It is a structural diagram of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0071] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0072] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0073] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the image generation method, apparatus, electronic device, and storage medium of the present disclosure can be applied.

[0074] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0075] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as image generation applications, voice recognition applications, short video social applications, audio and video conferencing applications, video live broadcast applications, document editing applications, input method applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0076] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) players, laptop computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the terminal devices listed above. It can be implemented as multiple software or software modules (for example, to provide image generation services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0077] In some cases, the image generation method provided by the present disclosure may be executed by the terminal devices 101, 102, 103, and accordingly, the image generation apparatus may be disposed in the terminal devices 101, 102, 103. In this case, the system architecture 100 may not include the server 105.

[0078] In some cases, the image generation method provided by the present disclosure can be jointly performed by the terminal devices 101, 102, 103 and the server 105. For example, the step of "obtaining the prompt text" can be performed by the terminal devices 101, 102, 103, and the steps of "obtaining an output image corresponding to the prompt text through the image generation module based on the preset noise and the above-mentioned joint text encoding features" can be performed by the server 105. The present disclosure does not limit this. Accordingly, the image generation device can also be respectively set in the terminal devices 101, 102, 103 and the server 105.

[0079] In some cases, the image generation method provided by the present disclosure may be executed by the server 105 , and accordingly, the image generation device may also be set in the server 105 . In this case, the system architecture 100 may not include the terminal devices 101 , 102 , and 103 .

[0080] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0081] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0082] Continue to refer Figure 2 , which shows a process 200 of an embodiment of an image generation method according to the present disclosure, the image generation method comprises the following steps:

[0083] Step 201, obtaining prompt text.

[0084] Hint text is used to guide the generation of images, such as "a house", "a star", "a game controller" and other texts that describe the content of the image.

[0085] Step 202: Based on the first text encoder, obtain a first text encoding feature corresponding to the prompt text.

[0086] Step 203: Based on the second text encoder, obtain the second text encoding feature corresponding to the prompt text.

[0087] The prompt text is input into the first text encoder and the second text encoder respectively, and the encoders will obtain the encoded representation of the text. The first text encoder and the second text encoder are different in structure or parameters, but the output dimensions of the first text encoding feature and the second text encoding feature obtained by them are the same, for example, both can be 768-dimensional vectors.

[0088] The first text encoder may be a text encoder in a picture-text matching model obtained by training a large number of sample picture-text pairs, for example, a text encoder in a CLIP (Contrastive Language-Image Pre-Training) model. It should be noted that at this stage, the trained first text encoder may be used directly without updating, or the first text encoder may be continuously updated in combination with the generated results.

[0089] The second text encoder may be a model (such as a Transformer network model) trained with the quality evaluation model and having text encoding capabilities, that is, a text encoder that uses the quality evaluation model for supervision and continuously improves its own performance.

[0090] If there is no second text encoder and only the first text encoder is used, a large amount of data is needed to train the first text encoder, and the training process is complicated and the effect is relatively poor. In this case, the second text encoder is added, and only the second text encoder is trained, and then combined with the result of the first text encoder. The obtained joint text encoding features can improve the alignment of features, thereby improving the generalization ability and accuracy of the overall model. The overall model only needs less data to greatly improve the quality of the output image of the overall model.

[0091] Step 204: obtaining a combined text encoding feature based on the first text encoding feature and the second text encoding feature.

[0092] Based on the first text encoding feature and the second text encoding feature, more patterns and information can be captured by using multiple feature vectors to improve the accuracy and generalization ability of the image generation module. By generating an image using the combined text encoding feature obtained by combining the first text encoding feature and the second text encoding feature, the performance of the image generation module can be fully improved, and the alignment between the prompt text and the generated image can be increased.

[0093] Step 205, based on the preset noise and the above-mentioned combined text encoding features, an output image corresponding to the above-mentioned prompt text is obtained through an image generation module.

[0094] The preset noise image can be a pure noise image or a noise image that is related to the prompt text. The image generation module is a trained model that can achieve the effect of generating a target image based on a vector and a preset image. Specifically, the image generation module can first convert the preset noise and the joint text encoding features into Gaussian noise, and then restore the Gaussian noise to obtain an output image corresponding to the prompt text. The implementation process is similar to the diffusion model. For example, the output image corresponds to the prompt text such as "a house", "a star", "a game handle", etc.

[0095] It should be noted that, similar to the diffusion model, the first text encoder, the second text encoder and the image generation module can be integrated together to form an image generation network. In this way, by inputting the prompt text into the image generation network, an output image corresponding to the prompt text can be directly obtained.

[0096] The following describes an implementation of an image generation module, where the image generation module includes an encoder, a deep neural network, and a decoder. Thus, step 205 may include the following sub-steps:

[0097] Sub-step 2051, based on the above preset noise, obtain the noise image coding feature through the above encoder.

[0098] The encoder in this step can be a vae-encoder encoder or other encoder, which is used to encode the preset noise into the noise image coding feature. The encoder directly generates a code, but in vae, in order to add appropriate noise to the code, the encoder will output two codes, one is the original code, and the other is the code that controls the degree of noise interference.

[0099] Sub-step 2052, based on the above-mentioned noise image coding features and the above-mentioned joint text coding features, obtain the image and text fusion coding features through the above-mentioned deep neural network.

[0100] The deep neural network in this step can be a unet neural network or other neural networks, which is used to extract features and effectively solve the detail loss caused by downsampling. By splicing features of various dimensions, more complete and sufficient features can be obtained in subsequent steps.

[0101] Sub-step 2053, based on the above image and text fusion coding features, through the above decoder, an output image corresponding to the above prompt text is obtained.

[0102] The encoder of this step can be a vae-decoder encoder or other encoders, which can obtain the final output image corresponding to the prompt text by decoding and restoring the encoded features of the above processed image and text fusion to an image.

[0103] The image generation method provided by the above-mentioned embodiment of the present disclosure combines the prompt text encoded by the second text encoder with the prompt text encoded by the first encoder, so that the generation effect of the input image generation module is better and the generated image is more aligned with the prompt text.

[0104] Continue to see Figure 3 , Figure 3 This is a training flow chart of a second text encoder of an embodiment of the image generation method disclosed herein. The training process of the second text encoder is exemplarily illustrated by the following steps:

[0105] Step 301: Determine the model structure and model parameter information of the initial second text encoder.

[0106] Step 302: Obtain model structures and model parameter information of the pre-trained first text encoder, image generation module, and quality assessment model respectively.

[0107] The quality evaluation model is used to evaluate the image generation quality of the target image obtained by image generation based on the target text. In order to increase the accuracy and comprehensiveness of the quality rating model evaluation, multiple evaluation rules can be set to evaluate the quality of the image, and then the multiple evaluation results can be combined to obtain the final evaluation.

[0108] Step 303: For the training samples in the training sample set, perform the following second text encoder parameter adjustment operation until a preset training end condition is met.

[0109] The training samples include training texts and corresponding matching training images.

[0110] The second text encoder parameter adjustment operation may include the following steps 3031 to 3035:

[0111] Step 3031: Based on the training text in the training sample, obtain the first training text encoding feature through the above-mentioned first text encoder.

[0112] Step 3032: Based on the training text in the training sample, obtain the second training text encoding features through the above-mentioned initial second text encoder.

[0113] Step 3033: Based on the first training text encoding feature and the second training text encoding feature, a joint training text encoding feature is obtained.

[0114] Step 3033 may include the following sub-steps a to c:

[0115] Sub-step a: Based on the training images in the training samples, the encoding features of the training images are obtained through the encoder.

[0116] Sub-step b: Based on the above-mentioned training image coding features and the above-mentioned joint training text coding features, the training image and text fusion coding features are obtained through the above-mentioned deep neural network.

[0117] Sub-step c: Based on the above training image and text fusion encoding features, the training output image is obtained through the above decoder.

[0118] The above sub-steps ac are similar to sub-steps 2051-2053, except that the prompt text is replaced by the training text in the training sample, and the joint text encoding feature is replaced by the joint training text encoding feature, which will not be repeated here.

[0119] Step 3034: Based on the training image in the training sample and the above-mentioned joint training text encoding features, the training output image is obtained through the above-mentioned image generation module.

[0120] The above steps 3031-3034 are similar to steps 202-205, except that the prompt text is replaced by the training text in the training sample, and the preset noise is replaced by the training image in the training sample. Since the training image is an image that has a certain correlation with the training text, replacing the preset noise with it before performing subsequent training steps can improve the training efficiency and reduce the training time.

[0121] Step 3035: Based on the training text in the training sample and the training output image, the training output image generation quality is obtained through the image generation quality evaluation model.

[0122] The image generation quality evaluation model can obtain an aesthetic score, a relevance score, and a clarity score based on the training text in the training sample and the training output image. Specifically, the image generation quality evaluation model can include an aesthetic scoring model, a text-image relevance scoring model, and an image clarity scoring model, which respectively correspond to the aesthetics of the generated image, the relevance to the training text, and the clarity. The quality of the generated image is judged comprehensively from multiple dimensions, and the quality of the next generated image is improved by adjusting the parameters of the model.

[0123] Specifically, the above step 3035 includes the following sub-steps A to D:

[0124] Sub-step A: Based on the above training output image, an aesthetic score is obtained through an aesthetic scoring model.

[0125] The aesthetic scoring model is an aesthetic predictor model trained based on the aesthetic scores of images and corresponding annotated images, for example, an aesthetic predictor trained by human ratings in the Aesthetic Visual Analysis dataset to quantify aesthetic preferences.

[0126] Sub-step B: Based on the training text in the training sample and the training output image, a relevance score is obtained through a picture-text relevance scoring model.

[0127] The image-text relevance scoring model is a trained language-image pre-trained CLIP model used to evaluate the comparative relevance between text and images.

[0128] Sub-step C: Based on the above training output image, a clarity score is obtained through an image clarity scoring model.

[0129] The execution logic of the clarity scoring model can be a preset algorithm, such as the Tenengrad gradient method, that is, the higher the gradient value in the same scene, the clearer the image. Specifically, the Sober operator is first used to calculate the horizontal gradient and vertical gradient of the above training output image in the horizontal direction and vertical direction respectively; then, in the preset gradient clarity correspondence table, the image clarity score corresponding to the above horizontal gradient and the above vertical gradient is searched, wherein the above gradient clarity correspondence table is used to characterize the correspondence between the first direction gradient and the second direction gradient and the image clarity score; finally, the image clarity score found is determined as the above clarity score.

[0130] Step D: Based on the above aesthetic score, relevance score and clarity score, the training output image generation quality is obtained.

[0131] Specifically, the aesthetic score, relevance score and clarity score may be added together to obtain a comprehensive score for evaluating the training output image generation quality.

[0132] Step 3036: Taking maximizing the quality of the training output image generation as the optimization goal, adjust the parameters of the second text encoder.

[0133] See also Figure 4 By executing the above steps, we can get that in order to improve the quality of image generation, the parameters of the second text encoder can be repeatedly adjusted to make the generation performance of the entire model better and better. This training process only adjusts the parameters of the second text encoder, so the learnable layer is the parameters of the second text encoder and its output. The other models are frozen layers, which only forward and do not back-propagate gradients, and the model parameters will not change.

[0134] The above process can be expressed in a formulaic form as follows:

[0135] L=maxE (pθ) R(I,T);

[0136] R(I,T)=[F B (I)]+[F T (I)]+[F G (I,T)];

[0137] Where pθ represents the parameters of the second text encoder, I represents the training output image, T represents the training text, and F B (I) is the aesthetic score, F T (I) is the clarity score, F G (I) is the correlation score.

[0138] The training process of the second text encoder disclosed in the present invention can improve the quality of the output image by using smaller-scale data by only changing the parameters of the second text encoder, with high efficiency and low resource consumption.

[0139] Continue to see Figure 5 , Figure 5 This is a training flow chart of an image generation module of an embodiment of the image generation method of the present disclosure. The training process of the image generation module is exemplarily described by the following steps:

[0140] Step 501: determine the model structure and model parameter information of the initial image generation module.

[0141] Step 502, respectively obtain the model structure and model parameter information of the pre-trained first text encoder and the second text encoder.

[0142] Step 503: For the training samples in the training sample set, perform the following initial image generation module parameter adjustment operation.

[0143] The training samples include training text and corresponding matching training images:

[0144] The initial image generation module parameter adjustment operation may include the following steps 5031 to 5035:

[0145] Step 5031: Based on the training text in the training sample, obtain the first training text encoding feature through the above-mentioned first text encoder.

[0146] Step 5032: Based on the training text in the training sample, obtain the second training text encoding feature through the second text encoder.

[0147] Step 5033: Based on the first training text encoding feature and the second training text encoding feature, a joint training text encoding feature is obtained.

[0148] Step 5034: Based on the training image in the training sample and the joint training text encoding features, the training output image is obtained through the initial image generation module.

[0149] Step 5035: Obtain a loss function based on the training images in the training sample and the training output images.

[0150] The loss function calculates the difference between the training image in the training sample and the training output image, which is the loss value. After obtaining the loss value, the model updates each parameter through back propagation to reduce the loss between the training image in the training sample and the training output image, so that the output image generated by the model is closer to the training image in the training sample, thereby achieving the purpose of learning.

[0151] Step 5036: Based on the loss function, adjust the parameters of the initial image generation module.

[0152] By performing the above training on the image generation model, the image generation model can be continuously learned, thereby improving the generation effect of the image generation model.

[0153] The present disclosure provides an embodiment of an image generating device. The device embodiment corresponds to the above method embodiment, and the device can be specifically applied to various electronic devices.

[0154] like Figure 6 As shown, the image generating device 600 of this embodiment includes:

[0155] The acquisition module 601 is used to acquire the prompt text;

[0156] A first encoding module 602, configured to obtain a first text encoding feature corresponding to the prompt text based on a first text encoder;

[0157] A second encoding module 603, configured to obtain a second text encoding feature corresponding to the prompt text based on a second text encoder, wherein the first text encoder and the second text encoder are different;

[0158] A combining module 604 is used to obtain a combined text encoding feature based on the first text encoding feature and the second text encoding feature;

[0159] The generation module 605 is used to obtain an output image corresponding to the above-mentioned prompt text through the image generation module based on the preset noise and the above-mentioned joint text encoding feature.

[0160] In some optional implementations, the image generation module includes an encoder, a deep neural network, and a decoder; and

[0161] The generation module 605 includes:

[0162] A coding unit is generated, which is used to obtain a noise image coding feature through the encoder based on the preset noise;

[0163] Generate a fusion unit, which is used to obtain the image and text fusion coding features through the deep neural network based on the noise image coding features and the joint text coding features;

[0164] A decoding unit is generated, which is used to obtain an output image corresponding to the prompt text through the decoder based on the image and text fusion coding features.

[0165] In some optional implementations, the second text encoder is pre-trained through the following training steps:

[0166] Determining the model structure and model parameter information of the initial second text encoder;

[0167] Respectively obtaining model structures and model parameter information of a pre-trained first text encoder, an image generation module, and a quality evaluation model, wherein the quality evaluation model is used to evaluate the image generation quality of a target image obtained by performing image generation based on a target text;

[0168] For a training sample in the training sample set, the following second text encoder parameter adjustment operation is performed until a preset training end condition is satisfied, wherein the training sample includes a training text and a corresponding matching training image:

[0169] Based on the training text in the training sample, obtaining a first training text encoding feature through the first text encoder;

[0170] Based on the training text in the training sample, obtaining a second training text encoding feature through the initial second text encoder;

[0171] Based on the first training text encoding feature and the second training text encoding feature, a joint training text encoding feature is obtained;

[0172] Based on the training image in the training sample and the above-mentioned joint training text encoding features, the training output image is obtained through the above-mentioned image generation module;

[0173] Based on the training text in the training sample and the training output image, the training output image generation quality is obtained through the image generation quality evaluation model;

[0174] The parameters of the initial second text encoder are adjusted with the maximization of the quality of the training output image generation as the optimization goal.

[0175] In some optional implementations, the image generation module includes an encoder, a deep neural network, and a decoder; and

[0176] The training output image is obtained by the image generation module based on the training image in the training sample and the joint training text encoding feature, including:

[0177] Based on the training images in the training samples, the encoding features of the training images are obtained by the encoder.

[0178] Based on the above training image coding features and the above joint training text coding features, the above deep neural network is used to obtain the training image and text fusion coding features;

[0179] Based on the above training image and text fusion coding features, the training output image is obtained through the above decoder.

[0180] In some optional implementations, the training output image generation quality is obtained based on the training text in the training sample and the training output image by using the image generation quality evaluation model, including:

[0181] Based on the training text in the training sample and the training output image, a quality evaluation model is generated through the image to obtain an aesthetic score, a relevance score, and a clarity score;

[0182] Based on the above aesthetic score, relevance score and clarity score, the training output image generation quality is obtained.

[0183] In some optional implementations, the image generation quality evaluation model includes an aesthetic scoring model, a picture-text relevance scoring model, and an image clarity scoring model; and

[0184] Based on the training text in the training sample and the training output image, the quality evaluation model is generated through the image to obtain the aesthetic score, relevance score and clarity score, including:

[0185] Based on the above training output image, an aesthetic score is obtained through the above aesthetic scoring model, where the above aesthetic scoring model is an aesthetic predictor model obtained by training based on the aesthetic scores of the image and the corresponding annotated image;

[0186] Based on the training text in the training sample and the training output image, a relevance score is obtained through the image-text relevance scoring model;

[0187] Based on the above training output image, a clarity score is obtained through the above image clarity scoring model;

[0188] The above-mentioned output image based on the training is used to obtain a clarity score through an image clarity scoring model, including:

[0189] Using a preset algorithm, respectively calculate the horizontal gradient and vertical gradient of the training output image in the horizontal direction and the vertical direction;

[0190] Searching for image clarity scores corresponding to both the horizontal gradient and the vertical gradient in a preset gradient clarity correspondence table, wherein the gradient clarity correspondence table is used to characterize the correspondence between the first direction gradient and the second direction gradient and the image clarity score;

[0191] The found image clarity score is determined as the above clarity score.

[0192] In some optional implementations, the image generation module is pre-trained through the following training steps:

[0193] Determine the model structure and model parameter information of the initial image generation module;

[0194] Respectively obtain the model structure and model parameter information of the pre-trained first text encoder and the second text encoder;

[0195] For a training sample in the training sample set, the following initial image generation module parameter adjustment operation is performed, and the training sample includes a training text and a corresponding matching training image:

[0196] Based on the training text in the training sample, obtaining a first training text encoding feature through the first text encoder;

[0197] Based on the training text in the training sample, obtaining a second training text encoding feature through the second text encoder;

[0198] Based on the first training text encoding feature and the second training text encoding feature, a joint training text encoding feature is obtained;

[0199] Based on the training image in the training sample and the joint training text encoding features, a training output image is obtained through the initial image generation module;

[0200] Based on the training images in the training sample and the training output images, a loss function is obtained;

[0201] Based on the above loss function, the parameters of the above initial image generation module are adjusted.

[0202] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 suitable for implementing the electronic device of the present disclosure. Figure 7 The computer system 700 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0203] like Figure 7 As shown, the computer system 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. Various programs and data required for the operation of the computer system 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0204] Typically, the following devices may be connected to the I / O interface 705: input devices 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc.; output devices 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 708 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 709. The communication devices 709 may allow the computer system 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The computer system 700 of the electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0205] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0206] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0207] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0208] The computer readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device implements the following Figure 2 The illustrated embodiment and its alternative implementations illustrate a method for generating an image.

[0209] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0210] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0211] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the acquisition module may also be described as "used to obtain the first text encoding feature corresponding to the above prompt text based on the first text encoder".

[0212] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

Claims

1. An image generation method, characterized in that: include: Get the prompt text; Based on the first text encoder, obtaining a first text encoding feature corresponding to the prompt text; Based on a second text encoder, obtaining a second text encoding feature corresponding to the prompt text, wherein the first text encoder and the second text encoder are different; Based on the first text encoding feature and the second text encoding feature, a joint text encoding feature is obtained; Based on the preset noise and the joint text encoding feature, an output image corresponding to the prompt text is obtained through an image generation module.

2. The method according to claim 1, characterized in that: The image generation module includes an encoder, a deep neural network and a decoder; and The method of obtaining an output image corresponding to the prompt text through an image generation module based on the preset noise and the combined text encoding feature includes: Based on the preset noise, obtaining a noise image coding feature through the encoder; Based on the noise image coding feature and the joint text coding feature, the image and text fusion coding feature is obtained through the deep neural network; Based on the image and text fusion coding features, an output image corresponding to the prompt text is obtained through the decoder.

3. The method according to claim 1, characterized in that The second text encoder is pre-trained by the following training steps: Determining the model structure and model parameter information of the initial second text encoder; Respectively obtaining model structures and model parameter information of a pre-trained first text encoder, an image generation module, and a quality evaluation model, wherein the quality evaluation model is used to evaluate the image generation quality of a target image obtained by performing image generation based on a target text; For a training sample in the training sample set, the following second text encoder parameter adjustment operation is performed until a preset training end condition is satisfied, wherein the training sample includes a training text and a corresponding matching training image: Based on the training text in the training sample, obtaining a first training text encoding feature through the first text encoder; Based on the training text in the training sample, obtaining a second training text encoding feature through the initial second text encoder; Based on the first training text encoding feature and the second training text encoding feature, a joint training text encoding feature is obtained; Based on the training image in the training sample and the joint training text encoding features, obtaining a training output image through the image generation module; Based on the training text in the training sample and the training output image, obtaining the training output image generation quality through the image generation quality evaluation model; The parameters of the initial second text encoder are adjusted with the maximization of the training output image generation quality as the optimization goal.

4. The method according to claim 3, characterized in that The image generation module includes an encoder, a deep neural network and a decoder; and The step of obtaining a training output image based on the training image in the training sample and the joint training text encoding feature through the image generation module includes: Based on the training image in the training sample, obtaining the training image encoding feature through the encoder; Based on the training image coding features and the joint training text coding features, obtaining training image and text fusion coding features through the deep neural network; Based on the training image and text fusion coding features, a training output image is obtained through the decoder.

5. The method according to claim 3, characterized in that: The step of obtaining the training output image generation quality based on the training text in the training sample and the training output image by using the image generation quality evaluation model comprises: Based on the training text in the training sample and the training output image, a quality evaluation model is generated through the image to obtain an aesthetic score, a relevance score, and a clarity score; Based on the aesthetic score, relevance score and clarity score, the training output image generation quality is obtained.

6. The method according to claim 5, characterized in that The image generation quality evaluation model includes an aesthetic scoring model, a picture-text relevance scoring model and an image clarity scoring model; and The step of generating a quality evaluation model based on the training text in the training sample and the training output image to obtain an aesthetic score, a relevance score and a clarity score includes: Based on the training output image, obtaining an aesthetic score through the aesthetic scoring model, wherein the aesthetic scoring model is an aesthetic predictor model obtained by training based on the aesthetic scores of the image and the corresponding annotated image; Based on the training text in the training sample and the training output image, obtaining a relevance score through the image-text relevance scoring model; Based on the training output image, obtaining a clarity score through the image clarity scoring model; Wherein, obtaining a clarity score based on the training output image through the image clarity scoring model includes: Calculate the horizontal gradient and vertical gradient of the training output image in the horizontal direction and the vertical direction respectively by using a preset algorithm; Searching for image clarity scores corresponding to both the horizontal gradient and the vertical gradient in a preset gradient clarity correspondence table, wherein the gradient clarity correspondence table is used to characterize the correspondence between the first direction gradient and the second direction gradient and the image clarity score; The found image clarity score is determined as the clarity score.

7. The method according to claim 1, characterized in that The image generation module is pre-trained through the following training steps: Determine the model structure and model parameter information of the initial image generation module; Respectively obtain the model structure and model parameter information of the pre-trained first text encoder and the second text encoder; For a training sample in the training sample set, the following initial image generation module parameter adjustment operation is performed, and the training sample includes a training text and a corresponding matching training image: Based on the training text in the training sample, obtaining a first training text encoding feature through the first text encoder; Based on the training text in the training sample, obtaining a second training text encoding feature through the second text encoder; Based on the first training text encoding feature and the second training text encoding feature, a joint training text encoding feature is obtained; Based on the training image in the training sample and the joint training text encoding features, obtaining a training output image through the initial image generation module; Obtaining a loss function based on the training image in the training sample and the training output image; Based on the loss function, the parameters of the initial image generation module are adjusted.

8. An image generating device, characterized in that: include: The acquisition module is used to obtain the prompt text; A first encoding module, used for obtaining a first text encoding feature corresponding to the prompt text based on a first text encoder; A second encoding module, configured to obtain a second text encoding feature corresponding to the prompt text based on a second text encoder, wherein the first text encoder and the second text encoder are different; A combining module, configured to obtain a combined text encoding feature based on the first text encoding feature and the second text encoding feature; A generation module is used to obtain an output image corresponding to the prompt text through an image generation module based on preset noise and the combined text encoding feature.

9. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by one or more processors, the method according to any one of claims 1 to 7 is implemented.

Images