Image generation method, device, equipment, medium and program product

Abstract

The present application discloses an image generation method, device, equipment, medium and program product in the field of computer technology. The present application selects a target image generation model corresponding to the image style of the target image according to a given target image, and then the target image generation model can fuse the subject image, subject mask and text description information of the target image in the target image to generate a new image, so that the subject and style of the new image are consistent with the subject and style of the given original target image, and the two are more similar, so that the new image can replace the original image.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image generation method, device, equipment, medium and program product. Background Art

[0002] At present, based on image generation technology, we can use the trained image generation model to generate a new image with the same image subject according to a given image, such as Figure 1 As shown in FIG. 1 , new images A1 to A4 can be obtained based on a given image A, and the subjects in these images are all dogs. However, the styles of these new images are not consistent with the style of image A, that is, the new images are only similar to the original images and cannot completely replace the original images.

[0003] Therefore, how to generate a new image that can replace the original image is a problem that those skilled in the art need to solve. Summary of the invention

[0004] In view of this, the purpose of this application is to provide an image generation method, device, equipment, medium and program product to generate a new image that can replace the original image. The specific scheme is as follows:

[0005] In a first aspect, the present application provides an image generation method, comprising:

[0006] Acquire a target image including a target subject and text description information of the target image, and determine an image style of the target image;

[0007] Selecting a target model corresponding to the image style from a plurality of preset image generation models;

[0008] The target image is input into the target model so that the target model splices the subject image features of the target subject and the random image features of the random noise, and generates a new image based on the spliced ​​result, the subject mask of the target subject and the text description information.

[0009] Optionally, determining the image style of the target image includes:

[0010] Comparing the target image with a preset image library to determine in the image library a known style image whose similarity with the target image is greater than a preset threshold;

[0011] Using the image style of the known style image as the image style of the target image;

[0012] or

[0013] Clustering the target image, and determining the image style of the target image according to the clustering result;

[0014] or

[0015] The target image is input into a preset style classification model so that the style classification model outputs the image style of the target image.

[0016] Optionally, selecting a target model corresponding to the image style from a plurality of preset image generation models includes:

[0017] Determining style identification information of the image style;

[0018] An image generation model marked with the style identification information among the multiple image generation models is used as the target model.

[0019] Optionally, the target model includes: a segmentation module, an encoding module, a random generation module, a noise addition module, a denoising module and a decoding module;

[0020] The segmentation module is used to extract the subject mask from the target image;

[0021] The encoding module is used to encode the target image into latent space image features;

[0022] The random generation module is used to generate the random noise at any time;

[0023] The noise adding module is used to perform noise adding processing on the latent space image features to obtain noisy image features;

[0024] The denoising module is used to splice the subject image features and the random image features, and generate new image features based on the spliced ​​result, the subject mask and the text description information;

[0025] The decoding module is used for decoding the new image feature into the new image.

[0026] Optionally, the denoising module includes a self-attention block;

[0027] The self-attention block is used to splice the subject image features and the random image features, and generate new image features based on the spliced ​​result, the subject mask and the text description information.

[0028] Optionally, the self-attention block is used to concatenate the subject image feature and the random image feature using a first formula and a second formula;

[0029] Among them, the first formula is: ; The second formula is: ; The subject image features include the subject key matrix and the subject value matrix ; The random image features include a random key matrix and a matrix of random values ; The result of splicing includes the splicing key matrix and the concatenated value matrix .

[0030] Optionally, the self-attention block is used to fuse and splice a result obtained by using the third formula, the fourth formula and the fifth formula and the subject mask;

[0031] Wherein, the third formula is: ; The fourth formula is: ; The fifth formula is: ;

[0032] in, represents the concatenation result of the random noise and the subject mask; represents the subject mask; ATT represents the self-attention weight value; H represents the splicing value matrix The weighted result with the self-attention weight value ATT; a query matrix representing the target subject; Represents the splicing key matrix The transposed matrix of .

[0033] In a second aspect, the present application provides an image generating device, comprising:

[0034] An acquisition module, used to acquire a target image including a target subject and text description information of the target image, and determine an image style of the target image;

[0035] A selection module, used to select a target model corresponding to the image style from a plurality of preset image generation models;

[0036] A generation module is used to input the target image into the target model so that the target model splices the main image features of the target subject and the random image features of the random noise, and generates a new image based on the splicing result, the main mask of the target subject and the text description information.

[0037] In a third aspect, the present application provides an electronic device, including:

[0038] Memory for storing computer programs;

[0039] A processor is used to execute the computer program to implement the image generation method disclosed above.

[0040] In a fourth aspect, the present application provides a readable storage medium for storing a computer program, wherein the computer program implements the aforementioned disclosed image generation method when executed by a processor.

[0041] In a fifth aspect, the present application provides a computer program product, comprising a computer program / instruction, which implements the steps of the aforementioned disclosed image generation method when executed by a processor.

[0042] It can be seen from the above scheme that the present application provides an image generation method, including: obtaining a target image including a target subject and text description information of the target image, and determining the image style of the target image; selecting a target model corresponding to the image style from a plurality of preset image generation models; inputting the target image into the target model so that the target model splices the subject image features of the target subject and the random image features of random noise, and generates a new image based on the spliced ​​result, the subject mask of the target subject and the text description information.

[0043] It can be seen that the present application can select a target image generation model corresponding to the image style of the target image based on a given target image, and then the target image generation model can fuse the subject image, subject mask and text description information in the target image to generate a new image, so that the subject and style of the new image are consistent with the subject and style of the given original target image, and the two are more similar, so that the new image can replace the original image.

[0044] Correspondingly, an image generation device, equipment, medium and program product provided by the present application also have the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0046] Figure 1 A schematic diagram of image generation disclosed in this application;

[0047] Figure 2 A flow chart of an image generation method disclosed in this application;

[0048] Figure 3 A schematic diagram of the structure of a Unet disclosed in this application;

[0049] Figure 4 A comparison chart of image generation effects of different image generation schemes disclosed in this application;

[0050] Figure 5 This is a schematic diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] At present, based on image generation technology, a new image with the same image subject can be generated based on a given image, such as Figure 1 As shown, new images A1~A4 can be obtained based on a given image A, and the subjects in these images are all dogs. However, the styles of these new images are inconsistent with the style of image A, that is, the new images are only similar to the original images and cannot completely replace the original images. To this end, the present application provides an image generation scheme that can generate new images that can replace the original images and have the same subject and style as the given original target image.

[0053] See also Figure 2 As shown, the embodiment of the present application discloses an image generation method, including:

[0054] S201: Acquire a target image including a target subject and text description information of the target image, and determine an image style of the target image.

[0055] In one embodiment, determining the image style of a target image includes: comparing the target image with a preset image library to determine a known style image in the image library whose similarity with the target image is greater than a preset threshold; using the image style of the known style image as the image style of the target image; or clustering the target image and determining the image style of the target image based on the clustering results; or inputting the target image into a preset style classification model so that the style classification model outputs the image style of the target image.

[0056] S202: Select a target model corresponding to the image style from a plurality of preset image generation models.

[0057] In this embodiment, multiple image generation models are preset, and different image generation models correspond to different image styles, which can be: ancient style, science fiction, 3D realism, 3D cartoon, and two-dimensional cartoon.

[0058] In one embodiment, selecting a target model corresponding to an image style from a plurality of preset image generation models includes: determining style identification information of the image style; and using an image generation model marked with style identification information among the plurality of image generation models as a target model.

[0059] S203, inputting the target image into the target model, so that the target model splices the subject image features of the target subject and the random image features of the random noise, and generates a new image based on the spliced ​​result, the subject mask of the target subject and the text description information.

[0060] In one embodiment, the target model includes: a segmentation module, an encoding module, a random generation module, a noise addition module, a denoising module and a decoding module; the segmentation module is used to extract a subject mask from a target image; the encoding module is used to encode the target image into latent space image features; the random generation module is used to generate random noise at any time; the noise addition module is used to perform noise processing on the latent space image features to obtain noisy image features; the denoising module is used to splice the subject image features and the random image features, and generate new image features based on the spliced ​​results, the subject mask and the text description information; the decoding module is used to decode the new image features into a new image.

[0061] In one embodiment, the denoising module includes a self-attention block; the self-attention block is used to splice the subject image features and the random image features, and generate new image features based on the spliced ​​results, the subject mask and the text description information.

[0062] Specifically, the self-attention block is used to splice the subject image features and the random image features using the first formula and the second formula; wherein the first formula is: ; The second formula is: ; The subject image features include the subject key matrix and the subject value matrix ; The random image features include a random key matrix and a matrix of random values ; The result of splicing includes the splicing key matrix and the concatenated value matrix .

[0063] In one implementation, the self-attention block is used to fuse the result obtained by splicing using the third formula, the fourth formula, and the fifth formula and the subject mask; wherein the third formula is: ; The fourth formula is: ; The fifth formula is: ;in, represents the concatenation result of the random noise and the subject mask; represents the subject mask; ATT represents the self-attention weight value; H represents the splicing value matrix The weighted result with the self-attention weight value ATT; a query matrix representing the target subject; Represents the splicing key matrix The transposed matrix of .

[0064] In this embodiment, Q is a query matrix (query), K is a key matrix (key), and V is a value matrix (value). K is used to calculate the association matrix ATT of Q and V.

[0065] It should be noted that it is often difficult for a single image generation model to learn every style well. Therefore, the present application pre-trains multiple image generation models of different styles. When a reference image is given, an image generation model that learns the corresponding style is selected from these image generation models to generate a new image. Among them, the image style of the reference image can be determined by clustering, classification, etc. After determining the image style of the reference image, the reference image is input into the image generation model corresponding to the corresponding style, and the encoder (encoding module) in the image generation model is used to convert the reference image into a reference latent (unnoised) in the latent space; then the forward denoising module in the model is used to denoise the reference latent to obtain the noisy reference latent at each time step. A noise latent is randomly initialized, and the noise latent and the noisy reference latent are sent to the Unet (denoising module) in the image generation model; the denoising result is output through each block in the Unet. Among them, when the noise latent and the reference latent after adding noise pass through the self-attention blocks in each block in Unet, the self-attention block will K and V splice the random latent with the reference latent at the current time step, that is: the self-attention block adaptively learns the information of the reference image to ensure the consistency of the subject image; at the same time, the mask of the reference subject will be introduced, and then the self-attention will be calculated so that the reference subject information can be injected into the new image; after denoising once by Unet, the result output by Unet will be input into Unet again, and then the above denoising process will be repeated in Unet until the denoised latent of the last time step is obtained. After the denoised latent is decoded by the decoder (decoding module) in the image generation model, a new image with the same style and subject is obtained. That is: after the last denoising, the last denoising result is decoded to obtain a newly generated image.

[0066] Among them, when the forward denoising module adds noise to the latent of the reference image, the denoising result of each step is retained.

[0067] In addition, the self-attention block extracts the matrices q, k, v and q_ref, k_ref, v_ref of the random latent and the reference latent respectively, and then concatenates the k_ref, v_ref of the reference latent and the k, v of the random latent, and then performs the self-attention calculation. The purpose of this is to integrate the information of the reference image into the image generation process, so that the generated image is consistent with the reference image.

[0068] See also Figure 3 , Unet consists of a Conv_in, three CrossAttnDownBlock2D, a DownBlock2D, a UnetMidBlock2DCrossAttn, an UpBlock2D, three CrossAttnUpBlock2D and a Conv_out. Each module containing CrossAttn contains a residual block (fused with the time step feature), a self-attention block (for its own feature extraction) and a cross-attention block (for feature fusion with the input text information). The self-attention block mainly performs the K and V splicing of the random latent and the reference latent.

[0069] Specifically, the processing flow of random latent and reference latent in Unet includes: in the first time step, the input random latent will pass through a Conv_in, three CrossAttnDownBlock2D, a DownBlock2D, a UnetMidBlock2DCrossAttn, an UpBlock2D, three CrossAttnUpBlock2D and a Conv_out in sequence to obtain the predicted noise. Among them, when passing through the self-attention block in CrossAttn, the K and V splicing of the input random latent and the reference latent will be performed, and the mask of the reference subject will be introduced for self-attention calculation, thereby realizing the injection of reference subject information. The denoised latent can be obtained according to the noise predicted by Unet, and then the denoised latent is sent to Unet again, and the above process is repeated for denoising until the denoised latent of the last time step is obtained.

[0070] In one example, the concatenation of K and V of the random latent and the reference latent, and the injection calculation of the reference subject information can refer to the following calculation formula:

[0071] , used to splice two Ks;

[0072] , used to splice two Vs;

[0073] , used to stitch the main Mask and random latent of the reference image;

[0074] ; ;

[0075] in, , Calculated by reference latent, is the mask of the reference image, ATT is the self-attention weight added to the mask, and H is the output result after ATT weighting.

[0076] See also Figure 4 By comparing the images generated by this application with those generated by other solutions, it can be seen that the new images generated by this application are closer to the reference images, specifically, the subject and style of the characters are closer to the reference images.

[0077] An image generating device provided in an embodiment of the present application is introduced below. The image generating device described below and the image generating method described above can refer to each other.

[0078] The present application embodiment discloses that the present application provides an image generating device, including:

[0079] An acquisition module, used to acquire a target image including a target subject and text description information of the target image, and determine an image style of the target image;

[0080] A selection module, used to select a target model corresponding to an image style from a plurality of preset image generation models;

[0081] The generation module is used to input the target image into the target model so that the target model splices the main image features of the target subject and the random image features of the random noise, and generates a new image based on the spliced ​​result, the main mask of the target subject and the text description information.

[0082] In one implementation, the acquisition module is specifically used to:

[0083] Comparing the target image with a preset image library to determine in the image library a known style image whose similarity with the target image is greater than a preset threshold;

[0084] Taking the image style of the known style image as the image style of the target image;

[0085] or

[0086] Clustering the target image, and determining the image style of the target image according to the clustering result;

[0087] or

[0088] The target image is input into a preset style classification model so that the style classification model outputs the image style of the target image.

[0089] In one embodiment, the selection module is specifically used to:

[0090] Determine style identification information of the image style;

[0091] An image generation model marked with style identification information among multiple image generation models is used as a target model.

[0092] In one embodiment, the target model includes: a segmentation module, an encoding module, a random generation module, a noise addition module, a noise removal module and a decoding module;

[0093] The segmentation module is used to extract the subject mask from the target image;

[0094] The encoding module is used to encode the target image into latent space image features;

[0095] The random generation module is used to generate random noise at any time;

[0096] The noise adding module is used to perform noise adding processing on the latent space image features to obtain the noisy image features;

[0097] The denoising module is used to splice the subject image features and the random image features, and generate new image features based on the spliced ​​results, the subject mask and the text description information;

[0098] The decoding module is used to decode the new image features into a new image.

[0099] In one embodiment, the denoising module includes a self-attention block;

[0100] The self-attention block is used to concatenate the subject image features and the random image features, and generate new image features based on the concatenated results, the subject mask and the text description information.

[0101] In one embodiment, the self-attention block is used to concatenate the subject image features and the random image features using a first formula and a second formula; wherein the first formula is: ; The second formula is: ; The subject image features include the subject key matrix and the subject value matrix ; The random image features include a random key matrix and a matrix of random values ; The result of splicing includes the splicing key matrix and the concatenated value matrix .

[0102] In one implementation, the self-attention block is used to fuse the result obtained by splicing using the third formula, the fourth formula, and the fifth formula and the subject mask; wherein the third formula is: ; The fourth formula is: ; The fifth formula is: ;in, represents the concatenation result of the random noise and the subject mask; represents the subject mask; ATT represents the self-attention weight value; H represents the splicing value matrix The weighted result with the self-attention weight value ATT; a query matrix representing the target subject; Represents the splicing key matrix The transposed matrix of .

[0103] Among them, for more specific working processes of each module and unit in this embodiment, reference can be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0104] It can be seen that this embodiment provides an image generating device, which can generate a new image whose subject and style are consistent with the subject and style of a given original target image and can replace the original image.

[0105] An electronic device provided in an embodiment of the present application is introduced below. The electronic device described below and the image generating method and device described above can be referenced to each other.

[0106] See also Figure 5 As shown, the embodiment of the present application discloses an electronic device, including:

[0107] Memory 501, used for storing computer programs;

[0108] The processor 502 is used to execute the computer program to implement the method disclosed in any of the above embodiments.

[0109] A readable storage medium provided in an embodiment of the present application is introduced below. The readable storage medium described below and the image generation method, device and apparatus described above can be referenced to each other.

[0110] A readable storage medium is used to store a computer program, wherein the computer program, when executed by a processor, implements the image generation method disclosed in the above embodiment. The specific steps of the method can refer to the corresponding contents disclosed in the above embodiment, and will not be repeated here.

[0111] A computer program product provided in an embodiment of the present application is introduced below. The computer program product described below can be cross-referenced with other embodiments described in this document.

[0112] A computer program product comprises a computer program / instruction, which implements the steps of the aforementioned disclosed image generation method when executed by a processor.

[0113] The "first", "second", "third", "fourth", etc. (if any) referred to in this application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods or devices.

[0114] It should be noted that the descriptions involving "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0115] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0116] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of readable storage medium known in the art.

[0117] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image generation method, characterized in that: include: Acquire a target image including a target subject and text description information of the target image, and determine an image style of the target image; Selecting a target model corresponding to the image style from a plurality of preset image generation models; Inputting the target image into the target model so that the target model splices the subject image features of the target subject and the random image features of the random noise, and generates a new image based on the spliced ​​result, the subject mask of the target subject and the text description information; The target model includes: a segmentation module, an encoding module, a random generation module, a noise addition module, a noise removal module and a decoding module; The segmentation module is used to extract the subject mask from the target image; The encoding module is used to encode the target image into latent space image features; The random generation module is used to generate the random noise at any time; The noise adding module is used to perform noise adding processing on the latent space image features to obtain noisy image features; The denoising module is used to splice the subject image features and the random image features, and generate new image features based on the spliced ​​result, the subject mask and the text description information; The decoding module is used to decode the new image feature into the new image; Wherein, the denoising module includes a self-attention block; The self-attention block is used to splice the subject image feature and the random image feature, and generate a new image feature based on the spliced ​​result, the subject mask and the text description information; Wherein, the self-attention block is used to splice the subject image feature and the random image feature using the first formula and the second formula; Among them, the first formula is: ; The second formula is: ; The subject image features include the subject key matrix and the subject value matrix ; The random image features include a random key matrix and a matrix of random values ; The result of splicing includes the splicing key matrix and the concatenated value matrix ; The self-attention block is used to fuse the result obtained by using the third formula, the fourth formula and the fifth formula and the subject mask; Wherein, the third formula is: ; The fourth formula is: ; The fifth formula is: ; in, represents the concatenation result of the random noise and the subject mask; represents the subject mask; ATT Represents the self-attention weight value; H Represents the concatenated value matrix and self-attention weight ATT The weighted result of a query matrix representing the target subject; Represents the splicing key matrix The transposed matrix of .

2. The method according to claim 1, characterized in that Determining the image style of the target image includes: Comparing the target image with a preset image library to determine in the image library a known style image whose similarity with the target image is greater than a preset threshold; Using the image style of the known style image as the image style of the target image; or Clustering the target image, and determining the image style of the target image according to the clustering result; or The target image is input into a preset style classification model so that the style classification model outputs the image style of the target image.

3. The method according to claim 1, characterized in that Selecting a target model corresponding to the image style from a plurality of preset image generation models includes: Determining style identification information of the image style; An image generation model marked with the style identification information among the multiple image generation models is used as the target model.

4. An image generating device, characterized in that: include: An acquisition module, used to acquire a target image including a target subject and text description information of the target image, and determine an image style of the target image; A selection module, used to select a target model corresponding to the image style from a plurality of preset image generation models; A generating module, configured to input the target image into the target model, so that the target model splices the subject image features of the target subject and the random image features of the random noise, and generates a new image based on the spliced ​​result, the subject mask of the target subject and the text description information; The target model includes: a segmentation module, an encoding module, a random generation module, a noise addition module, a noise removal module and a decoding module; The segmentation module is used to extract the subject mask from the target image; The encoding module is used to encode the target image into latent space image features; The random generation module is used to generate the random noise at any time; The noise adding module is used to perform noise adding processing on the latent space image features to obtain noisy image features; The denoising module is used to splice the subject image features and the random image features, and generate new image features based on the spliced ​​result, the subject mask and the text description information; The decoding module is used to decode the new image feature into the new image; Wherein, the denoising module includes a self-attention block; The self-attention block is used to splice the subject image feature and the random image feature, and generate a new image feature based on the spliced ​​result, the subject mask and the text description information; Wherein, the self-attention block is used to splice the subject image feature and the random image feature using the first formula and the second formula; Among them, the first formula is: ; The second formula is: ; The subject image features include the subject key matrix and the subject value matrix ; The random image features include a random key matrix and a matrix of random values ; The result of splicing includes the splicing key matrix and the concatenated value matrix ; The self-attention block is used to fuse the result obtained by using the third formula, the fourth formula and the fifth formula and the subject mask; Wherein, the third formula is: ; The fourth formula is: ; The fifth formula is: ; in, represents the concatenation result of the random noise and the subject mask; represents the subject mask; ATT Represents the self-attention weight value; H Represents the splicing value matrix and self-attention weight ATT The weighted result of a query matrix representing the target subject; Represents the splicing key matrix The transposed matrix of .

5. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 3.

6. A readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

7. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images