Method and apparatus for generating image having personalized style, and storage medium

Abstract

The present invention relates to the technical field of artificial intelligence. Disclosed are a method and apparatus for generating an image having a personalized style, and a storage medium, aiming at solving the technical problems of quality and accuracy of a generated image. The method comprises: selecting a style image, and inputting the style image into a pre-constructed VGG19_f3 network model to obtain feature maps having different sizes; calculating a Gram matrix of the feature maps to extract style information of the feature maps; inputting a code of a text obtained by a text encoder, a noise image, and the style information into a pre-constructed noise prediction network with a style guide module, to perform noise prediction so as to obtain predicted noise; and using the predicted noise to perform repeated denoising operation on the noise image to obtain a latent space image, and decoding the latent space image by means of an image decoder of an SD to obtain a finally generated image having a personalized style. According to the present invention, a model can generate, in an end-to-end mode, an image having a personalized style, so that the model has a stronger style personalization capability, and the quality and accuracy of a generated image are ensured.

Description

Method, device and storage medium for generating image with personalized style Technical Field

[0001] The present invention relates to a method, device and storage medium for generating an image with personalized style, and belongs to the field of artificial intelligence technology. Background Art

[0002] Intelligent generation of illustrations for electronic publications is essentially a process of generating images based on text. Existing large-scale image generation models can achieve high-quality and diverse image generation based on natural language text prompts. However, in order to attract more readers, illustration generation for electronic publications requires the model to generate personalized illustrations based on the user's personal style preferences. In this context, existing large-scale image generation models lack style personalization capabilities and cannot meet application requirements.

[0003] For the problem of image generation, existing research is usually based on pre-trained large-scale image generation models to fully utilize the image generation capabilities of large models. Most of the solutions require dividing the model into multiple stages or modules to train the model, or require fine-tuning the pre-trained large-scale image generation model in many steps. Compared with end-to-end models, the complexity is higher, and every time a new style is encountered, the model weights need to be re-fine-tuned for it, which is both time-consuming and storage-consuming. Therefore, the scalability and practicality of these methods are greatly limited. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a style-personalized image generation method, device and storage medium, which can use style features to guide the image generation process and do not need to retrain the model for each new style. At the same time, by constructing a noise prediction network with a style guidance module, the model has stronger style personalization capabilities while ensuring the quality and accuracy of the generated images.

[0005] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0006] In a first aspect, the present invention provides a method for generating an image with personalized style, comprising:

[0007] Select a style image and input it into the pre-built VGG19_f3 network model to obtain feature maps of different sizes;

[0008] Calculating the Gram matrix of the feature map to extract style information of the feature map;

[0009] The text encoding obtained by the text encoder, the randomly sampled noise image, and the style information are input into a pre-built noise prediction network with a style guidance module to perform noise prediction and obtain predicted noise;

[0010] Repeatedly performing denoising operations on the noisy image using the predicted noise to obtain a latent space image;

[0011] The latent space image is decoded by the SD image decoder to obtain the final generated style personalized image.

[0012] In combination with the first aspect, further, obtaining feature maps of different sizes includes:

[0013] The pixel size of the style image is set to 512×512;

[0014] The first three downsampling blocks of the VGG19 network model are used to form a VGG19_f3 network model, and the style image is input into the VGG19_f3 network model, outputting a first feature map m1, a second feature map m2, and a third feature map m3 of sizes 128×128×64, 64×64×128, and 32×32×256, respectively. In combination with the first aspect, further, extracting style information from the feature map includes:

[0015] Calculating the Gram matrices of the first feature map m1, the second feature map m2, and the third feature map m3 to obtain a first style feature G(m1), a second style feature G(m2), and a third style feature G(m3);

[0016] The first style feature (m1) and the second style feature G(m2) are subjected to a maximum pooling operation and then added to the third style feature G(m3) to obtain style information S.

[0017] In combination with the first aspect, further, the calculation expression of the Gram matrix of the feature maps m1, m2 and m3 is as follows:

[0018] Among them, G ij is an element of the Gram matrix; F ik is the kth element of channel i of the feature map; F jk is the kth element of channel j of the feature map; G(F) is the Gram matrix of the feature map F; H is the height of the feature map F; W is the width of the feature map F.

[0019] In combination with the first aspect, further, the construction process of the noise prediction network with the style guidance module is as follows:

[0020] The downsampling block and the intermediate block of the noise prediction network in the SD are copied to obtain a downsampling network. The downsampling network and the noise prediction network in the SD constitute a noise prediction network with a style guidance module.

[0021] In combination with the first aspect, further, obtaining the predicted noise includes:

[0022] Inputting the style information into the downsampling network to obtain first style information S1, second style information S2, third style information S3, fourth style information S4, and fifth style information S5;

[0023] The code C of the text obtained by the text encoder θ (p), randomly sampled noise image x t Input the downsampling block and the intermediate block in the noise prediction network in sequence, and output the first predicted noise;

[0024] The fifth style information S5, the fourth style information S4, the third style information S3, the second style information S2, and the first style information S1 are sequentially added to the first predicted noise to output the final predicted noise ε θ (x t ,t,C θ (p),S).

[0025] In combination with the first aspect, further, obtaining the latent space image includes:

[0026] The predicted noise is used to perform a first denoising operation on the noisy image to obtain a first denoised image x t-1 , the first denoised image x t-1 , style personalized features, text encoding and then input into the pre-built noise prediction network with style guidance module, outputting the first denoising prediction result ∈ θ (x t-1 ,t,C θ (p), S), using the first denoising prediction result to denoise the first denoising image x t-1 Perform the second denoising operation to obtain the second denoised image x t-2 ;

[0027] Repeat the above denoising operation 50 times to obtain the latent space image x0.

[0028] In combination with the first aspect, further, the expression of the denoising operation is as follows:

[0029] α t =1-β t

[0030] Among them, β tis the added noise variance; α t Because of β t The calculated decreasing sequence; z is a noise image randomly selected from the standard normal distribution; x t-1 is the noise image after one-step denoising; x t is the noise image at step t; C θ (p) is the encoded text information; t is the step size; S is the style information; ε θ (x t ,t,C θ (p),S) is the predicted noise.

[0031] In a second aspect, a device for generating an image with personalized style is provided, the device comprising:

[0032] An image input module is used to select a style image and input the style image into a pre-built VGG19_f3 network model to obtain feature maps of different sizes;

[0033] A style information extraction module is used to calculate the Gram matrix of the feature map to extract the style information of the feature map;

[0034] The noise prediction module is used to input the text encoding obtained by the text encoder, the randomly sampled noise image, and the style information into a pre-built noise prediction network with a style guidance module to perform noise prediction and obtain predicted noise;

[0035] a noise removal module, configured to perform repeated denoising operations on the noisy image using the predicted noise to obtain a latent space image;

[0036] The image decoding module decodes the latent space image through the SD image decoder to obtain the final style personalized image.

[0037] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the aforementioned methods when executed by a processor.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention uses the VGG19-f3 network model to extract feature maps of different pixel sizes, and uses the Gram matrix to calculate the style features of each layer of the feature map to extract style information. The style information is used to guide the image generation process. The content features of the image do not participate in the guidance, so there is no need to use images corresponding to the content to train the model. The noise prediction network with a style guidance module constructed by the present invention does not need to train a model separately for each new style, saving time cost and space cost used to store the model, and making full use of the image style features, which can not only make the image have stronger style personalization ability, but also ensure the quality and accuracy of the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] FIG1 is a schematic diagram of an image generation process provided by an embodiment of the present invention;

[0041] FIG2 is a diagram of style image data provided by an embodiment of the present invention;

[0042] FIG3 is a comparison diagram of an image provided by an embodiment of the present invention and the currently mainstream conditional image generation model ControlNet. DETAILED DESCRIPTION

[0043] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. Unless there is a conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0044] The term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " in this document generally indicates an "or" relationship between the related objects.

[0045] Example 1

[0046] FIG1 is a flowchart of a method for generating a personalized image in a style according to a first embodiment of the present invention. This flowchart merely illustrates the logical sequence of the method described in this embodiment. In other possible embodiments of the present invention, the steps shown or described may be performed in a different order than that shown in FIG1 , provided that no conflict exists.

[0047] Referring to FIG1 , the method of this embodiment specifically includes the following steps:

[0048] Step S1: Select a style image and input the style image into the pre-built VGG19_f3 network model to obtain feature maps of different sizes;

[0049] Specifically, referring to Figure 2, for the selected style image data, in order to adapt to the network model's requirements for input size, its pixel size is set to 512×512, and the VGG19_f3 network model composed of the first three downsampling blocks of the VGG19 network model is used. The style image data with a pixel size of 512×512 is input into the model to extract the first feature map m1, the second feature map m2 and the third feature map m3 with sizes of 128×128×64, 64×64×128 and 32×32×256 respectively, and these three feature maps are saved for the subsequent extraction of style information.

[0050] Among them, the composition and parameters of the VGG19_f3 network model are shown in Table 1 below:

[0051] Table 1

[0052] Step S2: Calculate the Gram matrix of the feature map to extract the style information of the feature map;

[0053] Specifically, the style information of the style image data is extracted by calculating the Gram matrix of the first feature map m1, the second feature map m2, and the third feature map m3. The calculation expression of the Gram matrix is ​​as follows:

[0054] Among them, G ij is an element of the Gram matrix; F ik is the kth element of channel i of the feature map; F jk is the kth element of channel j of the feature map; G(F) is the Gram matrix of the feature map F; H is the height of the feature map F; W is the width of the feature map F.

[0055] Furthermore, the style features of each layer are obtained by using the calculation expression of the above Gram matrix Perform the maximum pooling operation on G(m1) and G(m2) and add them to G(m3) to obtain the style information S.

[0056] Step S3: Inputting the text encoding obtained by the text encoder, the randomly sampled noise image, and the style information into a pre-built noise prediction network with a style guidance module to perform noise prediction and obtain predicted noise;

[0057] Specifically, a large-scale image generation model (SD) is used to generate images, and a noise prediction network with a style guidance module is designed based on the SD model structure.

[0058] The noise prediction network in SD includes 4 downsampling blocks D1, D2, D3 and D4, 1 middle block M and 4 upsampling blocks U1, U2, U3 and U4.

[0059] Furthermore, the downsampling blocks and intermediate blocks in the noise prediction network are copied to obtain downsampling block copies D′1, D′2, D′3, D′4 and the intermediate block M′. A total of five network blocks together constitute the downsampling network. The downsampling network obtained after network block replication and the noise prediction network in SD constitute a noise prediction network with a style guidance module.

[0060] It should be noted that the image feature sizes of the downsampled block copies D′1, D′2, D′3, D′4 and the intermediate block M′ are 64×64, 32×32, 16×16, 8×8, and 8×8, respectively.

[0061] From Figure 1, we can conclude that the output of the noise prediction network is added to the output of the downsampling block to obtain the final noise prediction. The specific process is as follows:

[0062] When the style personalized feature S obtained in step 2 is input into the downsampling network, the output D′1 can be obtained Output of D′2 Output of D′3 Output of D′4 and the output of M′ At the same time, the code C of the input text is obtained based on the input text prompt θ (p), and a randomly sampled noise image x1 is input to the downsampling block D1, the output of the downsampling block D1 is input to the downsampling block D2, the output of the downsampling block D2 is input to the downsampling block D3, the output of the downsampling block D3 is input to the downsampling block D4, and the output of the downsampling block D4 is input to the middle block M;

[0063] Then, add the output S5 of M′ to the output of the intermediate block M as the input of the upsampling block U4, add the feature map S4 to the output of the upsampling block U4 as the input of the upsampling block U3, add the feature map S3 to the output of the upsampling block U3 as the input of the upsampling block U2, add the feature map S2 to the output of the upsampling block U2 as the input of the upsampling block U1, and finally add the feature map S1 to the output of the upsampling block U1 to output the final predicted noise ∈ θ (x t ,t,C θ (p),S).

[0064] Step S4: repeatedly performing denoising operations on the noisy image using the predicted noise to obtain a latent space image;

[0065] Specifically, using the prediction noise ∈θ (x t ,t,C θ (p),S) for the noisy image x t Perform the first denoising operation to obtain the first denoised image x t-1 , the first denoised image x t-1 , style information S, text encoding C θ (p) is then input into the pre-built noise prediction network with style guidance module, and the first denoising prediction result ∈ θ (x t-1 ,t,C θ (p), S), using the first denoising prediction result to denoise the first denoising image x t-1 Perform the second denoising operation to obtain the second denoised image x t-2 , and again the second denoised image x t-2 , style information S, text encoding C θ (p) is then input into the pre-built noise prediction network with style guidance module, and the second denoising prediction result ∈ θ (x t-2 ,t,C θ (p), S); repeat the above denoising operation 50 times until the latent space image x0 is obtained.

[0066] It should be noted that the final prediction noise ∈ θ (x t ,t,C θ (p),S) for the noisy image x t Denoising, the denoising expression is as follows:

[0067] α t =1-β t

[0068] Among them, β t is the added noise variance, and is a hyperparameter that increases linearly with t, β t ∈[0,1]; α t Because of β t The calculated decreasing sequence; z is the noise randomly sampled from the standard normal distribution, that is, Use this expression for the noise image x t After one step of denoising, we get the noisy image x t-1 .

[0069] Furthermore, the training method of the noise prediction network with the style guidance module includes:

[0070] For a selected original image, noise is continuously added to it, and the encoding C of the noisy image and its corresponding text isθ (p), style information S is input to the noise prediction network ε with style guidance module θ , calculate the predicted noise ∈ θ (x t ,t,C θ (p), S) and the loss of added real noise ∈ to optimize the network, and its loss function L is expressed as:

[0071] Step S5: Use the image decoder in SD to decode the latent space image x0 to obtain the final style personalized image.

[0072] Example 2

[0073] In the embodiment of the present invention, the method and model of the above-mentioned embodiment 1 are compared with the current mainstream conditional image generation model ControlNet on a test set consisting of 2000 user data. The experiment on this dataset calculates the indicators of the images generated by the model in terms of true value and style image set respectively. The overall performance comparison results are shown in Table 2, where FID is Fréchet Inception Distance, an important indicator for measuring the quality of generated images in the field of image generation; KID is Kernel Inception Distance, an important indicator for measuring the diversity of generated images; CLIPSIM is an important indicator for measuring the degree of consistency between generated images and text. StySim is Style Similarity, an indicator proposed by the present invention for measuring image style similarity, and its calculation method is:

[0074] For two images I and P, use the first four network blocks of the VGG19 network to extract their features, and the output of each layer is I i ,i=1,2,3,4 and P i , i=1,2,3,4, calculate their Gram matrices respectively to get G(I i ),i=1,2,3,4 and G(P i ), i = 1, 2, 3, 4, calculate the mean square error of a pair of Gram matrices at each layer, and finally sum the calculation results of the four mean square errors to obtain the style similarity. The calculation formula of StySim is

[0075] Among them, n i ,i=1,2,3,4 is the number of elements in the Gram matrix in each layer. The smaller the value of this indicator is, the more similar the styles of the two images are, and the stronger the style personalization performance of the model is. i ) is 1 i Gram matrix; G(P i ) is Pi Gram matrix; StySim is the index Style Similarity for measuring the style similarity of images.

[0076] Table 2

[0077] As can be seen from Table 2, the performance of the present invention is comparable to that of ControlNet in terms of the quality index FID, but is lower than that of ControlNet in terms of the diversity index KID. This is because the style of the generated images is limited, resulting in a decrease in the diversity of the generated images, which indirectly shows that the style personalization performance of the present invention is stronger. Compared with ControlNet in terms of the style similarity index StySim, the present invention shows an improvement of 4.3% ((1.85-1.77) ÷ 1.85 × 100%) and 7.5% ((2.14-1.98) ÷ 2.14 × 100%) on the true value image set and style image set, respectively. In terms of the text image consistency index CLIPSIM, the present invention is comparable to ControlNet.

[0078] The generation results of different methods under a set of style conditions are shown in Figure 3. The lion image on the left is the true value, the lion image in the middle is the style-personalized image obtained by the present invention, and the lion on the far right is the image generated by the current mainstream conditional image generation model ControlNet. Therefore, compared with ControlNet, the style personalization effect of the present invention is more obvious and closer to the flat painted style of the style image and the true value.

[0079] Example 3

[0080] A device for generating an image with personalized style, comprising:

[0081] An image input module is used to select a style image and input the style image into a pre-built VGG19_f3 network model to obtain feature maps of different sizes;

[0082] A style information extraction module is used to calculate the Gram matrix of the feature map to extract the style information of the feature map;

[0083] The noise prediction module is used to input the text encoding obtained by the text encoder, the randomly sampled noise image, and the style information into a pre-built noise prediction network with a style guidance module to perform noise prediction and obtain predicted noise;

[0084] a noise removal module, configured to perform repeated denoising operations on the noisy image using the predicted noise to obtain a latent space image;

[0085] The image decoding module decodes the latent space image through the SD image decoder to obtain the final style personalized image.

[0086] Example 4

[0087] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the following method when executed by a processor.

[0088] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0089] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0090] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0092] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for generating an image with personalized style, characterized in that: include: Select a style image and input it into the pre-built VGG19_f3 network model to obtain feature maps of different sizes; Calculating the Gram matrix of the feature map to extract style information of the feature map; The text encoding obtained by the text encoder, the randomly sampled noise image, and the style information are input into a pre-built noise prediction network with a style guidance module to perform noise prediction and obtain predicted noise; Repeatedly performing denoising operations on the noisy image using the predicted noise to obtain a latent space image; The latent space image is decoded by the SD image decoder to obtain the final generated style personalized image.

2. The method for generating an image with personalized style according to claim 1, characterized in that: The obtaining of feature maps of different sizes includes: The pixel size of the style image is set to 512×512; The first three downsampling blocks of the VGG19 network model are used to form a VGG19_f3 network model, and the style image is input into the VGG19_f3 network model, and the first feature map m1, the second feature map m2 and the third feature map m3 with output sizes of 128×128×64, 64×64×128 and 32×32×256 respectively.

3. The method for generating an image with personalized style according to claim 2, characterized in that: Extracting style information of the feature map includes: Calculating the Gram matrices of the first feature map m1, the second feature map m2, and the third feature map m3 to obtain a first style feature G(m1), a second style feature G(m2), and a third style feature G(m3); The first style feature (m1) and the second style feature G(m2) are subjected to a maximum pooling operation and then added to the third style feature G(m3) to obtain style information S.

4. The method for intelligently generating illustrations based on electronic publications according to claim 3, characterized in that: The calculation expression of the Gram matrix of the feature maps m1, m2 and m3 is as follows: Among them, G ij is an element of the Gram matrix; F ik is the kth element of channel i of the feature map; F jk is the kth element of channel j of the feature map; G(F) is the Gram matrix of the feature map F; H is the height of the feature map F; W is the width of the feature map F.

5. The method for generating an image with personalized style according to claim 1, characterized in that: The construction process of the noise prediction network with style guidance module is as follows: The downsampling block and the intermediate block of the noise prediction network in the SD are copied to obtain a downsampling network. The downsampling network and the noise prediction network in the SD constitute a noise prediction network with a style guidance module.

6. The method for generating an image with personalized style according to claim 5, characterized in that: The obtaining of the prediction noise comprises: Inputting the style information into the downsampling network to obtain first style information S1, second style information S2, third style information S3, fourth style information S4, and fifth style information S5; The code C of the text obtained by the text encoder θ (p), randomly sampled noise image x t Input the downsampling block and the intermediate block in the noise prediction network in sequence, and output the first predicted noise; The fifth style information S5, the fourth style information S4, the third style information S3, the second style information S2, and the first style information S1 are sequentially added to the first predicted noise to output the final predicted noise ε θ (x t ,t,C θ (p),S).

7. The method for generating an image with personalized style according to claim 1, characterized in that: The obtaining of the latent space image comprises: The predicted noise is used to perform a first denoising operation on the noisy image to obtain a first denoised image x t-1 , the first denoised image x t-1 , style information S, text encoding C θ (p) is then input into the pre-built noise prediction network with style guidance module, and the first denoising prediction result ε is output θ (x t-1 ,t,C θ (p), S), using the first denoising prediction result to denoise the first denoising image x t-1 Perform the second denoising operation to obtain the second denoised image x t-2 ; Repeat the above denoising operation 50 times to obtain the latent space image x0.

8. The method for generating an image with personalized style according to claim 7, characterized in that: The expression of the denoising operation is as follows: α t =1-β t Among them, β t is the added noise variance; α t Because of β t The calculated decreasing sequence; z is a noise image randomly selected from the standard normal distribution; x t-1 is the noise image after one-step denoising; x t is the noise image at step t; C θ (p) is the encoded text information; t is the step size; S is the style information; ε θ (x t ,t,C θ (p),S) is the predicted noise.

9. A device for generating an image with personalized style, characterized in that: The device comprises: An image input module is used to select a style image and input the style image into a pre-built VGG19_f3 network model to obtain feature maps of different sizes; A style information extraction module is used to calculate the Gram matrix of the feature map to extract the style information of the feature map; The noise prediction module is used to input the text encoding obtained by the text encoder, the randomly sampled noise image, and the style information into a pre-built noise prediction network with a style guidance module to perform noise prediction and obtain predicted noise; a noise removal module, configured to perform repeated denoising operations on the noisy image using the predicted noise to obtain a latent space image; The image decoding module decodes the latent space image through the SD image decoder to obtain the final style personalized image.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images