Two-stage image generation method based on contour and text guidance

Abstract

The application discloses a two-stage image generation method based on contour and text guidance. First, based on the similarity of text features and image features, an original image is selected from a gallery. Edge detection is performed on the original image to generate a contour map. A diffusion model is constructed, and two-stage training is performed. In the first stage, network training is performed using text encoding. In the second stage, network training is performed using image encoding, thereby reducing the uncertainty of the generated image, improving consistency, and accelerating the training speed of the model. Finally, the trained network is used for noise prediction, the hidden space representation of the image is generated, and is converted into the pixel space to output the expected image. The method can have various applications in smart homes, provide users with more intuitive and personalized interactive experiences, improve the quality of life, and has extremely high practical value.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and relates to an image generation method, in particular to a two-stage image generation method based on contour and text guidance. BACKGROUND

[0002] Image generation technology refers to generating new images through computer algorithms and models. In addition to providing artistic value in artistic creation, visual effects, virtual reality, product design and other fields, it also has a positive role in assisting medical diagnosis.

[0003] In smart home, image generation technology can provide users with more intuitive and personalized interaction experience, including the following aspects:

[0004] (1) Custom home design and decoration: By describing the details of the dream home design, such as color, furniture style, decoration, and even space layout and lighting effect, image generation technology can generate virtual images based on these textual descriptions, accurately restore users' ideas, help users visualize design concepts, and let users understand the final presentation effect of the intended decoration style, providing a reference for determining the actual decoration plan.

[0005] (2) Personalized home security simulation: Through user-provided descriptions, image generation technology can draw a home floor plan and simulate the layout of security devices, including the location of security cameras, door and window sensors, and intelligent alarm devices, to visually display areas with security risks or the need for enhanced security, thereby improving home security.

[0006] (3) Virtual art and home decoration: Users can describe the type of art they need, such as oil paintings, sculptures, photography works, or even specific artistic styles or themes. Image generation technology can generate images of various art and decorative items based on the description, allowing users to preview the decoration effect in a virtual space, helping users choose appropriate decorative elements, and thus create a unique and personalized home atmosphere. It can also generate related home decorations, including furniture styles, color combinations and decorative item placement, to show users the visual effects of different decoration schemes, helping users make better decisions on home arrangement, and thus create an ideal home environment.

[0007] LDM (Latent Diffusion Models) generates images by iteratively refining the original noisy data in a high-dimensional representation space, and then decodes the representation results into complex and fine images, which can significantly reduce the computational complexity of diffusion models. Although good results have been achieved in the field of text-guided image generation, on the one hand, the quality of the keywords of the guiding text can change the results of the text encoding, thereby significantly affecting the quality of image generation, and the text-guided method has certain randomness and uncontrollability. On the other hand, although LDM reduces the computational complexity and speeds up the convergence, it still needs to be trained for a long time to get a good result, and improving the convergence speed is still a problem to be solved. SUMMARY

[0008] In view of the shortcomings of the prior art, the present application proposes a two-stage image generation method based on contour and text guidance, which takes contour and text as input and constrains the generated image from different modalities to reduce the randomness and uncontrollability of the generated image and improve consistency. And use the two-stage method to reduce the size of the hidden space representation and speed up the training speed of the model.

[0009] The two-stage image generation method based on contour and text guidance has the following specific steps:

[0010] Step 1, select the original image

[0011] Use the pre-trained text encoder and image encoder to encode the input text and gallery image respectively, calculate the cosine similarity between the text vector and the image vector, and select the gallery image with the highest cosine similarity with the text vector as the original image.

[0012] Step 2, edge detection

[0013] Based on the scene where the original image is located, set the edge detection algorithm, obtain the main contour of the image, and generate a mask image with only grayscale information as the contour image.

[0014] Step 3, text encoding

[0015] Use the pre-trained text encoder to encode the input text, and send the result to the mapping network to output the text vector.

[0016] Step 4, image encoding

[0017] Use the pre-trained CLIP model image encoder to encode the contour image obtained in step 2, and send the result to the mapping network to output the image vector.

[0018] Step 5, guided image generation

[0019] The diffusion model supporting external input conditions is constructed Latent Diffusion, the total time step T and the truncation time step t are set, the text guide is used in the training of the 0-t time step, and the contour guide is used in the training of the t-T time step, and the specific steps are as follows:

[0020] s5.1, the original image selected in step 1 is encoded by using the VAE encoder to obtain the latent space representation.

[0021] s5.2, random noise is added to the latent space representation obtained by encoding in s5.1, the time step range is set to 0-t, and the text vector obtained in step three is input into the Unet1 network for training iteration.

[0022] s5.3, the latent space representation obtained by encoding in s5.1 is down-sampled and random noise is added, the time step range is set to t-T, and the image vector obtained in step four is input into the Unet2 network for training iteration.

[0023] s5.4, random sampling is performed in the Gaussian distribution, and the obtained pure noise image and the image vector obtained in step four are input into the trained Unet2 network to obtain the noise-added latent space representation After (T-t) iterations, the noise-added latent space representation is obtained After upsampling, the noise-added latent space representation is input into the trained Unet1 network together with the text vector obtained in step three to obtain the noise-added latent space representation Z' t-1 , and after t iterations, the latent space representation Z' is obtained.

[0024] s5.5, the latent space representation Z' obtained in s.4 is converted to the pixel space by the VAE decoder, and the final generated image is output.

[0025] The present application has the following beneficial effects:

[0026] The contour guide and the text guide are used as the input of the model together, the generated image can be constrained from different modalities, and the screened original image can make the contour guide condition consistent with the text description, reduce the uncertainty of the generated image, and improve the semantic consistency of the text and the image. In addition, the two-stage training method can also accelerate the convergence speed of the model. Therefore, the present method can improve the quality of the generated image while greatly shortening the training time. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a two-stage image generation method based on contour and text guide. DETAILED DESCRIPTION

[0028] The present application will be further explained and described below in conjunction with the drawings;

[0029] As Figure 1 shown, the two-stage image generation method based on contour and text guidance specifically includes the following steps:

[0030] Step 1, select the original image.

[0031] Use the text encoder and image encoder of the pre-trained CLIP model or BERT model to encode the input text and gallery image respectively, to obtain a 512-dimensional text vector x i and an image vector y i , calculate the cosine similarity cos(θ) between them:

[0032]

[0033] Select the gallery image with the highest cosine similarity with the text vector as the original image.

[0034] Step 2, edge detection.

[0035] Use Mask R-CNN to perform edge detection on the original image obtained in step 1, obtain the main contour of the image, and generate a mask image with only grayscale information as the contour image:

[0036] Step 2.1, use a convolutional neural network to obtain the feature map feature map corresponding to the original image;

[0037] Step 2.2, according to the classification number N, set N ROI ranges for each pixel point in the feature map feature map, and obtain multiple candidate ROI regions.

[0038] Step 2.3, sequentially send the candidate ROI regions into the RPN network to predict the entity bounding box range, perform binary classification, and filter out part of the candidate ROI regions.

[0039] Step 2.4, perform ROI Align operation on the retained candidate ROI regions to one-to-one correspond the pixel positions in the original image and the feature map feature map.

[0040] Step 2.5, classify the candidate ROI regions, obtain classification information, and then perform deconvolution and pooling operations through FCN to obtain the segmentation edge of the image entity, and generate the corresponding Mask generation.

[0041] Step 3, text encoding.

[0042] The input text is encoded by using the text encoder of the pre-trained CLIP model, and the result is sent into the mapping network to output a 512-dimensional text vector. The mapping network is a fully connected layer network with three layers, the first layer input layer is 512-dimensional, the second hidden layer is 1024-dimensional, and the third output layer is 512-dimensional. The mapping network refines the encoding result, so that the encoding vector can better adapt to different situations and improve the generalization of the model.

[0043] Step 4, image encoding.

[0044] The contour map obtained in step 2 is encoded by using the pre-trained CLIP model image encoder, and the result is sent into the mapping network to output a 512-dimensional image vector.

[0045] Step 5, guided image generation.

[0046] A diffusion model Latent Diffusion supporting external input conditions is constructed, and the truncation round t is set. In the training of 0-t rounds, text guidance is used, and in the training of t-T rounds, contour guidance is used.

[0047] The diffusion model includes a forward process and a reverse process. The forward process is to encode the original image, add noise to the encoded image latent space representation T times to obtain T noisy latent space representations. The reverse process is to denoise the latent space representation according to the text representation T times to obtain the denoised latent space representation, and then decode to generate an image.

[0048] The diffusion model uses the Unet network to predict the noise added by the forward process at each time step, which is used for denoising in the reverse process.

[0049] Since the method is trained in two stages, two networks Unet1 and Unet2 are used to predict the noise added at 0-t time steps and t-T time steps, respectively. The structures of Unet1 and Unet2 networks are the same and symmetric, where the first half contains 8 main encoding blocks, and the second half contains 8 main decoding blocks. Each block contains a convolutional neural network ResNet for global feature encoding / decoding and an image attention network ViT containing cross-attention and self-attention mechanisms for local feature encoding / decoding.

[0050] The loss function is set as:

[0051] L LDM :=E ∈(x),∈～N(0,1),t [||∈-∈ θ (z t ,t,τ θ (y))||2 2 ]

[0052] where ∈ θ (z t ,t,τ θ (y)) represents the model noise prediction, θ represents the model parameters, z t represents the latent space variable at time step t, τ θ (y) is used to map external conditions to a specified size, y represents external conditions. E ∈(x),∈～N(0,1),t is the mean and variance loss function.

[0053] The specific steps of guiding image generation are:

[0054] s5.1, construct a diffusion model Latent Diffusion supporting external input conditions, set the total time step T, the truncation time step t, use text guidance in 0-t rounds of training, and use contour guidance in t-T rounds of training.

[0055] s5.2, encode the original image obtained in step 1 using the VAE encoder to obtain a latent space representation with a size of 64x64x4.

[0056] s5.3, add random noise to the latent space representation obtained in s5.2, set the time step range to 0-t, and input the text vector obtained in step three into the Unet1 network for training iteration.

[0057] s5.4, add random noise to the latent space representation obtained in s5.2 after downsampling, set the time step range to t-T, and input the image vector obtained in step four into the Unet2 network for training iteration.

[0058] s5.5, randomly sample in a Gaussian distribution, input the obtained pure noise image with a size of 16x16x4 and the image vector obtained in step four into the Unet2 network to obtain a noisy latent space representation After (T-t) iterations, a noisy latent space representation is obtained, which is then upsampled and input into the Unet1 network together with the text vector obtained in step three to obtain a noisy latent space representation Z' t-1 , and after t iterations, a latent space representation Z' is obtained.

[0059] s5.6, convert the latent space representation Z' obtained in s5.5 to pixel space through the VAE decoder to output the final generated image.

[0060] The final generated image can be evaluated using the FID (Frechet Inception Distance) and accuracy evaluation indicators. FID is used to represent the diversity and quality of the generated image, and the smaller the value, the better the quality of the generated image. The calculation method is:

[0061] FID(x,g) = ||μ x -μ g || 2 + Tr(∑ x +∑ g -2(∑ x ∑ g ) 0.5 )

[0062] where μ denotes the mean, Tr denotes the trace of a matrix, ∑ denotes the covariance matrix, and x and g denote the real and generated images, respectively.

[0063] Accuracy is used to evaluate the semantic consistency between the generated image and the given text description, and the larger the value indicates the higher similarity. The evaluation process is modeled as a ranking task: given a generated image and 5 candidate texts, including the matching text and 4 randomly unmatched texts, the 5 candidate texts are ranked by calculating the cosine similarity between the text vector and the image vector. If the matching text ranks first, the generated image is considered to be semantically related to the matching text.

Claims

1. A two-stage image generation method based on contour and text guidance, characterized in that: The specific steps are as follows: Step 1: Encode the input text and the image library image using a pre-trained text encoder and image encoder respectively, and select the image library image corresponding to the image encoding with the highest similarity to the text encoding result as the original image. Step 2: Based on the scene in which the original image is located, set an edge detection algorithm to obtain the outline of the main body of the image and generate a mask image as the outline map; Step 3: Encode the input text using a pre-trained text encoder, feed the result into a mapping network, and output the text vector. Step 4: Encode the contour map obtained in Step 2 using the pre-trained CLIP model image encoder, and feed the result into the mapping network to output the image vector. Step 5: Construct a diffusion model that supports external input conditions, set the total time step T and the truncation time step t, input the text vector obtained in step 3 during the training of time steps 0 to t, input the image vector obtained in step 4 during the training of time steps t to T, and use the trained diffusion model to output the generated image.

2. The two-stage image generation method based on contour and text guidance as described in claim 1, characterized in that: The text encoder and image encoder of the pre-trained CLIP model or BERT model are used to encode the text and the image library images respectively.

3. The two-stage image generation method based on contour and text guidance as described in claim 1 or 2, characterized in that: Calculate text encoding With image encoding Cosine similarity of the included angle between : ； Where n represents the encoding length.

4. The two-stage image generation method based on contour and text guidance as described in claim 1, characterized in that: Using Mask R-CNN, edge detection is performed on the original image obtained in step 1 to obtain the main outline of the image, and a mask image with only grayscale information is generated as the outline map.

5. The two-stage image generation method based on contour and text guidance as described in claim 1 or 4, characterized in that: The specific steps of edge detection are as follows: Step 2.1: Use a convolutional neural network to obtain the feature map corresponding to the original image; Step 2.2: Based on the number of classifications N, set N ROI ranges for each pixel in the feature map to obtain multiple candidate ROI regions; Step 2.3: Feed the candidate ROI regions into the RPN network in sequence to predict the entity bounding box range, perform binary classification, and filter out some candidate ROI regions. Step 2.4: Perform ROI Align operation on the retained candidate ROI regions to match the pixel positions in the original image and the feature map one by one; Step 2.5: Perform N-classification on the candidate ROI regions to obtain classification information, and then perform deconvolution and pooling operations through FCN to obtain the segmentation edges of image entities and generate the corresponding Mask.

6. The two-stage image generation method based on contour and text guidance as described in claim 1, characterized in that: The mapping network is a fully connected layer network with three layers: the first input layer is 512-dimensional, the second hidden layer is 1024-dimensional, and the third output layer is 512-dimensional.

7. The two-stage image generation method based on contour and text guidance as described in claim 1, characterized in that: The specific steps for generating the guided image are as follows: s5.

1. Encode the original image selected in step 1 using a VAE encoder to obtain the latent space representation; s5.2 Add random noise to the latent space representation obtained by encoding in s5.1, set the time step range to 0~t, and feed it and the text vector obtained in step 3 into the Unet1 network for training iteration; s5.

3. After downsampling the latent space representation obtained by encoding in s5.1, add random noise, set the time step range to t~T, and feed it and the image vector obtained in step 4 into the Unet2 network for training iteration; s5.

4. Randomly sample from a Gaussian distribution, and feed the resulting pure noisy image and the image vector obtained in step 4 into the trained Unet2 network to obtain the noisy latent space representation. After (Tt) iterations, the noisy latent space representation is obtained. After further upsampling, the vectors obtained in step three are fed into the trained Unet1 network to obtain a noisy latent space representation. After t iterations, the latent space representation is obtained. ; s5.

5. Represent the latent space obtained in s5.4 using the VAE decoder. Convert to pixel space and output the final generated image.

8. The two-stage image generation method based on contour and text guidance as described in claim 1 or 7, characterized in that: Set the loss function for model training as follows: ; in, This indicates the model's noise prediction. Indicates model parameters, express Latent space variables at time step, Used to map external conditions to a specified size. Indicates external conditions; Let be the sum and variance loss function.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1, 2, 4, 6 or 7.

Images