A novel text and semantic segmentation map co-driven image generation method
Patent Information
- Application Number
- CN202311209514.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-19
- Publication Date
- 2026-08-18
- Estimated Expiration
- 2043-09-19
AI Technical Summary
然而，这些视觉布局通常具有有限的类别级别标签，例如眼睛、鼻子、头发和眼镜，这可能会限制模型生成细粒度面部细节的能力
[0063] This invention provides a knowledge-driven adversarial generative network for generating faces based on semantic segmentation maps and text. This network aims to gradually optimize the image generation quality by utilizing prior knowledge of images and text. Through the integration of prior knowledge of text features, Fourier features, semantic segmentation maps, and coordinate embedding, the generated images are realistic and reasonable, while the generation results can be highly controlled.
Smart Images
Figure CN117557683B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image generation technology, and in particular relates to a knowledge-driven text-to-image generation method that uses prior knowledge as a driving force for multi-stage stepwise optimization. Background Technology
[0002] Face generation has become a growing area of interest in computer vision and multimedia due to its wide range of potential applications, including portrait simulation, security, and entertainment. Recent advances in image generation techniques, such as Generative Adversarial Networks (GANs) and diffusion models, have led to rapid development in the field of unconditional face image generation. Despite significant progress, controllability remains a major challenge in face generation. To address this, many researchers have turned to controllable generation using natural language descriptions, also known as text-to-face (T2F) generation. In T2F, the text description, as a semantic condition, is directly mapped to the output image in an end-to-end cross-modal transformation. Recent advances have also explored modifying the underlying code of pre-trained unconditional generative models to achieve the desired output. While existing methods can guarantee approximate alignment between text and image, descriptions remain prone to subjectivity and ambiguity, making it difficult to generate images as expected. Furthermore, verbose and detailed descriptions not only consume more human effort but also hinder image-text matching in existing methods.
[0003] Existing approaches have attempted to use visual layouts, such as sketches, segmentation maps, and scene maps, for conditional image generation. These methods treat image generation as a transformation task, converting an input layout image into the desired image. Given visual priors help reduce the difficulty of image generation and produce more believable results. For example, in face generation, segmentation masks can specify the location and shape of various facial features, making it easier to synthesize the desired portrait. However, these visual layouts typically have limited category-level labels, such as eyes, nose, hair, and glasses, which can limit the model's ability to generate fine-grained facial details. While segmentation masks can provide the shape of hair, they cannot reflect hair color. Adding labels to all possible attributes significantly increases the annotation burden. Summary of the Invention
[0004] The purpose of this invention is to provide a novel image generation method driven by both text and semantic segmentation maps, combining two types of conditions to achieve highly controllable face generation and manipulation. The segmentation map, as a visual prior, determines the coarse outline of the target face, including the layout, shape, and size of facial features and attachments. Compared to other conditions, it has lower labor intensity. Furthermore, text descriptions are used as open-ended inputs to present attributes and details of the target portrait, such as skin and hair color, beard, gender, and even age. The goal of this invention is to synthesize easily manipulated face images by editing text and segmentation map inputs, achieving highly controllable face generation.
[0005] This invention includes the following steps:
[0006] 1) Input a language description T, a semantic segmentation map M, random noise z, and coordinate embedding features E with the same resolution as the image. coor and Fourier feature E fo ;
[0007] 2) Extracting word features E of the language text ;
[0008] 3) Extract visual features E from the semantic segmentation map mask ;
[0009] 4) Calculate word features E text With visual feature E mask The text-image correlation matrix C;
[0010] 5) Pass the text-image correlation matrix through a multilayer perceptron to obtain the gated text-image correlation matrix C. gate ;
[0011] 6) Gated text-image correlation matrix C gate Perform a positional multiplication with the text-image correlation matrix C, and then multiply with the word features E. text Perform matrix multiplication and obtain cross-modal control information E through a fully connected network layer. render ;
[0012] 7) The cross-modal control information is passed through a feedforward neural network and used as a residual. This residual is then added to the original cross-modal control information to obtain the final cross-modal control feature embedding E. render ;
[0013] 8) After concatenating the Fourier features with the coordinate embedding, concatenate them with the cross-modal control feature embedding and the visual features of the semantic segmentation map to obtain the pixel embedding E;
[0014] 9) Perform pixel folding operation on the pixel embedding to obtain low-dimensional pixel embedding;
[0015] 10) Low-dimensional pixels are embedded and processed through a convolutional module to obtain low-dimensional image features;
[0016] 11) Obtain the current stage image by performing inverse pixel folding operation on the low-dimensional image features;
[0017] 12) Add up the pixels of the images from different stages to obtain the final generated image;
[0018] 13) Evaluate the consistency between the generated image, the input semantic segmentation map, and the input text.
[0019] In step 1), the maximum length of the language description is set to 30, the noise dimension is set to 256, and the coordinate embedding is... Where H is the height of the generated image, W is the width of the generated image, d is the dimension, and the coordinate embedding is randomly generated; Fourier features Where H is the height of the generated image, W is the width of the generated image, d is the dimension, and the Fourier features are obtained by linear projection of the sin function. (Semantic segmentation map) Where H is the height of the generated image and W is the width of the generated image.
[0020] In step 2), the pre-trained CLIP model's text encoder is used to extract word features of the language. Where N is the number of words, and d is the dimension of the feature. t =512.
[0021] In step 3), a shallow convolutional layer is used to extract visual features from the semantic segmentation map. Where H is the height of the generated image, W is the width of the generated image, and the dimension d is 128.
[0022] In step 4), the process of obtaining the text-image similarity feature matrix C is as follows:
[0023]
[0024] in, and Let b be the projection matrix. m and b t This is the corresponding deviation.
[0025] In step 5), the gated text-image correlation matrix C gate The acquisition process is as follows:
[0026] C gate =sigmoid(MLP(C)+b gate )
[0027] MLP stands for Multilayer Perceptron.
[0028] In step 6), the cross-modal control information E render The acquisition process is as follows:
[0029] E render =MLP[(C gate ⊙C)(W′ t E text +b′ t )]
[0030] Where ⊙ represents the dot product at the corresponding position, W′t and b′ t These are the weights and biases of the projection matrix; MLP stands for Multilayer Perceptron.
[0031] In step 7), the final cross-modal control feature embedding E render The acquisition process is as follows:
[0032] E render =E render +FFN(E render ).
[0033] FFN is a feedforward network with residual connections.
[0034] In step 8), the process of obtaining pixel embedding E is as follows:
[0035] E = E fo ||E coor ||E mask ||E render
[0036] Among them, E fo It is a Fourier feature, E coor It is a coordinate feature, E mask It is a visual feature of the semantic segmentation map, E render It is cross-modal control information.
[0037] In step 9), the pixel folding operation is as follows:
[0038] For each generation stage, its pixel embedding is:
[0039]
[0040] Pixel folding operation will reduce E i Process and project as follows:
[0041]
[0042] Where c is set to 32;
[0043] E′ is obtained after pixel folding operation. i :
[0044]
[0045] In step 10), the convolutional module consists of two ModConv (modulation convolutional layer) layers. The Adam optimizer can be used during the training of the convolutional module, with the initial learning rate and batch size set to 0.0002 and 32, respectively.
[0046] In step 11), the reverse folding operation is as follows:
[0047] For each input E′ i :
[0048]
[0049] E″ is obtained after inverse pixel folding operation. i :
[0050]
[0051] Then, E″ i Projecting onto the RGB space yields the generated image I. i ;
[0052]
[0053] In step 12), the pixel tensors of different layers are combined. i This further improves the connectivity of multi-stage generation;
[0054]
[0055] In step 13), the evaluation metrics include LPIPS, FID, CLIP-SIM, and R@Top-k;
[0056] The process of using metrics to evaluate image consistency is as follows:
[0057] Step 13-1: Use the trained ALEX model to extract features from the generated image and the real image, and then use the formula to calculate LPIPS:
[0058]
[0059] Step 13-2: Use InceptionV3 to extract the mean and variance of features from the real image and the generated image respectively, and then calculate the difference between the mean and variance of the two to obtain the FID;
[0060] Step 13-3: Use the CLIP model to extract global features from the input text and the generated image respectively, measure the cosine similarity between the two, and finally obtain the evaluation of text-image consistency (CLIP-SIM);
[0061] Step 13-4: Extract one hundred images as samples, including one positive sample and ninety-nine negative samples; use a text for retrieval, and use CLIP to measure the similarity between the text and the 100 images (including the generated images); finally, obtain the average accuracy of the generated images in the retrieval results, denoted as R@Top-k.
[0062] This invention provides additional prior knowledge for image generation by introducing Fourier features, coordinate features, face segmentation maps, and text features. Fourier features provide the positional information of each pixel, coordinate features enhance the model's potential for image generation, face segmentation maps provide approximate facial contour information, and text features, combined with the face segmentation map, provide fine-grained semantic information. Finally, the proposed PixelFace+ is validated through extensive experiments on the widely recognized benchmark MultiModal CelebA-HQ for conditional face generation and manipulation. Experimental results show that the proposed PixelFace+ outperforms previous text-guided face generation and editing methods in terms of image quality and image-text similarity. Furthermore, PixelFace+ can achieve more accurate face generation and manipulation by modifying text or mask inputs.
[0063] This invention provides a knowledge-driven adversarial generative network for generating faces based on semantic segmentation maps and text. This network aims to gradually optimize the image generation quality by utilizing prior knowledge of images and text. Through the integration of prior knowledge of text features, Fourier features, semantic segmentation maps, and coordinate embedding, the generated images are realistic and reasonable, while the generation results can be highly controlled. Attached Figure Description
[0064] Figure 1 This is an architecture diagram of the PixelFace+ network of this invention. Detailed Implementation
[0065] The following embodiments will describe in detail the technical solution and beneficial effects of the present invention with reference to the accompanying drawings.
[0066] This invention proposes PixelFace+, a novel cross-modal pixel synthesis network for highly controllable face generation and manipulation. Specifically, PixelFace+ utilizes a pre-trained language encoder with CLIP to extract semantic information from the input text description, then converts the obtained word embeddings into text prior tensors via a proposed gated cross-modal attention module. For visual information, shallow convolutional layers extract visual prior features from the input segmentation mask, and these textual and visual priors are then input into a progressive pixel synthesis generator to produce the final face image. During training, an adversarial loss objective is used to ensure controllability by learning a triple consistency between text, mask, and generated image. Furthermore, clip-based regularization is implemented to align visual-language features. After training, PixelFace+ can support highly controllable face generation and manipulation with single or two conditional inputs.
[0067] The embodiments of the present invention include the following steps:
[0068] Step 1: Input a language description, with a maximum length of 30, a noise dimension of 256, and coordinate embedding. Where H is the height of the generated image, W is the width of the generated image, d is the dimension, and the coordinate embedding is randomly generated. Fourier features Where H is the height of the generated image, W is the width of the generated image, d is the dimension, the Fourier features are obtained by linear projection of the sin function, and the semantic segmentation map is... Where H is the height of the generated image and W is the width of the generated image.
[0069] Step 2: Use the pre-trained CLIP model's text encoder to extract word features of the language. Where N is the number of words, and d is the dimension of the feature. t =512.
[0070] Step 3: Extract visual features from the semantic segmentation map using a shallow convolutional layer. Where H is the height of the generated image, W is the width of the generated image, and the dimension d is 128.
[0071] Step 4: The process of obtaining the text-image similarity feature matrix C is as follows:
[0072]
[0073] in, and Let b be the projection matrix. m and b t This is the corresponding deviation.
[0074] Step 5: Gated text-image correlation matrix C gate The acquisition process is as follows:
[0075] C gate =sigmoid(MLP(C)+b gate )
[0076] MLP stands for Multilayer Perceptron.
[0077] Step 6: Cross-modal control information E render The acquisition process is as follows:
[0078] E render =MLP[(C gate ⊙C)(W′ t E text +b′ t )]
[0079] Where ⊙ represents the dot product at the corresponding position, W′ t and b′ tThese are the weights and biases of the projection matrix; MLP stands for Multilayer Perceptron.
[0080] Step 7: Final cross-modal control feature embedding E render The acquisition process is as follows:
[0081] E render +=FFN(E render ).
[0082] FFN is a feedforward network with residual connections.
[0083] Step 8: The process of obtaining pixel embedding E is as follows:
[0084] E = E fo ||E coor ||E mask ||E render
[0085] Among them, E fo It is a Fourier feature, E coor It is a coordinate feature, E mask It is a visual feature of the semantic segmentation map, E render It is cross-modal control information.
[0086] Step 9: Perform pixel folding on the pixel embedding to obtain low-dimensional pixel embeddings. The pixel folding operation is as follows:
[0087] For each generation stage, its pixel embedding is:
[0088]
[0089] Pixel folding operation will reduce E i Process and project as follows:
[0090]
[0091] Here, c is set to 32.
[0092] E′ is obtained after pixel folding operation. i :
[0093]
[0094] Step 10: Embed the low-dimensional pixels through two ModConv layers to obtain pixel features. The Adam optimizer can be used during the training of the convolutional module, with the initial learning rate and batch size set to 0.0002 and 32, respectively.
[0095] Step 11: Obtain the current stage image by performing an inverse pixel folding operation on the low-dimensional image features. The inverse folding operation is as follows:
[0096] For each input E′ i :
[0097]
[0098] E″ is obtained after inverse pixel folding operation. i :
[0099]
[0100] Then, E″ i Projecting onto the RGB space yields the generated image I. i .
[0101]
[0102] Step 12: Sum the individual pixels of the images from different stages to obtain the final generated image. The operation of combining the pixel tensors of different layers is as follows:
[0103]
[0104] Among them, I i The image generated for the i-th layer.
[0105] Step 13: Evaluate the generated image. The evaluation metrics include LPIPS, FID, CLIP-SIM, and R@Top-k.
[0106] The specific process is as follows:
[0107] Step 13-1: Use the trained ALEX model to extract features from the generated image and the real image, and then use the formula to calculate LPIPS:
[0108]
[0109] Step 13-2: Extract the mean and variance of features from the real image and the generated image using InceptionV3, and then calculate the difference between the mean and variance of the two to obtain the FID.
[0110] Step 13-3: Use the CLIP model to extract global features from the input text and the generated image respectively, measure the cosine similarity between the two, and finally obtain the evaluation of text-image consistency (CLIP-SIM).
[0111] Step 13-4: Extract one hundred images as samples, including one positive sample and ninety-nine negative samples. Perform a retrieval using text, and use CLIP to measure the similarity between the text and the 100 images (including generated images). The final result is the average accuracy of the generated images evaluated in the retrieval results, denoted as R@Top-k.
[0112] like Figure 1 As shown, this invention incorporates prior knowledge such as text, coordinate information, semantic segmentation maps, and Fourier features during the generation process, making the generated images more realistic and consistent with common sense. This is analogous to humans possessing a certain amount of knowledge storage before creating a painting, thus producing an image that more closely reflects reality. Furthermore, through the joint control of semantic segmentation maps and text, a high degree of control over the generation result is achieved. The implementation of this invention includes the following steps:
[0113] I. Model Implementation Process
[0114] 1.1 Model Input:
[0115] like Figure 1 The diagram shows the network architecture of the model. The model's input consists of a language description, random noise, coordinate embedding features (with the same resolution as the image), Fourier features, and a semantic segmentation map. The maximum length of the language description is set to 30, the noise dimension is set to 256, and the coordinate embedding... Where H is the height of the generated image, W is the width of the generated image, d is the dimension, and the coordinate embedding is randomly generated. Fourier features Semantic segmentation graph Where H is the height of the generated image, W is the width of the generated image, d is the dimension, and the Fourier features are obtained by linear projection of the sin function.
[0116] 1.2 Language Feature Encoder:
[0117] For the input language description, CLIP's text encoder extracts global language features. Where N is the number of input text entries, and d is the dimension of the global feature. t =512.
[0118] 1.3 Generation of cross-modal control features:
[0119] Extracting visual features from semantic segmentation maps using a shallow convolutional layer Where H is the height of the generated image, W is the width of the generated image, and the dimension d is 128.
[0120] The process of obtaining the text-image similarity feature matrix C is as follows:
[0121]
[0122] in, and Let b be the projection matrix. m and b t This is the corresponding deviation.
[0123] Gated text-image correlation matrix C gate The acquisition process is as follows:
[0124] C gate =sigmoid(MLP(C)+b gate )
[0125] MLP stands for Multilayer Perceptron.
[0126] Cross-modal control information E render The acquisition process is as follows:
[0127] E render =MLP[(C gate ⊙C)(W′ t E text +b′ t )]
[0128] Where ⊙ represents the dot product at the corresponding position, w′ t and b′ t These are the weights and biases of the projection matrix; MLP stands for Multilayer Perceptron.
[0129] Cross-modal control feature embedding E render The acquisition process is as follows:
[0130] E render +=FFN(E render ).
[0131] FFN is a feedforward network with residual connections.
[0132] 1.4-pixel embedding generation:
[0133] After adding the Fourier features to the coordinate embedding, it is concatenated with the dynamic knowledge embedding to obtain the pixel embedding. The process of obtaining the pixel embedding is as follows:
[0134] E = E fo ||E coor ||E mask ||E render
[0135] Among them, E fo It is a Fourier feature, E coor It is a coordinate feature, E mask It is a visual feature of the semantic segmentation map, E renderIt is cross-modal control information.
[0136] 1.5 Single-stage image generation:
[0137] Pixel embedding is then subjected to pixel folding to obtain low-dimensional pixel embeddings. The pixel folding operation is as follows:
[0138] For each generation stage, its pixel embedding is:
[0139]
[0140] Pixel folding operation will reduce E i Process and project as follows:
[0141]
[0142] Here, c is set to 32.
[0143] E′ is obtained after pixel folding operation. i :
[0144]
[0145] Low-dimensional pixels are embedded through two ModConv (modulation convolutional layers) to obtain pixel features. The Adam optimizer can be used during the training of the convolutional module, with an initial learning rate of 0.0002 and a batch size of 32.
[0146] The low-dimensional image features are used to obtain the current stage image through an inverse pixel folding operation. The inverse folding operation is as follows:
[0147] For each input E′ i :
[0148]
[0149] E″ is obtained after inverse pixel folding operation. i :
[0150]
[0151] Then, E″ i Projecting onto the RGB space yields the generated image I for the i-th stage. i .
[0152]
[0153] 1.6 Final Image Generation:
[0154] The final image is obtained by summing the individual pixels of the images from different stages. The operation of combining the pixel tensors of different layers is as follows:
[0155]
[0156] Among them, I i The image generated for the i-th layer.
[0157] 1.7 Discriminator:
[0158] For the final stage, a discriminator is set up to determine whether the image generated in this stage is real or fake. Here, a CNN-based encoder similar to PixelFolder is used to calculate the score for whether the image generated in this stage is real. This score is used to calculate the loss function in the subsequent process.
[0159] II. Model Training Process
[0160] 2.1 Calculation of the loss function:
[0161] model loss function It consists of four parts, which calculate the loss of the discriminator, generator, R1 regularization, and regularized path respectively.
[0162]
[0163] Where D and G represent the discriminator and generator, respectively. and The corresponding adversarial loss is:
[0164]
[0165]
[0166] Where I represents a real image. To generate an image, M is the semantic segmentation map, and T is the input text description.
[0167] The R1 regularization loss is used (see Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. 2018. Which training methods for GANs do actually converge. In International conference on machine learning. PMLR, 3481-3490.). The path loss is used to guide the noise Z to be better decoupled into the W space (see Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 8110-8119.).
[0168] To further achieve semantic alignment between the generated images and the given text descriptions, this invention introduces CLIP as a weak supervisor to regularize T2F generation. For each x~p in the mini-batch processing... data This invention randomly samples a text t from the training data. Then, it uses CLIP's visual and language encoders to extract the expression of the synthesized image and the input text t. Since CLIP is a well-trained VL model, the features extracted by its encoder can objectively reflect the generated image. The semantic distance between t and t. In this case, a contrastive loss is further introduced to optimize T2F semantic matching.
[0169]
[0170]
[0171] Where sim(x,y) represents the cosine similarity between x and y. ψ(·) and These represent the visual and speech encoders of CLIP, respectively, used to extract average features from images and text.
[0172] 2.2 Model training parameter settings:
[0173] During training, the Adam optimizer was used with a learning rate of 0.0002 and hyperparameters of β1 = 0.5 and β2 = 0.999.
[0174] 2.3 Model Training:
[0175] For model training, the input image and language are first processed through step one to obtain the model's output. Then, the model's loss function is calculated through step 2.1, and the model parameters are updated using backpropagation of the gradient of the loss function. The training and iteration process is set up as described in step 2.2, and finally the training of the entire model is completed.
[0176] III. Model Evaluation:
[0177] LPIPS and FID are two widely used metrics in T2I, both designed to evaluate the sharpness and diversity of synthesized images. The formulas for these metrics are as follows:
[0178]
[0179] FID = ||μ r -μ g || 2 +Tr(∑ r +∑ g -2(∑ r ∑ g ) 1 / 2
[0180] Where, μ r and μ g ∑ are the mean values of the real image and the generated image, respectively. r and ∑ g These are the covariance matrices of the real image and the generated image, respectively. The lower the FID, the closer the generated image is to the real image at the feature layer.
[0181] In addition, another pseudo-expert method for text-to-image matching, called SIM, is provided. It directly measures the correlation between the outputs of different T2I methods and the same input. For an image synthesized by the T2I method, SIM evaluates the features f of the input text. t Features f of the output image v The correlation between them. Here, we use cosine similarity to define the correlation, that is:
[0182]
[0183] IV. Model Deployment Process:
[0184] After model training in step two, the model can be trained by taking the input language and processing it through step one to obtain the model's output as the corresponding generated result. That is, when a certain language description is input, the model outputs the image that best matches the description.
[0185] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.
Claims
1. A novel image generation method driven by both text and semantic segmentation graphs, characterized in that... Includes the following steps: 1) Input a language description T, a semantic segmentation map M, random noise z, and coordinate embedding features E with the same resolution as the image. coor and Fourier feature E fo ; 2) Extracting word features E of the language text ; 3) Extract visual features E from the semantic segmentation map mask ; 4) Calculate word features E text With visual feature E mask The text-image correlation matrix C; 5) Pass the text-image correlation matrix through a multilayer perceptron to obtain the gated text-image correlation matrix C. gate ; 6) Gated text-image correlation matrix C gate Perform a positional multiplication with the text-image correlation matrix C, and then multiply with the word features E. text Perform matrix multiplication and obtain cross-modal control information E through a fully connected network layer. render ; 7) The cross-modal control information is passed through a feedforward neural network and used as a residual. This residual is then added to the original cross-modal control information to obtain the final cross-modal control feature embedding E. render ; 8) After concatenating the Fourier features with the coordinate embedding, concatenate them with the cross-modal control feature embedding and the visual features of the semantic segmentation map to obtain the pixel embedding E; 9) Perform pixel folding operation on the pixel embedding to obtain low-dimensional pixel embedding; 10) Low-dimensional pixels are embedded and processed through a convolutional module to obtain low-dimensional image features; 11) Obtain the current stage image by performing inverse pixel folding operation on the low-dimensional image features; 12) Add up the pixels of the images from different stages to obtain the final generated image; 13) Evaluate the consistency between the generated image, the input semantic segmentation map, and the input text.
2. The novel image generation method driven by both text and semantic segmentation graphs as described in claim 1, characterized in that... In step 2), the word features E of the extracted language are... text It uses a pre-trained CLIP model text encoder to extract word features of a language. Where N is the number of words, and d is the dimension of the feature. t =512; In step 3), the visual features E of the semantic segmentation map are extracted. mask It uses a shallow convolutional layer to extract visual features from the semantic segmentation map. Where H is the height of the generated image, W is the width of the generated image, and the dimension d is 128.
3. The novel image generation method driven by both text and semantic segmentation graphs as described in claim 1, characterized in that... In step 4), the process of obtaining the text-image correlation matrix C is as follows: in, and Let b be the projection matrix. m and b t This is the corresponding deviation; In step 5), the gated text-image correlation matrix C gate The acquisition process is as follows: C gate =sigmoid(MLP(C)+b gate ) MLP stands for Multilayer Perceptron.
4. The novel image generation method driven by both text and semantic segmentation graphs as described in claim 1, characterized in that... In step 6), the cross-modal control information E render The acquisition process is as follows: E render =MLP[(C gate ☉C)(W′ t E text +b′ t )] Where ⊙ represents the dot product at the corresponding position, W′ t and b′ t These are the weights and biases of the projection matrix; MLP stands for Multilayer Perceptron.
5. The novel image generation method driven by both text and semantic segmentation graphs as described in claim 1, characterized in that... In step 7), the final cross-modal control feature embedding E render The acquisition process is as follows: E render =E render +FFN (E render ) FFN is a feedforward network with residual connections.
6. The novel image generation method driven by both text and semantic segmentation graphs as described in claim 1, characterized in that... In step 8), the process of obtaining the pixel embedding E is as follows: E=E fo ||And coor ||And mask ||And render Among them, E fo It is a Fourier feature, E coor It is a coordinate feature, E mask It is a visual feature of the semantic segmentation map, E render It is cross-modal control information.
7. The novel image generation method driven by both text and semantic segmentation graphs as described in claim 1, characterized in that... In step 9), the pixel folding operation is as follows: For each generation stage, its pixel embedding is: Pixel folding operation will reduce E i Process and project as follows: Where c is set to 32; E′ is obtained after pixel folding operation. i :
8. The novel image generation method driven by both text and semantic segmentation graphs as described in claim 1, characterized in that... In step 10), the convolutional module consists of two ModConv layers. The Adam optimizer can be used during the training of the convolutional module, with the initial learning rate and batch size set to 0.0002 and 32, respectively.
9. The novel image generation method driven by both text and semantic segmentation graphs as described in claim 1, characterized in that... In step 11), the inverse pixel folding operation is as follows: For each input E′ i : E″ is obtained after inverse pixel folding operation. i : Then, E″ i Projecting onto the RGB space yields the generated image I. i ; 10. The novel image generation method driven by both text and semantic segmentation graphs as described in claim 1, characterized in that... In step 13), the evaluation metrics include LPIPS, FID, SIM, and R@Top-k; The consistency evaluation process is as follows: Step 13-1: Use the trained ALEX model to extract features from the generated and real images, and then use this formula to calculate LPIPS: Step 13-2: Use InceptionV3 to extract the mean and variance of features from the real image and the generated image respectively, calculate the difference between the mean and variance of the two, and obtain the FID; Step 13-3: Use the CLIP model to extract global features from the input text and the generated image respectively, measure the cosine similarity between the two, and obtain the evaluation of text-image consistency CLIP-SIM; Step 13-4: Extract one hundred images as samples, including one positive sample and ninety-nine negative samples; use a text for retrieval, and use CLIP to measure the similarity between the text and the 100 images; finally, obtain the average accuracy of the generated images in the retrieval results, denoted as R@Top-k.