A multi-path parallel text-to-image generation method and system

Abstract

The present application relates to a kind of multi-path parallel text-to-image generation method and system.The method comprises the following steps:1. text cross-modal semantic representation is extracted using pre-training model, and is input to recurrent neural network, and according to the number of branch in image generation model, the same length text condition vector sequence is output.2. different text condition vectors are input into different branch generation network module, and corresponding image is generated according to different generation strategy.3. according to the depth information output by image space depth prediction model, different weights are given to the generated images of different branches, and the generated results of different branches are fused.4. in the training stage, the image generation model, space depth prediction model is realized based on the model training of the discrimination model by constructing discrimination model. The present application learns different generation parameters and strategies according to different image content characteristics, which can effectively reduce the local distortion and deformation in the image results of text-to-image generation method.

Description

Technical Field

[0001] This invention relates to the field of image generation, and more specifically to a multi-parallel text-to-image generation method and system. Background Technology

[0002] Text-to-image generation enables computers to automatically generate semantically consistent, realistic, and logically consistent visual image content from scratch based on a user-provided natural language text description (speech recognition, image OCR, text input, etc.).

[0003] In existing technologies, some methods obtain visual content related to text descriptions through retrieval techniques and then generate visual content through combination and splicing. For example, Wang et al. proposed Write-A-Video technology (Miao Wang, et al., Write-A-Video: Computational Video Montage from Themed Text, ACM Transactions on Graphics, 2019), which can search for candidate video shots that match the text description based on user input and automatically combine and edit the video. This type of retrieval-based generation technology mainly retrieves similar existing visual content by analyzing the similarity between the text description and existing visual content, making it difficult to meet users' personalized and diverse needs. Another type of method maps text information into image space by designing different generative network structures to achieve text-to-image generation. For example, Reed et al. (Scott E. Reed, et al., Learning What and Where to Draw, Annual Conference on Neural Information Processing Systems, 2016) proposed a generative adversarial network method based on text manifold interpolation and image-text matching. This method uses text semantic vectors as input to the generative model and employs image-text matching constraints to ensure semantic consistency between the text and the generated image, thus achieving text-to-image generation. Zhang et al. (Zizhao Zhang, et al., Photographic Text-to-Image Synthesis With a Hierarchically-Nested Adversarial Network, IEEE Conference on Computer Vision and Pattern Recognition, 2018) proposed a hierarchical generative adversarial network that can extend generated low-resolution images to high-resolution images.

[0004] However, none of the above methods take into account the nature of the generated content itself. They use a uniform network structure and parameters to generate different image content, which leads to the generation model failing to effectively model the content of some image regions. As a result, the generated images contain local distortions and other issues. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a multi-parallel text-to-image generation method. By using a parallel generation structure, different generation parameters and strategies are learned for different image content characteristics, which can effectively reduce local distortion and deformation in the generated results.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A multi-path parallel text-to-image generation method includes the following steps:

[0008] (1) Use a pre-trained text-image association model to extract cross-modal semantic features of text, and decouple the cross-modal semantic features of text through a recurrent neural network to obtain a text conditional vector sequence;

[0009] (2) Input different text condition vectors into the different branches of the image generation model to enable the different branches of the generation network modules to generate images according to different generation strategies.

[0010] (3) Use the spatial depth prediction model to predict the spatial depth information of the generated image, assign corresponding weights to the images generated by the different branches of the generative network module, and merge the images generated by the different branches of the generative network module into one image.

[0011] Furthermore, in the above method, during the training phase, a discriminative model is constructed to achieve adversarial training: by distinguishing between images generated by the image generation model and real paired images, the generation quality of the image generation model is improved; by distinguishing between the spatial depth information of the images predicted by the spatial depth prediction model and the spatial depth information of the extracted real images, the accuracy of the spatial depth prediction model is improved.

[0012] Furthermore, in the above method, the cross-modal semantic features of the text in step (1) are specifically the intermediate features of the text encoding module of a pre-trained text-image association model (e.g., CLIP). A deep model based on recurrent neural networks is used to input the cross-modal semantic features of the text into the recurrent neural network, and output the same number of text condition vectors according to the number of branches in the image generation model.

[0013] Furthermore, in the above method, the generative network modules of different branches in the image generation model of step (2) adopt a multi-level graph generation model based on StyleGAN. Using the corresponding text conditional vectors and the image results generated in the previous stage as input, images are generated progressively from low resolution to high resolution. The image pixels generated by the multi-level graph generation model are summed to obtain the final image:

[0014] imgk = Upsample(imgk) -1 )+RGBk

[0015] Where imgk represents the image generated at level k, RGBk represents the content generated at the current stage, and Upsample represents the upsampling operation.

[0016] Furthermore, in the above method, step (3) uses a spatial depth prediction model to predict the spatial depth information of the generated image, and then uses the predicted spatial depth information to fuse the images generated by the different branches of the generative network module into a single image. The pixel information of the k-th level image generated by the different branches of the generative network module is then used. Where n is the number of branches; its fusion method can be formulated as follows:

[0017]

[0018] Where dhk represents the spatial depth information of the k-th level image generated by the spatial depth prediction model, FC* is the weight mapping network, and Cov2D is the convolution operation used to achieve image pixel fusion.

[0019] Furthermore, in the above method, during the training phase, for the image generation model, the discriminative model aims to distinguish the results of the image generation model from the real image-text pairs, and its loss function is:

[0020]

[0021] The first term is the unconditional vector loss function, which aims to evaluate the generated image quality. The second term is the text-conditional vector loss function, which aims to evaluate the semantic consistency between the image and the text. Di represents the image discriminator, Dt represents the text-conditional vector-based image discriminator, Isa represents the text-conditional vector, x represents the image sample, and Ex... ～R eal represents the expectation when the image sample originates from a real image, Ex ～G This indicates that the image sample originates from the expectation when generating the image.

[0022] Furthermore, in the above method, during the training phase, for the spatial depth prediction model, the discriminant model aims to distinguish the prediction results of the spatial depth prediction model from the spatial depth information extracted from the real image, and its loss function is:

[0023]

[0024] Where x represents the image, depth represents the spatial depth prediction model, Ddep represents the image depth discriminator, and GT represents the depth information extracted from the image.

[0025] Furthermore, this invention proposes a multi-path parallel text-to-image generation system, which includes:

[0026] The text feature extraction module is responsible for extracting cross-modal semantic features of text using a pre-trained text-image association model and inputting them into a recurrent neural network. It outputs the same number of text condition vectors as the number of branches in the image generation model.

[0027] The multi-path image generation module is responsible for inputting different text condition vectors into the generation network modules of different branches in the image generation model, so that the generation network modules of different branches can generate images according to different generation strategies.

[0028] The image spatial depth prediction module is responsible for using the spatial depth prediction model to predict the spatial depth information of the generated image, assigning corresponding weights to the images generated by different branches of the generative network module, and fusing the images generated by different branches of the generative network module into a single image.

[0029] The advantage of this invention is that, compared with existing methods, this method takes into account the inherent differences in the nature of the generated image content and learns different generation parameters and strategies for different image content characteristics, which can effectively reduce local distortion and deformation in the image results of text-to-image generation methods.

[0030] The reason this method achieves the aforementioned effects is that it learns different generation parameters and strategies for different image content characteristics through a parallel generation structure. Furthermore, the method incorporates spatial depth information of the image to help the generation model analyze and decouple image content, providing a basis for the fusion of multi-path image generation results and further reducing the probability of local distortion and deformation during the generation process. Attached Figure Description

[0031] Figure 1 This is a flowchart of the multi-parallel text-to-image generation method of the present invention.

[0032] Figure 2This is a detailed diagram of the network structure of the present invention, where Conv 3x3 represents a convolution operation with a kernel size of 3x3, and AdaIn represents an affine transformation based on the mean and standard deviation of the image. Detailed Implementation

[0033] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0034] The multi-parallel text-to-image generation method of the present invention has the following process: Figure 1 As shown, it includes the following steps:

[0035] (1) Use a pre-trained text-image association model to extract cross-modal semantic features of the text and input them into a recurrent neural network to generate the same number of text condition vectors based on the number of branches in the generative network, i.e., the image generation model.

[0036] (2) Input the different text condition vectors in step (1) into the multi-level generative network modules of different branches, and generate corresponding images according to the different generative strategies learned.

[0037] like Figure 2 As shown, in step (2), the text conditional vector is input into a multi-level graph generation model (e.g., StyleGAN). The generated images from the previous stages and the text conditional vector are used as input to generate corresponding images, generating images step by step from low resolution to high resolution. The image pixels generated by the multi-level graph generation model are summed using residuals to obtain the final image.

[0038] imgk = Upsample(imgk) -1 )+RGBk

[0039] Where imgk represents the image generated at level k, RGBk represents the content generated at the current stage, and Upsample represents the upsampling operation.

[0040] (3) Use the spatial depth prediction network to predict the depth information of the generated image, assign corresponding weights to the generated pixels of different module branches, and fuse the generation results of different branches.

[0041] k-th level image pixel information generated by different modules The fusion result can be formalized as follows:

[0042]

[0043] Where dhk represents the spatial depth information of the k-th level image generated by the spatial depth prediction model, FC* is the weight mapping network, and Cov2D is the convolution operation used to achieve image pixel fusion.

[0044] (4) During the training phase, adversarial model training is achieved by constructing a discriminative model.

[0045] This model utilizes paired image-text data. The discriminative model improves the performance of the image generation model by distinguishing between the generated images and real images. For the generation model, the discriminative model aims to differentiate between the results of the image generation model and real images; its training loss function is:

[0046]

[0047] The first term is the unconditional vector loss function, which aims to evaluate the generated image quality. The second term is the text-conditional vector loss function, which aims to evaluate the semantic consistency between the image and the text. Di represents the image discriminator, Dt represents the text-conditional vector-based image discriminator, Isa represents the text-conditional vector, x represents the image sample, and Ex... ～R eal represents the expectation when the image sample originates from a real image, Ex ～G This indicates that the image sample originates from the expectation when generating the image.

[0048] For the spatial depth prediction model, the discriminant model improves the accuracy of the spatial depth prediction model by using the extracted spatial depth information from the real image and distinguishing the prediction results from the real image. Its training loss function is:

[0049]

[0050] Where x represents the image, depth represents the spatial depth prediction model, Ddep represents the image depth discriminator, and GT represents the depth information extracted from the image.

[0051] (5) In the generation stage, the text encoding module based on the cross-modal text-image association model extracts the representation of the user input text and generates an image that is semantically consistent with the user input text using the same method as in steps 2 and 3.

[0052] This embodiment uses the CUB dataset for experiments, which was proposed in the paper "The CALTECH-UCSD birds-200-2011dataset" (authors C. Wah et al.). We tested the following three methods for experimental comparison:

[0053] Existing Method 1: The StackGAN method in the paper "Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks" (authors Zhang H et al.).

[0054] Existing Method 2: The AttnGAN method in the paper "AttnGAN: Fine-grained text to image generation with attentional generative adversarial networks" (authors Xu T et al.).

[0055] Existing Method 3: The LAFITE method in the paper "Lafite: Towards language-free training for text-to-image generation" (authors Zhou Y et al.).

[0056] This invention: The method of this embodiment.

[0057] In evaluation metrics, IS (Intelligence Detection) is often used to represent the distinguishability of image content, with higher scores being better. For a clear image, the probability of it belonging to one class should be very high, while the probability of it belonging to other classes should be very low; the clearer the image, the greater the difference between its probability vector and the mean of all images. FID (Fixed Image ID) is often used to represent the generation quality of image content, with lower values ​​being better. For a high-quality generated image, it has a high similarity to the real image in visual features; the higher the quality of the image, the closer its statistical distribution of visual features is to the statistical distribution of visual features of the real image.

[0058] As shown in Table 1, our method improves upon the comparison methods in terms of IS and FID metrics, indicating that by learning different generation strategies, our method can effectively improve the quality of image generation. None of the three existing methods consider the properties of the generated content itself, using a uniform network structure and parameters to generate different image content. Because the generation model fails to effectively model some image regions, local distortion and warping occur in the generated images. Our invention, through a parallel generation structure, learns different generation parameters and strategies for different image content characteristics, achieving better generation results.

[0059] Table 1. Experimental results of each method on the CUB dataset

[0060]

[0061] Another embodiment of the present invention provides a multi-parallel text-to-image generation system, comprising:

[0062] The text feature extraction module is responsible for extracting cross-modal semantic features of text using a pre-trained text-image association model and inputting them into a recurrent neural network. It outputs the same number of text condition vectors as the number of branches in the image generation model.

[0063] The multi-path image generation module is responsible for inputting different text condition vectors into the generation network modules of different branches in the image generation model, so that the generation network modules of different branches can generate images according to different generation strategies.

[0064] The image spatial depth prediction module is responsible for using the spatial depth prediction model to predict the spatial depth information of the generated image, assigning corresponding weights to the images generated by different branches of the generative network module, and fusing the images generated by different branches of the generative network module into a single image.

[0065] For the specific implementation process of each module, please refer to the description of the method of the present invention above.

[0066] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0067] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0068] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A multi-path parallel text-to-image generation method, comprising the following steps: A pre-trained text-image association model is used to extract cross-modal semantic features of the text, and a recurrent neural network is used to decouple the cross-modal semantic features of the text to obtain a text conditional vector sequence. Different text condition vectors are input into the generative network modules of different branches in the image generation model, so that the generative network modules of different branches can generate images according to different generation strategies. The spatial depth information of the generated image is predicted using a spatial depth prediction model. The images generated by different branches of the generative network module are assigned corresponding weights and then merged into a single image. During the training phase, a discriminative model is constructed to achieve adversarial training: by distinguishing between images generated by the image generation model and real paired images, the generation quality of the image generation model is improved; By distinguishing between the spatial depth information of the image predicted by the spatial depth prediction model and the spatial depth information of the extracted real image, the accuracy of the spatial depth prediction model can be improved. The image generation model employs a multi-level graph generation model, with corresponding text conditional vectors and the image generated in the previous stage as input, generating images progressively from low resolution to high resolution. The pixels generated by the multi-level graph generation model are summed using residuals to obtain the final image. in, Indicates the first k The image generated at level 1 This indicates the content generated in the current stage. Indicates an upsampling operation; The spatial depth prediction model generates the first [number] for different branches of the generative network module. k Level image pixel information Its fusion method is formalized as follows: in, The first generation of spatial depth prediction model k Spatial depth information of the image, For weight mapping networks, This is a convolution operation used to achieve image pixel fusion.

2. The method as described in claim 1, characterized in that, The cross-modal semantic features of the text are specifically the intermediate features of the text encoding module in the pre-trained text-image association model. A deep model based on recurrent neural networks is adopted, and the cross-modal semantic features of the text are input into the recurrent neural network to generate a text condition vector with the same number of branches as in the image generation model.

3. The method as described in claim 1, characterized in that, During the training phase, for the image generation model, the discriminative model aims to distinguish the differences between the results of the image generation model and real images. Its loss function used for training is: The first term is the loss function based on unconditional vectors, which aims to evaluate the quality of image generation; the second term is the loss function based on text conditional vectors, which aims to evaluate the semantic consistency between the image and the text. Represents an image discriminator. This represents an image discriminator based on text conditional vectors. Represents a text condition vector. Represents image samples, This represents the expectation when the image sample originates from a real image. This indicates that the image sample originates from the expectation when generating the image.

4. The method as described in claim 1, characterized in that, During the training phase, for the spatial depth prediction model, the discriminant model aims to distinguish the prediction results of the spatial depth prediction model from the spatial depth information extracted from the real image. Its loss function is: in, Represents an image. This represents a spatial depth prediction model. Represents an image depth discriminator. This represents depth information extracted from the image. This represents the expectation when the image sample originates from a real image. This indicates that the image sample originates from the expectation when generating the image.

5. A multi-parallel text-to-image generation system employing the method described in any one of claims 1 to 4, characterized in that, include: The text feature extraction module is responsible for extracting cross-modal semantic features of text using a pre-trained text-image association model and inputting them into a recurrent neural network. It outputs the same number of text condition vectors as the number of branches in the image generation model. The multi-path image generation module is responsible for inputting different text condition vectors into the generation network modules of different branches in the image generation model, so that the generation network modules of different branches can generate images according to different generation strategies. The image spatial depth prediction module is responsible for using the spatial depth prediction model to predict the spatial depth information of the generated image, assigning corresponding weights to the images generated by different branches of the generative network module, and fusing the images generated by different branches of the generative network module into a single image.

6. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 4.

Images