A method and system for generating images from text based on spectral normalization and contrastive learning

Abstract

The present invention discloses a method and system for generating images from text based on spectral normalization and contrastive learning. The method includes performing contrastive learning iterative training on a generator including spectral normalization according to two different key semantic features and word-level text features respectively corresponding to two different text descriptions, using the generator to generate multiple images of different resolutions respectively corresponding to the two text descriptions in each iterative training; a discriminator evaluates the similarity between the multiple images of different resolutions generated in each iterative training and the text descriptions; iteratively optimizes the generator according to each similarity result, until the generator that has been trained is obtained when the similarity result meets the conditions, and uses the trained generator to generate a target image from the text description. The present invention uses spectral normalization and contrastive learning in the generator network to assist the model in more in-depth learning of the complex associations of the same image under different text descriptions, so as to significantly improve the quality of the generated image.

Description

Technical Field

[0001] The present invention relates to the technical field of text-to-image generation, and in particular to a text-to-image generation method and system based on spectral normalization and contrastive learning. Background Art

[0002] Text-generated images are an important area of ​​cross-modal research, covering multiple key areas such as natural language processing, computer vision, and artificial intelligence. This technology can convert text descriptions into corresponding images, which brings great convenience to many application fields. For example, it can generate highly simulated product images based on the text description of the product, or generate artistic images of a specific style based on the text description, which promotes the development of image editing. In addition, in the field of virtual reality, this technology also has a wide range of applications. Using this technology, it can help virtual reality systems restore real scenes more accurately.

[0003] In recent years, with the rapid development of text-generated image technology, a series of classic models have achieved remarkable results in this field. For example, StackGAN++ uses a stacked generation method to achieve 256×256 resolution images and achieves the generation of high-quality images from text descriptions. However, there is still semantic inconsistency between the generated image and the text description. AttnGAN effectively promotes the fusion of word text features and image features by introducing a transmembrane state attention mechanism, thereby enhancing the model's ability to understand text descriptions. In addition, AttnGAN also uses a deep attention multimodal similarity model to further improve the similarity calculation between text descriptions and generated images, which significantly improves the details of generated images. However, there are still problems such as insufficient details of generated images, unstable model training, and large differences in images generated by different text descriptions under the same image. Summary of the invention

[0004] The present invention provides a method and system for generating images from text based on spectral normalization and contrastive learning, which ensures that the texture and color of the generated image are clear and consistent with the semantic content described in the text, thereby achieving accurate generation from text to image.

[0005] In a first aspect, a method for generating images from text based on spectral normalization and contrastive learning is provided, comprising the following steps:

[0006] Randomly select two different text descriptions of the same image in the training dataset, and use the text encoder to obtain the key semantic features and word-level text features corresponding to the two different text descriptions;

[0007] Performing contrastive learning iterative training on a generator including spectral normalization according to the two different key semantic features and the word-level text features, and using the generator to generate a plurality of images of different resolutions corresponding to the two text descriptions in each iterative training;

[0008] The discriminator is used to evaluate the similarity between the multiple images of different resolutions generated by each iterative training and the text description, and the contrast loss is calculated for the two highest resolution images to evaluate the similarity between the two.

[0009] The generator is iteratively optimized according to each similarity result until the similarity result meets the conditions, and the trained generator is obtained, and the trained generator is used to generate the target image from the text description.

[0010] According to the first aspect, in a first possible implementation manner of the first aspect, the step of performing contrastive learning iterative training on a generator including spectral normalization according to the two different key semantic features and the word-level text features, and using the generator to generate a plurality of images of different resolutions corresponding to the two text descriptions in each iterative training specifically includes the following steps:

[0011] The generator includes an initial generation module using spectral normalization and two different word feature fusion refinement modules;

[0012] In each iteration of training, an initial generation module using spectral normalization is used to generate first resolution images and first image hidden features corresponding to two text descriptions respectively according to random noise sampled from a standard normal distribution and two different key semantic features;

[0013] Using one of the word feature fusion and refinement modules to fuse the first image hidden features and the word-level text features, and then generating a second resolution image and a second image hidden feature corresponding to two text descriptions respectively through a spectral normalization residual block and an upsampling network;

[0014] Another word feature fusion and refinement module is used to fuse the second image hidden features and the word-level text features, and then a spectral normalization residual block and an upsampling network are used to generate third-resolution images corresponding to the two text descriptions.

[0015] According to a first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, the step of generating the first resolution image and the first image hidden feature corresponding to two text descriptions respectively according to random noise sampled from a standard normal distribution and two different key semantic features using an initial generation module using spectral normalization specifically includes the following steps:

[0016] A perception layer in an initial generation module using spectral normalization is used to obtain a deep feature representation after combining the key semantic features with random noise from a standard normal distribution, and then a convolutional layer in the initial generation module is used to obtain image features after the deep feature representation, while spectral normalization is used to limit the gradient value in network propagation, and a gated activation function is used to generate image features rich in semantic information;

[0017] The convolution layer, the normalization layer, and the gated activation function operations are repeated four times to gradually generate an image of the first resolution and the corresponding first image hidden features.

[0018] According to the first possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, the calculation formula for the spectral normalization is as follows:

[0019]

[0020] Where W represents the initialization weight matrix; W SN represents the weight parameter matrix after spectral normalization operation; σ(W) represents the matrix spectral norm calculated by singular value decomposition; h represents the parameter matrix of network input.

[0021] According to the first possible implementation manner of the first aspect, according to the fourth possible implementation manner of the first aspect, the step of using one of the word feature fusion and refinement modules to fuse the first image hidden features and the word-level text features, and then generating two second resolution images and second image hidden features corresponding to the two text descriptions respectively through a spectral normalization residual block and an upsampling network, specifically includes the following steps:

[0022] Using an attention module in one of the word feature fusion and refinement modules to fuse the first image hidden feature and the word-level text feature, then using two residual blocks using spectral normalization to generate image features based on the fused features, and finally using an upsampling block to increase the resolution of the generated image features to generate a second image hidden feature;

[0023] A generator module in one of the word feature fusion and refinement modules is used to generate a second resolution image according to the second image hidden features.

[0024] According to a fourth possible implementation of the first aspect, according to a fifth possible implementation of the first aspect, the calculation formula of the step of using an attention module in one of the word feature fusion and refinement modules to fuse the first image hidden features and the word-level text features, and then using two residual blocks using spectral normalization to generate image features according to the fused features, and finally using an upsampling block to increase the resolution of the generated image features to generate the second image hidden features is as follows:

[0025]

[0026] In the formula, is the hidden feature of the first image; w is the word-level sentence feature of the text description; F attns and They are the spatial attention function and channel attention function of the attention module; SN res is the residual block function of spectral normalization; F up is the upsampling network function; They are the hidden image features obtained after the attention module and the residual block respectively; Hide features for the second image.

[0027] According to the first possible implementation manner of the first aspect, in a sixth possible implementation manner of the first aspect, the step of using another word feature fusion and refinement module to fuse the second image hidden features and the word-level text features, and then generating two third resolution images corresponding to the text descriptions respectively through a spectral normalization residual block and an upsampling network, specifically includes the following steps:

[0028] Using a dynamic memory module in another word feature fusion and refinement module to fuse the second image hidden feature and the word-level text feature, then using two residual blocks using spectral normalization to generate image features based on the fused features, and finally using an upsampling block to increase the resolution of the generated image features to generate a third image hidden feature;

[0029] A generator module in another word feature fusion and refinement module is used to generate a third resolution image according to the third image hidden features.

[0030] According to a sixth possible implementation manner of the first aspect, in a seventh possible implementation manner of the first aspect, the calculation formula of the step of using a dynamic memory module in another word feature fusion and refinement module to fuse the second image hidden feature and the word-level text feature, and then using two residual blocks using spectral normalization to generate image features according to the fused features, and finally using an upsampling block to increase the resolution of the generated image features to generate a third image hidden feature is as follows:

[0031]

[0032] Where w is the word-level sentence feature of the text description; F DM It is the dynamic memory module function; It is the hidden image feature after the dynamic memory module fuses the word-level text feature and the second image hidden feature; is the hidden image feature after the residual block is spectrally normalized; F upis the upsampling network function; SN res is the residual block function of spectral normalization; Hiding features for the second image; Hiding features for the third image.

[0033] According to the first aspect, in an eighth possible implementation manner of the first aspect, a calculation formula of the loss function L corresponding to the generator is as follows:

[0034]

[0035]

[0036] L NT-Xent =F(G(text1),G(text2))

[0037] In the formula, is the generator loss value of the i-th stage; L CA To enhance sentence-level text features and prevent overfitting loss function; L DAMSM To minimize the matching loss between image and text; L NT-Xent is the contrast loss function between images generated by two different text descriptions; text1 and text2 are two different text descriptions randomly obtained from the same image; G is the generator; F is the contrast loss between the two generated images; λ1, λ2, λ3 are the ratios for balancing different losses; Distribution of images generated from the generator False images; To generate the authenticity discriminant loss of the image; To generate consistency loss between images and text descriptions.

[0038] In a second aspect, a text-to-image system based on spectral normalization and contrastive learning is provided, comprising:

[0039] The encoding module is used to randomly select two different text descriptions of the same image in the training data set, and use the text encoder to obtain the key semantic features and word-level text features corresponding to the two different text descriptions;

[0040] A generator training module, which is in communication with the encoding module, is used to perform contrastive learning iterative training on a generator including spectral normalization according to two different key semantic features and word-level text features, and use the generator to generate a plurality of images of different resolutions corresponding to two text descriptions in each iterative training;

[0041] a discriminator module, which is in communication with the generator training module and is used to use the discriminator to evaluate the similarity between the multiple images of different resolutions generated by each iterative training and the text description, and to calculate the contrast loss of the two highest resolution images to evaluate the similarity between the two images; and

[0042] The target image generation module is in communication with the discrimination module and is used to iteratively optimize the generator according to each similarity result until the similarity result meets the conditions, obtain the generator that has been trained at this time, and use the trained generator to generate the target image from the text description.

[0043] Compared with the prior art, the advantages of the present invention are as follows: by performing contrastive learning iterative training on a generator including spectral normalization, during the training process, a spectral normalized generator is used for contrastive learning iterative training, which can retain and enhance important image features and improve the detail performance of the generated image and the consistency with the text description. At the same time, the contrastive learning strategy can further guide the model to learn the intrinsic connection between the same image and different text descriptions, improve the model's in-depth understanding of the text description, and improve the consistency between the generated image and the text description. By calculating the contrast loss between images generated by different text descriptions under the same image, the generator generation process is guided, so that the image generated by the generator is more consistent with the text description and the generated image quality is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of an embodiment of a method for generating images from text based on spectral normalization and contrastive learning of the present invention;

[0045] Figure 2 It is a flow chart of another embodiment of a method for generating images from text based on spectral normalization and contrastive learning of the present invention;

[0046] Figure 3 It is a structural schematic diagram of a text-to-image system based on spectral normalization and contrastive learning of the present invention. DETAILED DESCRIPTION

[0047] Reference will now be made in detail to specific embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Although the present invention will be described in conjunction with specific embodiments, it will be understood that it is not intended to limit the present invention to the embodiments described. On the contrary, it is intended to cover changes, modifications and equivalents included in the spirit and scope of the present invention as defined by the appended claims. It should be noted that the method steps described herein can all be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of the two.

[0048] In order to enable those skilled in the art to better understand the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0049] Note: The example to be introduced below is only a specific example, and is not intended to limit the embodiments of the present invention to the following specific steps, values, conditions, data, sequences, etc. Those skilled in the art can apply the concept of the present invention to construct more embodiments not mentioned in this specification by reading this specification.

[0050] See also Figure 1 and Figure 2 The embodiment of the present invention provides a method for generating images from text based on spectral normalization and contrastive learning, comprising the following steps:

[0051] S100, randomly obtaining two different text descriptions of the same image from the training data set, and using a text encoder to obtain key semantic features and word-level text features corresponding to the two different text descriptions;

[0052] According to the given image in the dataset, two different text descriptions are obtained, and the sentence-level text features and word-level text features of the two different text descriptions are obtained using a text encoder; then, the two sentence-level text features are semantically enhanced and key semantic features are extracted through conditional enhancement.

[0053] Sentence-level text features are a global understanding and representation of the entire text description. These features capture the overall meaning of the sentence and are used to guide image generation; word-level text features encode each word in a vector, pay more attention to specific words or phrases in the text description, and provide detailed information about each sub-region or detail in the image. For example, if the text description mentions "a bird with blue feathers", the word-level features may capture the three keywords "blue", "feathers", and "bird", guiding the generator to generate an image of a bird with blue feathers.

[0054] The sentence-level text features are semantically enhanced, specifically, obtaining the mean and variance of the text features through the perception layer, and then sampling based on the mean and variance using the heavy parameter technique to obtain a conditional vector that conforms to the normal distribution, and then concatenating it with random noise of the sampling and standard normal distribution to obtain key semantic features.

[0055] S200, performing contrastive learning iterative training on a generator including spectral normalization according to the two different key semantic features and the word-level text features, and using the generator to generate a plurality of images of different resolutions corresponding to the two text descriptions in each iterative training;

[0056] S300, using a discriminator to evaluate the similarity between a plurality of images of different resolutions generated by each iterative training and the text description, and calculating a contrast loss on the two highest resolution images to evaluate the similarity between the two;

[0057] S400, iteratively optimizing the generator according to each similarity result until the similarity result meets the conditions, obtaining the generator that has been trained at this time, and using the trained generator to generate a target image from the text description.

[0058] Specifically, in this embodiment, the present invention performs contrastive learning iterative training on a generator including spectral normalization. During the training process, a spectral normalized generator is used to perform contrastive learning iterative training, which can retain and enhance important image features and improve the detail performance of the generated image and the consistency with the text description. At the same time, the contrastive learning strategy can further guide the model to learn the intrinsic connection between the same image and different text descriptions, improve the model's in-depth understanding of the text description, and improve the consistency between the generated image and the text description. The generator generation process is guided by calculating the contrast loss between images generated by different text descriptions under the same image, so that the image generated by the generator is more consistent with the text description and the generated image quality is higher.

[0059] Preferably, in another embodiment of the present application, the S200, performing contrastive learning iterative training on a generator including spectral normalization according to the two different key semantic features and the word-level text features, and using the generator to generate a plurality of images of different resolutions corresponding to the two text descriptions in each iterative training, specifically includes the following steps:

[0060] The generator includes an initial generation module using spectral normalization and two different word feature fusion refinement modules;

[0061] S210, in each iteration of training, using an initial generation module using spectral normalization, based on random noise sampled from a standard normal distribution and two different key semantic features, generating first resolution images and first image hidden features corresponding to two text descriptions respectively;

[0062] S220, using one of the word feature fusion and refinement modules to fuse the first image hidden features and the word-level text features, and then generating a second resolution image and a second image hidden feature corresponding to two text descriptions respectively through a spectral normalization residual block and an upsampling network;

[0063] S230, using another word feature fusion and refinement module to fuse the second image hidden features and the word-level text features, and then generating third resolution images corresponding to two text descriptions respectively through a spectral normalization residual block and an upsampling network.

[0064] Preferably, in another embodiment of the present application, the S210, using an initial generation module using spectral normalization according to random noise sampled from a standard normal distribution and two different key semantic features, generates a first resolution image and a first image hidden feature step corresponding to two text descriptions, respectively, specifically includes the following steps:

[0065] A perception layer in an initial generation module using spectral normalization is used to obtain a deep feature representation after combining the key semantic features with random noise from a standard normal distribution, and then a convolutional layer in the initial generation module is used to obtain image features after the deep feature representation, while spectral normalization is used to limit the gradient value in network propagation, and a gated activation function is used to generate image features rich in semantic information;

[0066] The convolution layer, the normalization layer, and the gated activation function operations are repeated four times to gradually generate an image of the first resolution and the corresponding first image hidden features.

[0067] Preferably, in another embodiment of the present application, the calculation formula for spectrum normalization is as follows:

[0068]

[0069]

[0070] Where W represents the initialization weight matrix; W SN represents the weight parameter matrix after spectral normalization operation; σ(W) represents the matrix spectral norm calculated by singular value decomposition; h represents the parameter matrix of network input.

[0071] Preferably, in another embodiment of the present application, the S220, using one of the word feature fusion and refinement modules to fuse the first image hidden features and the word-level text features, and then generating two second resolution images and second image hidden features corresponding to the text descriptions respectively through a spectral normalization residual block and an upsampling network, specifically includes the following steps:

[0072] Using an attention module in one of the word feature fusion and refinement modules to fuse the first image hidden feature and the word-level text feature, then using two residual blocks using spectral normalization to generate image features based on the fused features, and finally using an upsampling block to increase the resolution of the generated image features to generate a second image hidden feature;

[0073] A generator module in one of the word feature fusion and refinement modules is used to generate a second resolution image according to the second image hidden features.

[0074] Preferably, in another embodiment of the present application, the step S230, using another word feature fusion and refinement module to fuse the second image hidden features and the word-level text features, and then generating two third resolution images corresponding to the text descriptions respectively through a spectral normalization residual block and an upsampling network, specifically includes the following steps:

[0075] Using a dynamic memory module in another word feature fusion and refinement module to fuse the second image hidden feature and the word-level text feature, then using two residual blocks using spectral normalization to generate image features based on the fused features, and finally using an upsampling block to increase the resolution of the generated image features to generate a third image hidden feature;

[0076] A generator module in another word feature fusion and refinement module is used to generate a third resolution image according to the third image hidden features.

[0077] Specifically, in this embodiment, two groups of different text descriptions are passed through a generator network with the same parameters to obtain corresponding generated images. The process is to concatenate the key semantic features and random noise of the text description as input, obtain the input of the initial generation module using spectral normalization after full connection and dimension conversion, then use an upsampling network to increase the image feature dimension size, and use a convolutional neural network to strengthen the image features. After spectral normalization, the gradient value in network propagation is limited, and then a gated activation function is used to further process the image features to generate image features rich in semantic information. The same feature extraction is repeated four times to obtain a first resolution image with a resolution of 64×64 that basically meets the text description and a first image hidden feature.

[0078] Using one of the word feature fusion and refinement modules to fuse the first image hidden features and the word-level text features, and then generating a second resolution image and a second image hidden features with a resolution of 128×128 corresponding to two text descriptions respectively through a spectral normalization residual block and an upsampling network;

[0079] Another word feature fusion and refinement module is used to fuse the second image hidden features and the word-level text features, and then a third resolution image with a resolution of 256×256 corresponding to the two text descriptions is generated through a spectral normalization residual block and an upsampling network to achieve text-to-image generation.

[0080] Based on the generator with the same parameters, corresponding images generated by two different text descriptions were obtained, and the loss function was calculated based on the generated images. Specifically, the original generator loss function was calculated using generated image 1 and generated image 2, and then the similarity loss value between generated image 1 and generated image 2 was calculated using contrast loss. The final generator loss value was obtained to guide the generator to continuously learn the relationship between different text descriptions and images during the training process, gradually improve the detail performance of the image, thereby improving the consistency between the generated image and the text description, and finally achieving the task of generating high-quality target images.

[0081] Preferably, in another embodiment of the present application, the calculation formula of S220 is as follows:

[0082]

[0083] In the formula, is the hidden feature of the first image; w is the word-level sentence feature of the text description; and Respectively represent the spatial attention and channel attention module functions of the attention module; SN res is the residual network function of spectral normalization; F up is the upsampling block function; They represent the hidden image features obtained after the attention module and the residual network module respectively; Hide features for the second image.

[0084] Preferably, in another embodiment of the present application, the calculation formula of S230 is as follows:

[0085]

[0086] In the formula, F DM It is the dynamic memory module function; It is the hidden image feature after the dynamic memory module fuses the word-level text feature and the second image hidden feature; is the hidden image feature after the residual block is spectrally normalized; Hiding features for the third image.

[0087] Preferably, in another embodiment of the present application, in order to generate a high-quality image that conforms to the text description content, the loss function L corresponding to the generator is calculated as follows:

[0088]

[0089]

[0090] L NT-Xent=F(G(text1),G(text2))

[0091] In the formula, is the generator loss value of the i-th stage; L CA To enhance sentence-level text features and prevent overfitting loss function; L DAMSM To minimize the matching loss between image and text; L NT-Xent is the contrast loss function between images generated by two different text descriptions; text1 and text2 are two different text descriptions randomly obtained from the same image; G is the generator; F is the contrast loss between the two generated images; λ1, λ2, λ3 are the ratios for balancing different losses; Distribution of images generated from the generator False images; To generate the authenticity discriminant loss of the image; To generate consistency loss between images and text descriptions.

[0092] It should be noted that three different discriminators need to be trained, and their loss functions are defined as follows:

[0093]

[0094]

[0095]

[0096] in is the unconditional loss function used to determine the authenticity of the input image, is the conditional loss function for judging the consistency between the input image and text features, x represents the image distribution from the real image P data The real image.

[0097] See also Figure 3 As shown, the embodiment of the present invention further provides a text-to-image system based on spectral normalization and contrastive learning, comprising:

[0098] The encoding module is used to randomly select two different text descriptions of the same image in the training data set, and use the text encoder to obtain the key semantic features and word-level text features corresponding to the two different text descriptions;

[0099] A generator training module, which is in communication with the encoding module, is used to perform contrastive learning iterative training on a generator including spectral normalization according to two different key semantic features and word-level text features, and use the generator to generate a plurality of images of different resolutions corresponding to two text descriptions in each iterative training;

[0100] a discriminator module, which is in communication with the generator training module and is used to use the discriminator to evaluate the similarity between the multiple images of different resolutions generated by each iterative training and the text description, and to calculate the contrast loss of the two highest resolution images to evaluate the similarity between the two images; and

[0101] The target image generation module is in communication with the discrimination module and is used to iteratively optimize the generator according to each similarity result until the similarity result meets the conditions, obtain the generator that has been trained at this time, and use the trained generator to generate the target image from the text description.

[0102] Therefore, the present invention adopts a spectrally normalized initial generation module to generate an initial image according to the semantic features of the text description, and limits the gradient value of the generator network to better stabilize the generator network training. The first and second different feature fusion refinement modules are used to retain and enhance important image features and improve the detail performance of the generated image and the consistency with the text description. At the same time, a contrastive learning strategy is used to guide the model to learn the intrinsic connection between the same image and different text descriptions, improve the model's in-depth understanding of the text description, and improve the consistency between the generated image and the text description. The generator generation process is guided by calculating the contrast loss between images generated by different text descriptions under the same image, so that the image generated by the generator is more consistent with the text description and the generated image quality is higher.

[0103] Specifically, this embodiment corresponds one-to-one to the above method embodiment, and the functions of each module have been described in detail in the corresponding method embodiment, so they will not be repeated here.

[0104] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, all or part of the method steps of the above method are implemented.

[0105] The present invention implements all or part of the process in the above method, and can also be completed by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of each method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0106] Based on the same inventive concept, an embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program running on the processor, and when the processor executes the computer program, all or part of the method steps in the above method are implemented.

[0107] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of a computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0108] The memory can be used to store computer programs and / or modules, and the processor realizes various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the mobile phone (such as audio data, video data, etc.). In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMed i aCard, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0109] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, servers or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program codes.

[0110] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), servers, and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as a combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0111] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0113] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A text-to-image generation method based on spectral normalization and contrastive learning, characterized in that: The following steps are involved: Randomly select two different text descriptions of the same image in the training dataset, and use the text encoder to obtain the key semantic features and word-level text features corresponding to the two different text descriptions; Performing contrastive learning iterative training on a generator including spectral normalization according to the two different key semantic features and the word-level text features, and using the generator to generate a plurality of images of different resolutions corresponding to the two text descriptions in each iterative training; The discriminator is used to evaluate the similarity between the multiple images of different resolutions generated by each iterative training and the text description, and the contrast loss is calculated for the two highest resolution images to evaluate the similarity between the two. The generator is optimized iteratively according to each similarity result until the similarity result meets the conditions, and the trained generator is obtained, and the trained generator is used to generate the target image from the text description; The step of performing contrastive learning iterative training on a generator including spectral normalization according to the two different key semantic features and the word-level text features, and using the generator to generate a plurality of images of different resolutions corresponding to the two text descriptions in each iterative training, specifically includes the following steps: The generator includes an initial generation module using spectral normalization and two different word feature fusion refinement modules; In each iteration of training, an initial generation module using spectral normalization is used to generate first resolution images and first image hidden features corresponding to two text descriptions respectively according to random noise sampled from a standard normal distribution and two different key semantic features; Using one of the word feature fusion and refinement modules to fuse the first image hidden features and the word-level text features, and then generating a second resolution image and a second image hidden feature corresponding to two text descriptions respectively through a spectral normalization residual block and an upsampling network; Using another word feature fusion and refinement module to fuse the second image hidden features and the word-level text features, and then generating third-resolution images corresponding to two text descriptions respectively through a spectral normalization residual block and an upsampling network; The step of generating the first resolution image and the first image hidden feature corresponding to two text descriptions respectively by using the initial generation module using spectral normalization according to the random noise sampled from the standard normal distribution and the two different key semantic features specifically includes the following steps: A perception layer in an initial generation module using spectral normalization is used to obtain a deep feature representation after combining the key semantic features with random noise from a standard normal distribution, and then a convolutional layer in the initial generation module is used to obtain image features after the deep feature representation, while spectral normalization is used to limit the gradient value in network propagation, and a gated activation function is used to generate image features rich in semantic information; Repeat the convolution layer, the normalization layer, and the gated activation function operations four times to gradually generate an image of the first resolution and corresponding first image hidden features; The step of using one of the word feature fusion and refinement modules to fuse the first image hidden features and the word-level text features, and then generating the second resolution image and the second image hidden features corresponding to the two text descriptions respectively through a spectral normalization residual block and an upsampling network, specifically includes the following steps: Using an attention module in one of the word feature fusion and refinement modules to fuse the first image hidden feature and the word-level text feature, then using two residual blocks using spectral normalization to generate image features based on the fused features, and finally using an upsampling block to increase the resolution of the generated image features to generate a second image hidden feature; A generator module in one of the word feature fusion and refinement modules is used to generate a second resolution image according to the second image hidden features.

2. The text-to-image generation method based on spectral normalization and contrastive learning as claimed in claim 1, characterized in that: The calculation formula for the spectrum normalization is as follows: in, W Represents the initialization weight matrix; W SN Represents the weight parameter matrix after spectral normalization operation; Indicates that the matrix spectral norm is calculated by singular value decomposition; h A parameter matrix representing the network input.

3. The text-to-image generation method based on spectral normalization and contrastive learning as claimed in claim 1, characterized in that: The calculation formula of the step of using the attention module in one of the word feature fusion and refinement modules to fuse the first image hidden feature and the word-level text feature, then using two residual blocks using spectral normalization to generate image features according to the fused features, and finally using the upsampling block to increase the resolution of the generated image features to generate the second image hidden feature is as follows: ； In the formula, Hiding features for the first image; Word-level sentence features for text description; and They are the spatial attention function and channel attention function of the attention module respectively; is the residual block function of spectral normalization; is the upsampling network function; , They are the hidden image features obtained after the attention module and the residual block respectively; Hide features for the second image.

4. The text-to-image generation method based on spectral normalization and contrastive learning as claimed in claim 1, characterized in that: The step of fusing the second image hidden features and the word-level text features using another word feature fusion and refinement module, and then generating the third resolution images corresponding to the two text descriptions respectively through a spectral normalization residual block and an upsampling network, specifically includes the following steps: Using a dynamic memory module in another word feature fusion and refinement module to fuse the second image hidden feature and the word-level text feature, then using two residual blocks using spectral normalization to generate image features based on the fused features, and finally using an upsampling block to increase the resolution of the generated image features to generate a third image hidden feature; A generator module in another word feature fusion and refinement module is used to generate a third resolution image according to the third image hidden features.

5. The text-to-image generation method based on spectral normalization and contrastive learning as claimed in claim 4, characterized in that: The calculation formula of the step of using the dynamic memory module in another word feature fusion and refinement module to fuse the second image hidden feature and the word-level text feature, then using two residual blocks using spectral normalization to generate image features according to the fused features, and finally using the upsampling block to increase the resolution of the generated image features to generate the third image hidden feature is as follows: ； In the formula, Word-level sentence features for text description; It is the dynamic memory module function; It is the hidden image feature after the dynamic memory module fuses the word-level text feature and the second image hidden feature; is the hidden image feature after the residual block is spectrally normalized; is the upsampling network function; is the residual block function of spectral normalization; Hiding features for the second image; Hiding features for the third image.

6. The text-to-image generation method based on spectral normalization and contrastive learning as claimed in claim 1, characterized in that: The loss function corresponding to the generator L The calculation formula is as follows: ； In the formula, is the generator loss value of the i-th stage; A loss function to enhance sentence-level text features to prevent overfitting; To minimize the matching loss between image and text; Contrastive loss between images generated for two different text descriptions; text 1 and text 2 are two different text descriptions randomly obtained from the same image; G For the generator; F To calculate the contrast loss between two generated images; To balance the proportions between different losses; Distribution of images generated from the generator False images; To generate the authenticity discriminant loss of the image; To generate consistency loss between images and text descriptions.

7. A text-to-image system based on spectral normalization and contrastive learning, characterized in that: include: The encoding module is used to randomly select two different text descriptions of the same image in the training data set, and use the text encoder to obtain the key semantic features and word-level text features corresponding to the two different text descriptions; A generator training module, which is in communication with the encoding module, is used to perform contrastive learning iterative training on a generator including spectral normalization according to two different key semantic features and word-level text features, and use the generator to generate a plurality of images of different resolutions corresponding to two text descriptions in each iterative training; A discriminator module is connected to the generator training module for evaluating the similarity between the multiple images with different resolutions generated by each iterative training and the text description by using the discriminator, and calculating the contrast loss of the two highest resolution images to evaluate the similarity between the two images; as well as, A target image generation module, which is in communication with the discrimination module and is used to iteratively optimize the generator according to each similarity result until the similarity result meets the condition, obtain the generator that has been trained at this time, and use the trained generator to generate the target image from the text description; The step of performing contrastive learning iterative training on a generator including spectral normalization according to the two different key semantic features and the word-level text features, and using the generator to generate a plurality of images of different resolutions corresponding to the two text descriptions in each iterative training, specifically includes: The generator includes an initial generation module using spectral normalization and two different word feature fusion refinement modules; In each iteration of training, an initial generation module using spectral normalization is used to generate first resolution images and first image hidden features corresponding to two text descriptions respectively according to random noise sampled from a standard normal distribution and two different key semantic features; Using one of the word feature fusion and refinement modules to fuse the first image hidden features and the word-level text features, and then generating a second resolution image and a second image hidden feature corresponding to two text descriptions respectively through a spectral normalization residual block and an upsampling network; Using another word feature fusion and refinement module to fuse the second image hidden features and the word-level text features, and then generating third-resolution images corresponding to two text descriptions respectively through a spectral normalization residual block and an upsampling network; The step of generating the first resolution image and the first image hidden feature corresponding to the two text descriptions respectively according to the random noise sampled from the standard normal distribution and the two different key semantic features by using the initial generation module using spectral normalization specifically includes: A perception layer in an initial generation module using spectral normalization is used to obtain a deep feature representation after combining the key semantic features with random noise from a standard normal distribution, and then a convolutional layer in the initial generation module is used to obtain image features after the deep feature representation, while spectral normalization is used to limit the gradient value in network propagation, and a gated activation function is used to generate image features rich in semantic information; Repeat the convolution layer, the normalization layer, and the gated activation function operations four times to gradually generate an image of the first resolution and corresponding first image hidden features; The step of using one of the word feature fusion and refinement modules to fuse the first image hidden features and the word-level text features, and then generating the second resolution image and the second image hidden features corresponding to the two text descriptions respectively through a spectral normalization residual block and an upsampling network specifically includes: Using an attention module in one of the word feature fusion and refinement modules to fuse the first image hidden feature and the word-level text feature, then using two residual blocks using spectral normalization to generate image features based on the fused features, and finally using an upsampling block to increase the resolution of the generated image features to generate a second image hidden feature; A generator module in one of the word feature fusion and refinement modules is used to generate a second resolution image according to the second image hidden features.

Images