Text-to-image model training method and system, and text-to-image method and system

Abstract

The present invention provides a text-generated image model training method and system, as well as a text-generated image method and system, belonging to the field of computer vision and artificial intelligence technology. The text description is input into a pre-trained text encoder to extract global sentence features and word feature matrices, and randomly sampled noise features are connected with sentence features and input into a generation network to extract low-scale visual text features in the first generation module. The image features and visual text features are enhanced by a cross-scale channel activation module, and the image features are respectively passed into the generator in order to obtain a generated image. The generation network of the cross-scale channel activation module of the present invention fuses low-scale visual text features containing rich global semantic information with image features to avoid the final image generation quality being overly dependent on the generation result of the initial image; by adding punitive weights to difficult samples, the discriminant network focuses on learning difficult samples, thereby realizing the mining of difficult samples.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and artificial intelligence technology, and in particular to a text-to-image model training method and system, as well as a text-to-image method and system. Background Art

[0002] The task of generating images from text (Text to Image, TTI) involves cross-modal data ("text-image" pairs). Cross-modal data refers to data with the same high-level semantics but different information carriers, for example, describing the semantics of "bird" through text, images, videos, audio, etc. Text-to-image technology uses training and learning on cross-modal data to obtain high-performance generative models to reduce the high labor costs of tasks such as story visualization, image editing, image completion, and intelligent recipe generation. Text-to-image generation is a cutting-edge task in the cross-modal field. The acquisition of images requires expensive labor costs, and it is difficult to obtain large quantities of high-quality images. For example, obtaining one million high-definition images of birds can create some images with unique styles, such as images in the style of abstract paintings.

[0003] The text-to-image task aims to use a generative model to learn the mapping relationship between text features and image features, and ultimately use text descriptions to generate high-quality, high-resolution images that are not included in the training set and are semantically consistent with the input text.

[0004] In 2016, Reed et al. first applied generative adversarial networks to the task of generating images from text, sparking a wave of research on GAN-based methods for generating images from text. The GAN-INT-CLS model, based on a conditional GAN, extracts features from text descriptions and combines them with randomly sampled noise as input, generating images guided by the text. While the GAN-INT-CLS model achieves image generation from text, the quality of the generated images is poor, and the resolution of the generated images is only 64×64. Furthermore, the large domain gap between the text and image domains remains difficult to address. To address these issues, many excellent image generation mechanisms have been introduced into the task of generating images from text.

[0005] (1) Generative model based on cascade generation mechanism. The StackGAN model is based on iterative generative adversarial networks and adopts a two-stage training method. The first stage generates a 64×64 low-resolution image, which focuses on generating the color and rough shape of the image target. The second stage uses the generated image of the first stage as input to generate a 128×128 or 256×256 high-resolution image, while enhancing the details of the content in the low-resolution image to make the final generated image more refined. In addition, since the text feature space is often high-dimensional (generally greater than 100 dimensions) and the text description is relatively small, this will lead to the problem of data flow discontinuity in the high-dimensional space for limited text data. To address this problem, Zhang et al. proposed a conditional enhancement module. This module generates additional text features to enhance the smoothness of the data flow of text features in the high-dimensional space. However, the two-stage training method of the StackGAN model is not end-to-end, and the model has obvious mode collapse problems. To address this problem, Zhang et al. proposed the StackGAN++ model based on the StackGAN model. The StackGAN++ model also serves as the foundational framework for subsequent text-to-image generation methods based on generative adversarial networks. The StackGAN++ model proposes a multi-stage generative adversarial network structure, addressing the StackGAN model's non-end-to-end nature. The model progressively generates images with 64×64, 128×128, and 256×256 resolutions by cascading three sets of generative adversarial networks. Furthermore, the StackGAN++ model incorporates a mean squared error loss into the generator loss function to enhance color consistency in the multi-stage generated images and mitigate the problem of pattern collapse. However, the StackGAN and StackGAN++ models focus solely on global sentence features, ignoring the guiding role of fine-grained word features in image generation.

[0006] (2) Generative model based on attention mechanism. In response to the defects of the StackGAN++ model, Xu et al. proposed the AttnGAN model. In the generative network, the AttnGAN model uses the cross-attention mechanism to semantically align the word feature matrix with the sub-region features of the generated image, dynamically converting text features into image features, thereby improving the quality of the generated image. In order to narrow the difference between the text feature space and the image feature space, the AttnGAN model adds a cross-modal similarity matching loss to the generator. The generation results of the AttnGAN model not only show the importance of fine-grained word features for image generation, but also verify the effectiveness of the attention mechanism for the text-to-image task. However, the AttnGAN model only considers the connection between the word feature matrix and the image sub-region, ignoring the role of global information in image generation. In response to the problems of the AttnGAN model, the MirrorGAN model proposes a corresponding improvement method. The MirrorGAN model introduces global sentence features and global image features, and uses the cross-attention mechanism to build a connection between the two. This method allows the image generation process to receive global information, enhancing the diversity and semantic consistency of the generated image. In addition, the Control-GAN model proposes a channel attention mechanism. The Control-GAN model uses word-level attention and channel-level attention to control the generated image category, color, texture and other attributes.

[0007] (3) Generative model based on dynamic memory mechanism. The StackGAN++ model proposes a multi-level generative adversarial network structure, which has a significant effect on generating high-resolution images. However, the multi-level generative adversarial network structure still has shortcomings. During the generation process of this structure, the quality of the high-resolution image is directly related to the generation effect of the low-resolution image. The low-resolution image mainly generates rough information of the target, such as position, color, shape, etc., while the high-resolution image only refines the rough content of the low-resolution image and cannot correct the original defects. In order to alleviate the impact of the final generated image quality on the lowest resolution generated image, the DMGAN model introduces a dynamic memory method into the text generation image task. The model proposes a dynamic memory module. This module calculates the similarity matrix between the local word feature matrix and the initial generated image feature, and dynamically converts the word feature matrix into visual features through attention weighting. The DMGAN model incorporates this visual feature into image generation to alleviate the problem that the image generation quality depends on the quality of the initial image.

[0008] (4) Generative models based on mirroring mechanisms. In recent years, the image captioning task has become increasingly popular. This task aims to obtain text information that is semantically consistent with the image content from the input image. Inspired by the CycleGAN model and the AugCycleGAN model, Qiao et al. proposed the MirrorGAN model. The text-to-image task aims to generate images from text, and Qiao et al. introduced the "text-image-text" mirroring method. This model uses image captioning to extract the corresponding text description for the generated image, and by constructing a loss function to align the input text with the extracted text semantics, thereby enhancing the semantic consistency between the generated image and the text. At the same time, MirrorGAN improves AttnGAN by adding an attention mechanism to both global text features and local word matrices during the generation process. This method allows the generated image to consider both global and local text information, thereby enhancing the semantic consistency between the text description and the generated image. However, the performance of the MirrorGAN model is highly dependent on the performance of the pre-trained image captioning generation model.

[0009] (5) Image mask-based generative models. The AGAN-CL model and the SegAttnGAN model introduce image masks into the text generation image task. The AGAN-CL model uses image masks to provide coarse-grained information such as layout, shape, and quantity for the generated image. The model first uses a generative network to construct a mapping relationship between text and image masks, and then uses a cyclic transformation autoencoder to achieve dynamic conversion between salient images and generated images. The SegAttnGAN model introduces image mask information into the generative network and incorporates the image mask information into the generated image through the attention mechanism to constrain the layout of the generated image and the shape of the target, thereby improving the quality of the generated image. Although the AGAN-CL model and the SegAttnGAN model have achieved good results, both require the training set to contain image mask label information, which means that it costs more to build the dataset.

[0010] (6) Generative model based on the twin network framework. Yin et al. introduced the twin network framework and proposed the SD-GAN model to address the problem that images generated by semantically identical text descriptions have large differences. The SD-GAN model uses the twin network to learn the semantic commonalities between different texts with consistent semantics, so that the text description and the generated image are semantically consistent. The model constructs two generation branches, with different text description pairs as input. The SD-GAN uses the contrastive loss function to minimize the feature distance of the generated image generated by the semantically consistent text description or maximize the feature distance of the generated image generated by the semantically different text description. In addition, the SE-GAN model also borrows the framework of the twin network. Compared with the SD-GAN model, the SE-GAN model uses the twin network to extract the features of the generated image and the real image respectively. It minimizes the feature distance between the generated image and the real image of the same text description through the contrastive loss function, while maximizing the feature distance between the generated image and the real image of different text descriptions. Secondly, inspired by the focal loss function, the SE-GAN model uses the sliding loss function combined with the contrastive loss function to balance the weights of difficult samples and simple samples, further enhancing the semantic consistency between the generated image and the text description.

[0011] (7) Generative model based on target detection. The Obj-GAN model borrows the idea of ​​target detection. It takes into account the spatial structure of the generated object and realizes text-generated image generation through hierarchical mapping. The Obj-GAN model first obtains the target box information through text description, thereby obtaining the layout information of the target; then, it generates a target mask for the target box in combination with the text description; finally, it uses the generative model to generate the final image. At the same time, the Obj-GAN model proposes a target-level discriminant model. This discriminant model uses Faster-RCNN as the target detection model to capture the foreground target of the generated image and detect whether the foreground target matches the target layout. However, the Obj-GAN model requires that the training image contain the bounding box and shape information of each object. This is time-consuming and expensive when processing large datasets.

[0012] In summary, many existing methods, such as the MirrorGAN model, the Leica-GAN model, and the CPGAN model, are based on multi-stage generative adversarial network structures. However, a drawback of cascaded generation methods is that the quality of the final generated image is highly dependent on the quality of the initial generated image. If the generated low-resolution image is poor, it will greatly affect the quality of the subsequent high-resolution image generation. Although the DM-GAN model addresses this issue by proposing a dynamic memory module to dynamically update the generated images at each level, which to some extent reduces the dependence of the high-resolution image generation results on the quality of the low-resolution images, this model ignores the guiding role of visual text features with richer semantic information in image generation, resulting in poor generated image quality. Secondly, due to the sample imbalance problem in commonly used datasets for text-generated images and the lack of mining of difficult samples in mainstream methods, some images generated by these methods still have extremely poor results. Summary of the Invention

[0013] The purpose of the present invention is to provide a text-to-image model training method and system, as well as a text-to-image method and system, to solve at least one technical problem existing in the above-mentioned background technology.

[0014] In order to achieve the above object, the present invention adopts the following technical solutions:

[0015] In one aspect, the present invention provides a method for training a text-to-image model, comprising:

[0016] Acquire training data; the training data includes multiple sets of data, each set of data includes matching pairs of text description information and natural images, and annotates labels in the natural images that match the text description information;

[0017] The obtained training data is used to train the network model to obtain a trained text-to-image model;

[0018] The network model includes a text encoder, an image encoder, a generative network, and an adaptive discriminant network; the text encoder is used to extract the global sentence features and word feature matrix of the text description information; the image encoder is used to extract the global image features of the natural image, and extract the global image features of the generated image generated by the generative network; the generative network is used to obtain image features based on the global sentence features and word feature matrix, and obtain the generated image based on the image features and fuse the image features to obtain the generated image; the adaptive discriminant network is used to judge the authenticity of the generated image and whether the content of the generated image is consistent with the text description based on the global image features of the natural image and the global image features of the generated image;

[0019] Among them, the generative network includes a cross-scale channel activation module that uses the global information of visual text features in the global sentence features to enhance the image features, thereby obtaining a fused image feature with stronger representation ability.

[0020] Optionally, the generation network includes multiple cascaded image feature generators, each of which is connected to an image generator; the first image feature generator is used to extract low-scale visual text features from global sentence features and generate a first image feature, and the first image generator is used to generate a first generated image based on the first image feature; each subsequent image feature generator is used to generate high-scale image features based on the image features and word feature matrix generated by the previous image feature generator, and the cross-scale channel activation module is connected to the corresponding image generator, and the cross-scale channel activation module fuses the high-scale image features and low-scale visual text features generated by the corresponding image feature generator to obtain the fused image feature.

[0021] Optionally, use a bidirectional long short-term memory network as the text encoder:

[0022] W,s=F LSTM (T)

[0023] in, w l is the feature vector of the lth word, corresponding to the lth hidden layer output, is the output of the last hidden layer, is the long short-term memory network, D is w l and the vector dimensions of s;

[0024] The conditional enhancement module is used to expand the number of text features and enhance the smoothness of text features in high-dimensional space:

[0025] s ca =F ca (s);

[0026] in, D′ is the vector dimension after conditional enhancement;

[0027] Use the pre-trained Inception-v3 model as the image encoder:

[0028] f c ,f v =Inc(I img );

[0029] Among them, I img is the input image, The local features of the image extracted from the image; is the global feature of the image.

[0030] Optionally, the cross-scale channel activation module consists of a pooling layer, a convolutional network layer, a Sigmoid activation layer and a ReLU activation layer; the input of the cross-scale channel activation module is high-scale image features and low-scale visual text features; wherein, the pooling layer downsamples the visual text features, the convolutional network layer downsamples them, and uses the ReLU activation function to map the input data to a nonlinear space; then, another convolutional network layer is used to map the latent vector to be consistent with the image feature channel, and the Sigmoid function is used as the activation function; thereafter, the two feature vectors are combined using channel multiplication, and the high-scale image features are enhanced using the global information of the visual text features to obtain image features with stronger representation capabilities.

[0031] Optional, attention module The cross-attention mechanism is used to achieve semantic alignment between word feature matrix and image feature, promoting the consistency between image content and text description; the input of the attention module is the word feature matrix and image features Among them, the i-th column of the word feature matrix represents the feature vector of the i-th word, and the j-th column of the image feature matrix represents the feature vector of the j-th image sub-region.

[0032] Optionally, use matrix multiplication to get the attention weight matrix R:

[0033] R=h T W;

[0034] in R i,j Represents the similarity between the i-th word and the j-th image subregion;

[0035] Normalize the attention weight matrix and then dynamically convert word features into image features:

[0036]

[0037]

[0038] Finally, get

[0039] Optionally, the input of the adaptive discriminant network includes a real image, a generated image, and global sentence features; the input image is subjected to a downsampling module to extract image features, and then the image features are passed to two branches; wherein the first branch determines the authenticity of the image, and the second branch determines whether the image content is consistent with the text description;

[0040] Based on the role of the first branch, an adaptive weight module is proposed Get the confidence of the foreground target in the image and add adaptive weights to the discriminant loss function:

[0041] p conf =F score (x);

[0042] The unconditional discriminant loss function of the first branch is:

[0043]

[0044] Among them D i represents the i-th discriminator, x is the image feature, p conf represents the confidence level, (1-p conf ) is the modulation factor, γ is a hyperparameter, and the hyperparameter γ is used to control the degree of effect of the penalty weight;

[0045] In the second branch, the image features extracted by downsampling are first connected with the global text description, and then input into the subsequent convolutional network to obtain the text-image matching probability;

[0046] The conditional discriminant loss function of the second branch is:

[0047]

[0048] The total loss function of the final adaptive discriminant network is:

[0049]

[0050] In a second aspect, the present invention provides a text-to-image model training system, comprising:

[0051] An acquisition module is configured to acquire training data; the training data includes multiple sets of data, each set of data including a matching pair of text description information and a natural image, and a label annotated in the natural image that matches the text description information;

[0052] A training module is used to train the network model using the acquired training data to obtain a trained text-to-image model;

[0053] The network model includes a text encoder, an image encoder, a generative network, and an adaptive discriminant network; the text encoder is used to extract the global sentence features and word feature matrix of the text description information; the image encoder is used to extract the global image features of the natural image, and extract the global image features of the generated image generated by the generative network; the generative network is used to obtain image features based on the global sentence features and word feature matrix, and obtain the generated image based on the image features and fuse the image features to obtain the generated image; the adaptive discriminant network is used to judge the authenticity of the generated image and whether the content of the generated image is consistent with the text description based on the global image features of the natural image and the global image features of the generated image;

[0054] Among them, the generative network includes a cross-scale channel activation module that uses the global information of visual text features in the global sentence features to enhance the image features, thereby obtaining a fused image feature with stronger representation ability.

[0055] In a third aspect, the present invention provides a method for generating an image from text, comprising:

[0056] Get the text description information to be processed;

[0057] The acquired text description information to be processed is processed using a pre-trained text-to-image model to obtain a generated image; wherein the text-to-image model is trained using the model training method described above.

[0058] In a fourth aspect, the present invention provides a text-to-image system, comprising:

[0059] An acquisition module is used to obtain text description information to be processed;

[0060] The processing module is used to process the acquired text description information to be processed using a pre-trained text-to-image model to obtain a generated image; wherein the text-to-image model is trained using the model training method described above.

[0061] In a fifth aspect, the present invention provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the text-to-image method as described above is implemented.

[0062] In a sixth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed on one or more processors, is used to implement the above-described method for generating an image from text.

[0063] In the seventh aspect, the present invention provides an electronic device comprising: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing the text-to-image method as described above.

[0064] Explanation of terms:

[0065] Text-to-image generation: Generate high-resolution, high-quality images that are semantically consistent with a given text description.

[0066] Cross-scale channel activation: using channel activation to fuse features of different scales

[0067] Generative Adversarial Network: A generative adversarial network consists of a pair of generators and discriminators. The generator aims to generate simulated data that conforms to the sample distribution; the discriminator needs to calculate the probability value of the input data being real data.

[0068] The beneficial effects of the present invention are as follows: the visual text features and image features are fused across scales by using channel activation, global semantic information is added to the image features, and the representation of the image features is enhanced; the sample weights in the discriminant loss function can be adaptively modified to realize the mining of difficult samples and improve the image generation quality of difficult samples.

[0069] Additional advantages of the present invention will be more clearly given in the following description or learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0071] Figure 1 This is a structural diagram of the text-to-image network model based on cross-scale channel activation described in an embodiment of the present invention.

[0072] Figure 2 This is a schematic diagram of the structure of the cross-scale channel activation module of the text-to-image network based on cross-scale channel activation according to an embodiment of the present invention.

[0073] Figure 3 Schematic diagram of the structure of the adaptive discriminant network of the text-to-image network based on cross-scale channel activation according to an embodiment of the present invention.

[0074] Figure 4This is a flowchart of the training process of the text-to-image network model based on cross-scale channel activation according to an embodiment of the present invention.

[0075] Figure 5 This is a flowchart for testing the text-to-image network model based on cross-scale channel activation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention.

[0077] Those skilled in the art will understand that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs.

[0078] It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have a meaning consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless as defined herein.

[0079] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.

[0080] In the description of this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and integrate different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless otherwise inconsistent.

[0081] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings, and the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0082] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0083] Example 1

[0084] A text-to-image system based on cross-scale channel activation was developed, and a text-to-image method was implemented using this system. In this system and method, a generative network based on a cross-scale channel activation module fuses low-scale visual text features containing rich global semantic information with image features, preventing the quality of the final generated image from being overly dependent on the generated initial image. A discriminative network based on an adaptive module adds penalizing weights to difficult samples, allowing the discriminative network to focus on learning about these difficult samples and thus achieve mining of these difficult samples.

[0085] In this embodiment 1, a text-to-image generation system is first provided, which includes: an acquisition module for acquiring text description information to be processed; a processing module for processing the acquired text description information to be processed using a pre-trained text-to-image generation model to obtain a generated image.

[0086] In this embodiment 1, the text-to-image model is trained by the following method, including:

[0087] Acquire training data; the training data includes multiple sets of data, each set of data includes matching pairs of text description information and natural images, and annotates labels in the natural images that match the text description information;

[0088] The obtained training data is used to train the network model to obtain a trained text-to-image model;

[0089] The network model includes a text encoder, an image encoder, a generative network, and an adaptive discriminant network; the text encoder is used to extract the global sentence features and word feature matrix of the text description information; the image encoder is used to extract the global image features of the natural image, and extract the global image features of the generated image generated by the generative network; the generative network is used to obtain image features based on the global sentence features and word feature matrix, and obtain the generated image based on the image features and fuse the image features to obtain the generated image; the adaptive discriminant network is used to judge the authenticity of the generated image and whether the content of the generated image is consistent with the text description based on the global image features of the natural image and the global image features of the generated image;

[0090] Among them, the generative network includes a cross-scale channel activation module that uses the global information of visual text features in the global sentence features to enhance the image features, thereby obtaining a fused image feature with stronger representation ability.

[0091] In which, the generation network includes multiple cascaded image feature generators, each of which is connected to an image generator; the first image feature generator is used to extract low-scale visual text features from global sentence features and generate a first image feature, and the first image generator is used to generate a first generated image based on the first image feature; each subsequent image feature generator is used to generate high-scale image features based on the image features and word feature matrix generated by the previous image feature generator, and the cross-scale channel activation module is connected to the corresponding image generator, and the cross-scale channel activation module fuses the high-scale image features and low-scale visual text features generated by the corresponding image feature generator to obtain the fused image feature.

[0092] Specifically, a bidirectional long short-term memory network is used as the text encoder:

[0093] W,s=F LSTM (T)

[0094] in, w l is the feature vector of the lth word, corresponding to the lth hidden layer output, is the output of the last hidden layer, is the long short-term memory network, D is w l and the vector dimensions of s;

[0095] The conditional enhancement module is used to expand the number of text features and enhance the smoothness of text features in high-dimensional space:

[0096] s ca =F ca (s);

[0097] in, D′ is the vector dimension after conditional enhancement;

[0098] Use the pre-trained Inception-v3 model as the image encoder:

[0099] f c ,f v =Inc(I img );

[0100] Among them, I img is the input image, The local features of the image extracted from the image; is the global feature of the image.

[0101] The cross-scale channel activation module consists of a pooling layer, a convolutional network layer, a Sigmoid activation layer, and a ReLU activation layer. The input of the cross-scale channel activation module is high-scale image features and low-scale visual text features. The pooling layer downsamples the visual text features, the convolutional network layer downsamples them, and uses the ReLU activation function to map the input data to a nonlinear space. Then, another convolutional network layer is used to map the latent vector to the same channel as the image feature, and the Sigmoid function is used as the activation function. After that, the two feature vectors are combined using channel multiplication, and the high-scale image features are enhanced using the global information of the visual text features to obtain image features with stronger representation capabilities.

[0102] Attention Module The cross-attention mechanism is used to achieve semantic alignment between word feature matrix and image feature, promoting the consistency between image content and text description; the input of the attention module is the word feature matrix and image features Among them, the i-th column of the word feature matrix represents the feature vector of the i-th word, and the j-th column of the image feature matrix represents the feature vector of the j-th image sub-region. The attention weight matrix R is obtained by matrix multiplication:

[0103] R=h T W;

[0104] in R i,j Represents the similarity between the i-th word and the j-th image subregion;

[0105] Normalize the attention weight matrix and then dynamically convert word features into image features:

[0106]

[0107]

[0108] Finally, get

[0109] The input of the adaptive discriminant network includes real images, generated images, and global sentence features. The input image is processed through a downsampling module to extract image features. After that, the image features are passed to two branches. The first branch determines the authenticity of the image, and the second branch determines whether the image content is consistent with the text description.

[0110] Based on the role of the first branch, an adaptive weight module is proposed Get the confidence of the foreground target in the image and add adaptive weights to the discriminant loss function:

[0111] p conf =F score (x);

[0112] The unconditional discriminant loss function of the first branch is:

[0113]

[0114] Among them D i represents the i-th discriminator, x is the image feature, p conf represents the confidence level, (1-p conf ) is the modulation factor, γ is a hyperparameter, and the hyperparameter γ is used to control the degree of effect of the penalty weight;

[0115] In the second branch, the image features extracted by downsampling are first connected with the global text description, and then input into the subsequent convolutional network to obtain the text-image matching probability;

[0116] The conditional discriminant loss function of the second branch is:

[0117]

[0118] The total loss function of the final adaptive discriminant network is:

[0119]

[0120] Example 2

[0121] In this embodiment 2, a text-to-image model training system is provided, which includes:

[0122] an acquisition module for acquiring training data; the training data includes multiple sets of data, each set of data including matching pairs of text description information and natural images, and labels annotated in the natural images that match the text description information; a training module for training a network model using the acquired training data to obtain a trained text-to-image model;

[0123] The network model includes a text encoder, an image encoder, a generative network, and an adaptive discriminant network; the text encoder is used to extract the global sentence features and word feature matrix of the text description information; the image encoder is used to extract the global image features of the natural image, and extract the global image features of the generated image generated by the generative network; the generative network is used to obtain image features based on the global sentence features and word feature matrix, and obtain the generated image based on the image features and fuse the image features to obtain the generated image; the adaptive discriminant network is used to judge the authenticity of the generated image and whether the content of the generated image is consistent with the text description based on the global image features of the natural image and the global image features of the generated image;

[0124] Among them, the generative network includes a cross-scale channel activation module that uses the global information of visual text features in the global sentence features to enhance the image features, thereby obtaining a fused image feature with stronger representation ability.

[0125] In this embodiment 2, the above-mentioned text-to-image model training system is used to implement a text-to-image model training method, and a text-to-image model based on cross-scale channel activation is obtained.

[0126] like Figure 1 As shown in Figure 1, given a text T, the text encoder will output a global sentence feature vector s and a word feature matrix W. A bidirectional long short-term memory network (Bi-LSTM) is used as the text encoder. This process is formally expressed as:

[0127] W,s=F LSTM (T)

[0128] in w l is the feature vector of the lth word, corresponding to the lth hidden layer output, and is the output of the last hidden layer, is the long short-term memory network, D is w l and the vector dimensions of s.

[0129] Since text features are high-dimensional features and the number of text descriptions is limited, the text features are discontinuous in the high-dimensional feature space, which reduces the training effect of the generative network. To address this problem, this embodiment uses a conditional enhancement module to expand the number of text features, thereby enhancing the smoothness of text features in the high-dimensional space. The formula is as follows: ca =F ca (s); where D′ is the dimension of the vector after conditional enhancement.

[0130] In this example, the pre-trained Inception-v3 model is used as the image encoder. This model is a convolutional neural network model. It is formally expressed as follows:

[0131] f c ,f v =Inc(I img )

[0132] Among them I img is the input image, The local features of the image extracted from the image are obtained by the middle layer of the model. The global image features are obtained by the last layer of the model. First, the input image is resized to a resolution of 299×299. Then, the image features of the “mixed-6e” sublayer of the Inception-v3 model are extracted as the local image features. The local image features f c Each column is the feature vector of the image sub-region. Finally, the output of the last pooling layer of the Inception-v3 model is used as the global feature of the image.

[0133] The cross-scale channel activation module fuses low-scale visual text features containing rich global semantic information with image features to avoid the quality of the final image generation being overly dependent on the generation results of the initial image. The cross-scale channel activation module proposed in this embodiment has a module structure as follows Figure 2 shown.

[0134] The cross-scale channel activation module consists of a pooling layer, a convolutional network layer, a Sigmoid activation layer, and a ReLU activation layer. The input of the cross-scale channel activation module is a high-scale image feature h and a low-scale visual text feature U. Among them, the scale of the image feature h is 128×128, and the scale of the visual text feature U is 8×8. Specifically, the visual text feature is first downsampled to 4×4 using an adaptive average pooling layer. Secondly, it is downsampled to 1×1 using a convolutional network layer, and the input data is mapped to a nonlinear space using the ReLU activation function. Then, another convolutional network layer is used to map the latent vector to be consistent with the image feature channel, and the Sigmoid function is used as the activation function. Afterwards, the two feature vectors are combined using channel multiplication, and the high-scale image features are enhanced using the global information of the visual text features to obtain image features with stronger representation capabilities.

[0135] Attention Module The cross attention mechanism is used to achieve semantic alignment between word feature matrix and image feature, promoting the consistency between image content and text description. The input of the attention module is the word feature matrix and image features Among them, the i-th column of the word feature matrix represents the feature vector of the i-th word, and the j-th column of the image feature matrix represents the feature vector of the j-th image subregion. Specifically, we first use matrix multiplication to obtain the attention weight matrix R, which is formally expressed as follows: R = h T W; R i,j Represents the similarity between the i-th word and the j-th image subregion. After that, the attention weight matrix is ​​normalized and the word features are dynamically converted into image features as follows:

[0136]

[0137]

[0138] Finally, you can get

[0139] The generator and discriminator of the generative adversarial network learn from each other through game play, where the discriminator is the main source of gradient learning for the generator. In order to solve the problem that existing methods lack the ability to mine difficult samples, this embodiment proposes an adaptive discriminant module. By adding adaptive weights to the discriminant loss function, it encourages the generative network to focus on learning difficult samples and thus achieves the mining of difficult samples. Figure 3 shown.

[0140] The discriminative network inputs include real images, generated images, and global sentence features. Specifically, the input image is processed through a downsampling module, which consists of four downsampling layers, to extract image features. The image features are then passed to two branches. The first branch determines the authenticity of the image, while the second branch determines whether the image content is consistent with the text description.

[0141] Based on the role of the first branch, an adaptive weight module is proposed Obtain the confidence of the foreground target in the image and add adaptive weights to the discriminant loss function. This process is formulated as follows:

[0142] p conf =F score (x)

[0143] The target detection model used is the YOLOv3 model, which has a fast detection speed and has little impact on the network training speed. The unconditional discriminant loss function of the first branch is as follows:

[0144]

[0145] Among them D i represents the i-th discriminator, x is the image feature, p conf represents the confidence level, (1-p conf ) is the modulation factor, and γ is the hyperparameter. In the initial training stage, the confidence level p of the image conf = 0, the modulation factor is 1, and the effect on the discriminant loss function is negligible, which avoids the problem that the initial generated image quality is poor and is misjudged as a difficult sample. As the model trains, the image quality of the simple sample gradually improves, and the confidence level p conf→1, the modulation factor tends to 0, thereby reducing the weight of the sample in the discriminant loss function. For difficult samples, whose image generation quality is still poor and whose confidence is low, the sample still has a large weight in the discriminant loss function, prompting the discriminant model to focus on learning difficult samples. The hyperparameter γ is used to control the degree of the penalty weight.

[0146] In the second branch, the image features extracted by downsampling are first connected with the global text description, and then input into the subsequent convolutional network to obtain the text-image matching probability. The conditional discriminant loss function of the second branch is:

[0147]

[0148] To sum up, the loss function of the final discriminant network is as follows:

[0149]

[0150] The generation loss function of the CSCA-GAN model proposed in the network training process of this embodiment consists of three terms, as shown in the following formula:

[0151]

[0152] Among them, G i represents the i-th generator, To generate the loss function, L ca is the conditional enhancement loss function in the StackGAN++ model, L DAMSM is the DAMSM loss function in the AttnGAN model, L c is the color consistency loss function. λ1, λ2, λ3 are the weights of the corresponding loss function. The first term is the generation loss function, and its expansion is as follows:

[0153]

[0154] Among them, D i Denotes the i-th discriminator. The first term of this equation is the unconditional generative loss function, which encourages the generator to produce high-definition, realistic images to confuse the discriminator. The second term is the conditional generative loss function, which introduces the global sentence feature s to promote semantic consistency between the generated image and the text description.

[0155] Since the generative network adopts a multi-level generative network structure, the structure generates images of different resolutions through a cascade of multiple generators. In order to avoid the problem of inconsistent colors of images with different resolutions, the present invention introduces a color-consistency regularization term. The color consistency regularization term uses the mean square error loss function to reduce the difference in mean and variance between images of different resolutions, thereby improving the quality of the final image. The expansion is as follows:

[0156]

[0157] Where n is the batch size, and are the mean and variance of the jth image generated in the i-th generation stage, and β1 and β2 are the hyperparameters of the loss function.

[0158] The loss function of the discriminator has been introduced in 4.1(4). Figure 4 The training process of the network.

[0159] The specific training process is as follows: 1) Read the matching pairs of text descriptions and natural images. 2) Use the text encoder and image encoder to extract text features and image features from the text descriptions and natural images. 3) Forward propagation. The extracted text features are input into the following Figure 1 4) Calculate the loss of the generative network. 5) Back-propagate and update the parameters of the generative network. Adam is used as the optimizer for gradient propagation to update the parameters of the generative model. 6) Calculate the loss of the discriminative network. 7) Back-propagate and update the parameters of the discriminative network. Adam is used as the optimizer for gradient propagation to update the parameters of the discriminative model. 8) Post-processing, such as saving model parameters and outputting training results.

[0160] For the testing phase of the network, such as Figure 5 The test flow chart of the network is as follows: 1) Read the matching pairs of text description and natural image. 2) Use the pre-trained text encoder and image encoder to extract text features and natural image features from the text description and natural image respectively. 3) Input the extracted text features into Figure 1In the network shown, the generator generates the corresponding generated image. 4) The image encoder is used again to extract generated image features from the generated image. 5) The generated image features are dynamically converted into visual text features using the attention mechanism, and the cosine similarity is used to calculate the similarity between the visual text features and the text features, that is, the matching degree between the generated image and the text description. 6) The IS (inception score, IS) and FID (Fréchet inception distance, FID) scores are calculated for the generated image features and the natural image features.

[0161] Example 3

[0162] In this Example 3, a text generation image algorithm CSCA-GAN based on cross-scale channel activation is proposed. The overall framework of the algorithm is as follows: Figure 1 As shown. The CSCA-GAN model aims to generate l |l=0,1,…,L-1) semantic matching image X={x i |i=0,1,2}. Where L represents the length of the text description and i represents the generation stage of different resolutions.

[0163] The CSCA-GAN model consists of four parts: a text encoder, an image encoder, a generative network, and an adaptive discriminative network. The generative network includes multiple cascaded image feature generators, each of which is connected to an image generator. The first image feature generator is used to extract low-scale visual text features from global sentence features and generate first image features. The first image generator is used to generate a first generated image based on the first image features. Each subsequent image feature generator is used to generate high-scale image features based on the image features generated by the previous image feature generator and the word feature matrix. Each image feature generator is connected to the corresponding image generator via a cross-scale channel activation module, which fuses the high-scale image features and low-scale visual text features generated by the corresponding image feature generator to obtain the fused image features.

[0164] In this embodiment, the generation network includes three cascaded image feature generators, which is a three-level generation network. The three-level generation network includes generation modules and cross-scale channel activation module First, the text description T is input into the pre-trained text encoder to extract the global sentence feature s and word feature matrix W. Then, the global sentence feature s is input into the conditional enhancement module. Get the global sentence feature s caThen the randomly sampled noise feature z is connected with the sentence feature and passed into the three-level generative adversarial network. The sentence feature is used to obtain the image feature h through the generation module. i Among them, the low-scale visual text features U1, U2 are extracted in the first generation module (the first image feature generator). Then the image feature h i With visual text features U i Enter the cross-scale channel activation module to obtain image features This module adds global semantic information to the image features. Then the obtained image feature h0 is Pass them into generators G0, G1, and G2 in order to get the generated image x i (i=0,1,2). The adaptive discriminant network contains the discriminator D i (i=0,1,2) and target detection module Generate image x i Input into the adaptive weight module to obtain the pseudo label p conf , and pass the pseudo label into the discriminator D i In the algorithm, the weights of difficult samples in the loss function are adaptively modified to achieve the mining of difficult samples.

[0165] like Figure 1 As shown in Figure 1, given a text T, the text encoder will output a global sentence feature vector s and a word feature matrix W. In this embodiment, a bidirectional long short-term memory network (Bi-directional Long Short Term Memory, Bi-LSTM) is used as the text encoder. This process is formally expressed as:

[0166] W,s=F LSTM (T) (1)

[0167] in w l is the feature vector of the lth word, corresponding to the lth hidden layer output, and is the output of the last hidden layer, is the long short-term memory network, D is w l and the vector dimensions of s.

[0168] Because text features are high-dimensional and the number of text descriptions is limited, they are discontinuous in the high-dimensional feature space, which reduces the training effect of the generative network. To address this problem, a conditional enhancement module is used to expand the number of text features and thus enhance the smoothness of text features in the high-dimensional space. The formula is as follows:

[0169] s ca =F ca (s) (2)

[0170] in D′ is the dimension of the vector after conditional enhancement.

[0171] In this example, the pre-trained Inception-v3 model is used as the image encoder. This model is a convolutional neural network model. It is formally expressed as follows:

[0172] f c ,f v =Inc(I img ) (3)

[0173] Among them I img is the input image, The local features of the image extracted from the image are obtained by the middle layer of the model. The global image features are obtained by the last layer of the model. First, the input image is resized to a resolution of 299×299. Then, the image features of the “mixed-6e” sublayer of the Inception-v3 model are extracted as the local image features. The local image features f c Each column is the feature vector of the image sub-region. Finally, the output of the last pooling layer of the Inception-v3 model is used as the global feature of the image.

[0174] The cross-scale channel activation module fuses low-scale visual text features containing rich global semantic information with image features to avoid the quality of the final image generation being overly dependent on the generation results of the initial image. The present invention proposes a cross-scale channel activation module, the module structure is as follows Figure 2 shown.

[0175] The cross-scale channel activation module consists of a pooling layer, a convolutional network layer, a Sigmoid activation layer, and a ReLU activation layer. The input of the cross-scale channel activation module is a high-scale image feature h and a low-scale visual text feature U. Among them, the scale of the image feature h is 128×128, and the scale of the visual text feature U is 8×8. Specifically, the visual text feature is first downsampled to 4×4 using an adaptive average pooling layer. Secondly, it is downsampled to 1×1 using a convolutional network layer, and the input data is mapped to a nonlinear space using the ReLU activation function. Then, another convolutional network layer is used to map the latent vector to be consistent with the image feature channel, and the Sigmoid function is used as the activation function. Afterwards, the two feature vectors are combined using channel multiplication, and the high-scale image features are enhanced using the global information of the visual text features to obtain image features with stronger representation capabilities.

[0176] Attention Module The cross attention mechanism is used to achieve semantic alignment between word feature matrix and image feature, promoting the consistency between image content and text description. The input of the attention module is the word feature matrix and image features Among them, the i-th column of the word feature matrix represents the feature vector of the i-th word, and the j-th column of the image feature matrix represents the feature vector of the j-th image subregion. Specifically, we first use matrix multiplication to obtain the attention weight matrix R, which is formally expressed as follows:

[0177] R=h T W (4)

[0178] in R i,j Represents the similarity between the i-th word and the j-th image subregion. After that, the attention weight matrix is ​​normalized and the word features are dynamically converted into image features as follows:

[0179]

[0180]

[0181] Finally, you can get

[0182] For the adaptive discriminant network, the generator and discriminator of the generative adversarial network learn from each other through game, in which the discriminator is the main source of the generator's gradient learning. In order to solve the problem that the existing methods lack the ability to mine difficult samples, this embodiment proposes an adaptive discriminant module, which adds adaptive weights to the discriminant loss function to encourage the generator network to focus on learning difficult samples and thus achieve mining of difficult samples. Figure 3 shown.

[0183] The discriminative network inputs include real images, generated images, and global sentence features. Specifically, the input image is processed through a downsampling module, which consists of four downsampling layers, to extract image features. The image features are then passed to two branches. The first branch determines the authenticity of the image, while the second branch determines whether the image content is consistent with the text description.

[0184] Based on the role of the first branch, an adaptive weight module is proposed Obtain the confidence of the foreground target in the image and add adaptive weights to the discriminant loss function. This process is formulated as follows:

[0185] p conf =F score (x) (7)

[0186] The target detection model used is the YOLOv3 model, which has a fast detection speed and has little impact on the network training speed. The unconditional discriminant loss function of the first branch is as follows:

[0187]

[0188] Among them D i represents the i-th discriminator, x is the image feature, p conf represents the confidence level, (1-p conf ) is the modulation factor, and γ is the hyperparameter. In the initial training stage, the confidence level p of the image conf = 0, the modulation factor is 1, and the effect on the discriminant loss function is negligible, which avoids the problem that the initial generated image quality is poor and is misjudged as a difficult sample. As the model trains, the image quality of the simple sample gradually improves, and the confidence level p conf →1, the modulation factor tends to 0, thereby reducing the weight of the sample in the discriminant loss function. For difficult samples, whose image generation quality is still poor and whose confidence is low, the sample still has a large weight in the discriminant loss function, prompting the discriminant model to focus on learning difficult samples. The hyperparameter γ is used to control the degree of the penalty weight.

[0189] In the second branch, the image features extracted by downsampling are first connected with the global text description, and then input into the subsequent convolutional network to obtain the text-image matching probability.

[0190] The conditional discriminant loss function of the second branch is as follows:

[0191]

[0192] To sum up, the loss function of the final discriminant network is as follows:

[0193]

[0194] In this embodiment, the generation loss function of the proposed CSCA-GAN model is as follows:

[0195]

[0196] Among them, G i represents the i-th generator, To generate the loss function, L ca is the conditional enhancement loss function in the StackGAN++ model, L DAMSM is the DAMSM loss function in the AttnGAN model, L c is the color consistency loss function. λ1, λ2, λ3 are the weights of the corresponding loss function. The first term is the generation loss function, and its expansion is as follows:

[0197]

[0198] Among them, D i Denotes the i-th discriminator. The first term of this equation is the unconditional generative loss function, which encourages the generator to produce high-definition, realistic images to confuse the discriminator. The second term is the conditional generative loss function, which introduces the global sentence feature s to promote semantic consistency between the generated image and the text description.

[0199] Since the generative network adopts a multi-level generative network structure, the structure generates images of different resolutions through multiple cascaded generators. In order to avoid the problem of inconsistent colors of images with different resolutions, the present invention introduces a color-consistency regularization term. The color consistency regularization term uses the mean square error loss function to reduce the difference in mean and variance between images of different resolutions, thereby improving the quality of the final image. The expansion is as follows:

[0200]

[0201] Where n is the batch size, and are the mean and variance of the jth image generated in the i-th generation stage, and β1 and β2 are the hyperparameters of the loss function.

[0202] The loss function of the discriminator has been introduced in 4.1(4). Figure 4 The training process of the network.

[0203] The specific training process is as follows: 1) Read the matching pairs of text descriptions and natural images. The text descriptions need to be preprocessed, and only 18 text words are extracted as text input, and the natural images are transformed into a size of 299×299. 2) Use the text encoder and image encoder to extract text features and image features from the text descriptions and natural images. 3) Forward propagation. The extracted text features are input as follows Figure 1The propagation calculation is performed in the network shown to obtain the generated image. Due to the characteristics of the generative adversarial network, alternating training is adopted in this embodiment, that is, the generative network and the discriminative network are trained alternately. 4) Calculate the loss of the generative network. The loss of the generative network is calculated by formula (11). 5) Back propagation, update the parameters of the generative network. Adam is used as the optimizer for gradient propagation, update the parameters of the generative model, and the learning rate is set to 0.0002. 6) Calculate the loss of the discriminative network. The loss of the discriminative network is calculated by formula (10). 7) Back propagation, update the parameters of the discriminative network. Adam is used as the optimizer for gradient propagation, update the parameters of the discriminative model, and the learning rate is set to 0.0002. 8) Post-processing. Such as saving model parameters, outputting training results, etc.

[0204] like Figure 5 The network test flow chart is as follows: 1) Read the matching pairs of text descriptions and natural images. The text descriptions need to be preprocessed, and only 18 text words are extracted as text input, and the natural images are transformed into a size of 299×299. 2) Use the pre-trained text encoder and image encoder to extract text features and natural image features from the text description and natural image respectively. 3) Input the extracted text features into Figure 1 In the network shown, the generator generates the corresponding generated image. 4) The image encoder is used again to extract generated image features from the generated image. 5) The generated image features are dynamically converted into visual text features using the attention mechanism, and the cosine similarity is used to calculate the similarity between the visual text features and the text features, that is, the matching degree between the generated image and the text description. 6) The IS (inception score) and FID (Fréchet inception distance) scores are calculated using the generated image features and the natural image features.

[0205] Example 4

[0206] This embodiment 4 provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, a text-to-image generation method based on cross-scale channel activation is implemented.

[0207] Example 5

[0208] This embodiment 5 provides a computer program product, including a computer program, which, when running on one or more processors, is used to implement a method for generating images from text based on cross-scale channel activation.

[0209] Example 6

[0210] This embodiment 6 provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing a method for generating images from text based on cross-scale channel activation.

[0211] In summary, the present invention describes a method for generating images from text based on cross-scale channel activation (Cross-Scale Channel Activation Generative Adversarial Networks for Text to Image Generation, CSCA-GAN). This algorithm combines visual text features containing rich semantic information with image features across scales, reducing the dependence of the final generated image on the initial image and ensuring good consistency between the generated image and the text description. In addition, CSCA-GAN uses an object detection mechanism to mine difficult samples among the generated samples, allowing the generative network to focus more on learning difficult samples, thereby generating higher-quality images. Specifically, CSCA-GAN mainly consists of two modules: the Cross-Scale Channel Activation Module (CSCAM) and the Adaptive Weighting Module (AWM). From the perspective of the generative network, CSCAM uses channel activation to combine visual text features with generated image features across scales, adding global semantic information to the image features, making the high-resolution generated image not only dependent on the initial image, and improving the semantic consistency between the text description and the generated image. AWM, from the perspective of the discriminant network, utilizes object detection mechanisms to identify foreground objects in the generated image and obtain confidence scores for these foreground objects. It then uses the confidence scores as pseudo-labels to adaptively change the sample weights in the loss function, guiding the discriminant model to learn from difficult samples, thereby mining these difficult samples and improving the quality of the generated images.

[0212] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solutions disclosed in the present invention without the need for creative work should be included in the scope of protection of the present invention.

Claims

1. A text-to-image model training method, characterized in that: include: Acquire training data; the training data includes multiple sets of data, each set of data includes matching pairs of text description information and natural images, and annotates labels in the natural images that match the text description information; The obtained training data is used to train the network model to obtain a trained text-to-image model; The network model includes a text encoder, an image encoder, a generative network, and an adaptive discriminant network; the text encoder is used to extract the global sentence features and word feature matrix of the text description information; the image encoder is used to extract the global image features of the natural image, and extract the global image features of the generated image generated by the generative network; the generative network is used to obtain image features based on the global sentence features and word feature matrix, and obtain the generated image based on the image features and fuse the image features to obtain the generated image; the adaptive discriminant network is used to judge the authenticity of the generated image and whether the content of the generated image is consistent with the text description based on the global image features of the natural image and the global image features of the generated image; The generative network includes a cross-scale channel activation module that uses the global information of visual text features in the global sentence features to enhance image features, thereby obtaining fused image features with stronger representation capabilities. In which, the generation network includes multiple cascaded image feature generators, each of which is connected to an image generator; the first image feature generator is used to extract low-scale visual text features from global sentence features and generate a first image feature, and the first image generator is used to generate a first generated image based on the first image feature; each subsequent image feature generator is used to generate high-scale image features based on the image features and word feature matrix generated by the previous image feature generator, and the cross-scale channel activation module is connected to the corresponding image generator, and the cross-scale channel activation module fuses the high-scale image features and low-scale visual text features generated by the corresponding image feature generator to obtain the fused image feature.

2. The text-to-image model training method according to claim 1, characterized in that: Use bidirectional long short-term memory network as text encoder: in, w l is the feature vector of the lth word, corresponding to the lth hidden layer output, is the output of the last hidden layer, is the long short-term memory network, D is w l and the vector dimensions of s; The conditional enhancement module is used to expand the number of text features and enhance the smoothness of text features in high-dimensional space: in, D′ is the vector dimension after conditional enhancement; Use the pre-trained Inception-v3 model as the image encoder: f c ,f v =Inc(I img ); Among them, I img is the input image, The local features of the image extracted from the image; is the global feature of the image.

3. The text-to-image model training method according to claim 1, wherein: The cross-scale channel activation module consists of a pooling layer, a convolutional network layer, a Sigmoid activation layer, and a ReLU activation layer. The input of the cross-scale channel activation module is high-scale image features and low-scale visual text features. The pooling layer downsamples the visual text features, the convolutional network layer downsamples them, and uses the ReLU activation function to map the input data to a nonlinear space. Then, another convolutional network layer is used to map the latent vector to the same channel as the image feature, and the Sigmoid function is used as the activation function. After that, the two feature vectors are combined using channel multiplication, and the high-scale image features are enhanced using the global information of the visual text features to obtain image features with stronger representation capabilities.

4. The text-to-image model training method according to claim 3, characterized in that: Attention Module The cross-attention mechanism is used to achieve semantic alignment between word feature matrix and image feature, promoting the consistency between image content and text description; the input of the attention module is the word feature matrix and image features Among them, the i-th column of the word feature matrix represents the feature vector of the i-th word, and the j-th column of the image feature matrix represents the feature vector of the j-th image sub-region.

5. The text-to-image model training method according to claim 4, characterized in that: Use matrix multiplication to get the attention weight matrix R: R=h T W; in R i,j Represents the similarity between the i-th word and the j-th image subregion; Normalize the attention weight matrix and then dynamically convert word features into image features: Finally, get 6. The text-to-image model training method according to claim 1, characterized in that: The input of the adaptive discriminant network includes real images, generated images and global sentence features; The input image is processed through a downsampling module to extract image features. After that, the image features are passed to two branches. The first branch determines the authenticity of the image, while the second branch determines whether the image content is consistent with the text description. Based on the role of the first branch, an adaptive weight module is proposed Get the confidence of the foreground target in the image and add adaptive weights to the discriminant loss function: The unconditional discriminant loss function of the first branch is: Among them D i represents the i-th discriminator, x is the image feature, p conf represents the confidence level, (1-p conf ) is the modulation factor, γ is a hyperparameter, and the hyperparameter γ is used to control the degree of effect of the penalty weight; In the second branch, the image features extracted by downsampling are first connected with the global text description, and then input into the subsequent convolutional network to obtain the text-image matching probability; The conditional discriminant loss function of the second branch is: The total loss function of the final adaptive discriminant network is:

7. A text-to-image model training system, characterized in that: include: An acquisition module is configured to acquire training data; the training data includes multiple sets of data, each set of data including a matching pair of text description information and a natural image, and a label annotated in the natural image that matches the text description information; A training module is used to train the network model using the acquired training data to obtain a trained text-to-image model; The network model includes a text encoder, an image encoder, a generative network, and an adaptive discriminant network; the text encoder is used to extract the global sentence features and word feature matrix of the text description information; the image encoder is used to extract the global image features of the natural image, and extract the global image features of the generated image generated by the generative network; the generative network is used to obtain image features based on the global sentence features and word feature matrix, and obtain the generated image based on the image features and fuse the image features to obtain the generated image; the adaptive discriminant network is used to judge the authenticity of the generated image and whether the content of the generated image is consistent with the text description based on the global image features of the natural image and the global image features of the generated image; The generative network includes a cross-scale channel activation module that uses the global information of visual text features in the global sentence features to enhance image features, thereby obtaining fused image features with stronger representation capabilities. In which, the generation network includes multiple cascaded image feature generators, each of which is connected to an image generator; the first image feature generator is used to extract low-scale visual text features from global sentence features and generate a first image feature, and the first image generator is used to generate a first generated image based on the first image feature; each subsequent image feature generator is used to generate high-scale image features based on the image features and word feature matrix generated by the previous image feature generator, and the cross-scale channel activation module is connected to the corresponding image generator, and the cross-scale channel activation module fuses the high-scale image features and low-scale visual text features generated by the corresponding image feature generator to obtain the fused image feature.

8. A method for generating an image from text, characterized in that: include: Get the text description information to be processed; The acquired text description information to be processed is processed using a pre-trained text-to-image model to obtain a generated image; wherein the text-to-image model is trained using the model training method according to any one of claims 1-6.

9. A text-to-image system, characterized in that: include: An acquisition module is used to obtain text description information to be processed; A processing module is used to process the acquired text description information to be processed using a pre-trained text-to-image model to obtain a generated image; wherein the text-to-image model is trained using the model training method described in any one of claims 1-6.

Images