Common sense-guided text-to-image generation method and device

Abstract

The present invention discloses a common sense-guided text-to-image generation method and device. By adding rich common sense knowledge descriptions step by step and one by one to upsampling fusion blocks, a semantic mapping relationship between the knowledge description and the generated image area is established, the visual authenticity of the generated image area is enhanced, and finally a high-quality image is synthesized by fusing all image areas. The method includes: retrieving relevant common sense triples from the ConceptNet external common sense library and filtering them; performing scene graph parsing on the input description sentence to obtain scene triples; inputting the original sentence and knowledge description into the generator of the generative adversarial network to generate an image; the discriminator calculates the adversarial loss and updates the parameters of the generator network; the generator and the discriminator are alternately optimized to finally generate the best quality image for the field of intelligent generation of news content. The device includes: a processor and a memory.

Description

Technical Field

[0001] The present invention relates to the field of image generation, and in particular to a common sense-guided text-to-image generation method and device. Background Art

[0002] In recent years, with the development of artificial intelligence, generating a related image from a given text has received increasing attention. [1 , 2] Research on text-derived images is of great significance and has great potential applications, such as computer-aided design and image editing. Text-derived image tasks require not only realistic visuals but also semantic consistency with the given text description. However, due to the significant gap between text and image modalities, text-derived image tasks are extremely challenging, making it difficult to ensure that the generated images are consistent with the given text, both globally and locally.

[0003] Most of the current methods based on generative adversarial networks [3] The image generation process is constrained and adjusted based on the input of a single text sentence. However, the abstract and information-limited nature of natural language makes the image generation process difficult. On the one hand, the information covered by a single text is relatively limited, so it is difficult to fully describe a visually realistic image. On the other hand, natural language is abstract. For descriptive sentences containing multiple objects and complex scene relationships, it is difficult for the generator to grasp the scene relationships of multiple objects. For example, many text-to-graph methods based on adversarial generative networks can achieve relatively satisfactory results on single-target datasets, but the results on multi-target datasets with complex scene relationships are unsatisfactory. SttackGAN [4] 、AttnGAN [5] DM-GAN [6] Classic generative adversarial network models, such as these, use a stacked network architecture and, constrained by a single input text, achieve multi-resolution output. The resulting image is a 256x256 high-resolution image, but the generated image lacks semantic consistency with the input sentence in terms of local details. For example, given an input image of "a brown bird with black and white striped wings," the generated image may depict a brown bird overall, but without the black and white stripes on its wings.

[0004] In order to make up for the shortcoming of limited information content of a single input text, MA-GAN [7] , RIFE-GAN [8]Models such as

[15] utilize multiple input sentences to generate images. These methods use multiple input sentences as knowledge descriptions and input them into the generative network for semantic enrichment. However, this knowledge enrichment approach presents two problems: First, these models require fusing multiple knowledge descriptions into a single feature vector representing the knowledge information, which is then fed into the model for semantic enrichment. However, this feature fusion approach complicates the model's understanding of the knowledge descriptions. Each upsampling process uses the same fused features, making it difficult for the model to map each knowledge description to the generated image. Second, these knowledge enrichment models fail to utilize knowledge outside of the dataset, resulting in limited semantic enrichment using knowledge descriptions. The resulting generated images often lack semantic consistency with the input sentences in terms of local details, significantly impacting their practical application value. For example, in computer-aided design (CAD), such as fashion design, designers seeking inspiration from AI-generated clothing images must ensure that the generated clothing images are consistent with the description text in terms of local details, rather than simply depicting the general outline of the garment.

[0005] Therefore, there is an urgent need to propose a new method, which is to extract richer and more effective knowledge information to enrich semantics, and to efficiently use knowledge descriptions to generate high-quality and semantically consistent images. Summary of the Invention

[0006] The present invention provides a common-sense-guided text-to-image generation method and device. By gradually and individually adding rich common-sense knowledge descriptions to an upsampling fusion module, the present invention establishes a semantic mapping relationship between the knowledge descriptions and the generated image regions, enhancing the visual realism of the generated image regions. Finally, by fusing all image regions to synthesize a high-quality image, the present invention achieves the generation and conversion of input text into high-quality, highly semantically consistent images. The generated images are not only globally semantically consistent with the input text, but also exhibit good semantic consistency in local image regions corresponding to fine-grained vocabulary, thereby improving the practical application value of the generated images. Details are described below:

[0007] In a first aspect, a common sense-guided text-to-image generation method is provided, the method comprising:

[0008] Build a priori knowledge base and select text descriptions related to the input sentence to enrich the single text input;

[0009] Extract the nouns in the input sentence, use each noun as a condition, search for relevant common sense triples in the ConceptNet external common sense database, and filter the common sense triples;

[0010] The input description sentence is parsed into a scene graph to obtain several scene triples consisting of <subject-verb-object> and use them as structured knowledge;

[0011] Input the original sentence and knowledge description into the generator of the generative adversarial network to generate an image;

[0012] The discriminator uses the synthesized image and the corresponding text vector as input to calculate the adversarial loss and update the parameters of the generator network. The generator and discriminator are optimized alternately, and the generator ultimately generates the best quality image, which is used in the field of intelligent news content generation.

[0013] In a second aspect, a common sense-guided text-to-image generation device is provided, comprising: a processor and a memory, wherein the processor calls program instructions stored in the memory to enable the device to execute any one of the method steps described in the first aspect.

[0014] A third aspect is a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes any one of the method steps described in the first aspect.

[0015] The prior knowledge base constructed by the present invention is composed of manual annotations in the data set, and these manual annotations are descriptions of the images made by the annotation workers based on their daily common sense. They are common sense descriptions based on common sense knowledge. On the other hand, the scene triples are extracted from the common sense manual annotations and naturally carry common sense information. Therefore, the present invention refers to the prior knowledge, scene triple knowledge and common sense extracted from the ConceptNet external common sense library as common sense knowledge in a broad category. The present invention is based on a common sense guided generative adversarial network, which uses prior knowledge and external knowledge to enrich the semantic information of a single text input and uses scene knowledge to perform fine-grained scene division on the image. The knowledge is added step by step and one by one to the upsampling knowledge fusion block, so that the generator can focus on the relevant image area in a step-by-step manner and thus improve the quality of the image area. It has the following advantages:

[0016] 1. The present invention constructs a priori knowledge base consisting of manually annotated sentences from a dataset, and selects and inputs other text annotations corresponding to the text describing the image as a supplement to the single text. The other text selected from the priori knowledge base can describe the image from different angles without focusing on a particular aspect. Compared with a single text input, the multi-text sentence description of the image scene is more comprehensive and detailed, compensating for the shortcoming of the limited information content of a single text.

[0017] 2. This method extracts nouns from the input sentence, uses each noun as a condition, searches for related common sense triples in the ConceptNet external common sense database, and performs secondary filtering and denoising on the common sense triples. This can provide knowledge information that does not exist in the prior knowledge database, further compensating for the limited information content of a single text.

[0018] 3. The present invention performs scene graph parsing on the input description sentence, parsing the input sentence into several scene triples consisting of <subject-verb-object>. The predicate in the scene triple describes the scene relationship between the subject and object, and is a fine-grained scene unit. The present invention uses the extracted scene triples as structured knowledge to specifically and singly divide the multi-object scene relationship in the image, which can help the generator more clearly understand the complex scene relationship in the image, making up for the shortcomings of natural language abstraction and the difficulty in understanding multi-object scene relationships.

[0019] 4. This invention establishes a semantic mapping relationship between the knowledge description and the generated image region by adding rich common sense knowledge descriptions step by step and one by one to the upsampling fusion module, thereby enhancing the visual authenticity of the generated image region. Finally, by fusing all image regions to synthesize a high-quality image, this method realizes the generation and conversion from input text to high-quality, highly semantically consistent images, effectively improving the semantic consistency between fine-grained description vocabulary and local images, so that the generated images better meet the diverse and fine-grained needs of users, and enhances the practical application value of the generated images.

[0020] 5. The present invention applies the generated images to the field of intelligent generation of news content, so that news readers have an intuitive visual experience based on the text news content, enhancing the interest, readability and comprehensibility of the text news content. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Flowchart of the common sense-guided text-to-image generation method;

[0022] Figure 2 A schematic diagram of the process of extracting knowledge from input text;

[0023] Figure 3 This is the overall framework for common sense-guided text-to-image generation methods. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0025] Example 1

[0026] The embodiment of the present invention provides a method for generating text to image based on common sense guidance, see Figure 1 , the method comprises the following steps:

[0027] 101: Build a priori knowledge base and select text descriptions related to the input sentence to enrich the single text input;

[0028] 102: Extract the nouns in the input sentence, use each noun as a condition, search for relevant common sense triples in the ConceptNet external common sense database, and filter the common sense triples;

[0029] 103: Perform scene graph parsing on the input description sentence to obtain several scene triples consisting of <subject-verb-object> and use them as structured knowledge;

[0030] 104: Input the original sentence and knowledge description into the generator of the generative adversarial network to generate an image;

[0031] 105: The discriminator uses the synthetic image and the corresponding text vector as input to calculate the adversarial loss and update the parameters of the generator network; the generator and discriminator are optimized alternately, and the generator ultimately produces the best quality image, which is used in the field of intelligent news content generation.

[0032] In summary, the image generated by the above steps 101-105 in the embodiment of the present invention is not only consistent with the input text in terms of global semantics, but also shows good semantic consistency in the local image area corresponding to the fine-grained vocabulary, thereby improving the practical application value of the generated image, allowing news readers to have an intuitive visual experience based on the text news content, and enhancing the interest, readability, and comprehensibility of the text news content.

[0033] Example 2

[0034] The following combination Figure 1-Figure 3 , specific calculation formulas, examples are given to further introduce the scheme in Example 1, as described below:

[0035] 201: Input a sentence describing an image and encode the description sentence into a sentence feature vector;

[0036] Among them, for a given text input, a text encoder is used to encode it to obtain a feature representation of the text description. The text encoder used in the embodiment of the present invention is a bidirectional LSTM (Long Short-Term Memory Network), which minimizes the loss of the Deep Attention Multimodal Similarity Model (DAMSM) and uses real image-text pairs for pre-training. The embodiment of the present invention encodes a given text sentence into a 256-dimensional sentence feature vector through the text encoder. The embodiment of the present invention does not elaborate on this in detail during the specific implementation.

[0037] 202: Build a priori knowledge base and select text descriptions related to the input sentence to enrich the single text input;

[0038] Due to the limited information content of natural language, a single text sentence often only describes a portion of an image, failing to provide a comprehensive and detailed description. This limited information content leads to information gaps in image generation, hindering the generator from producing semantically consistent images. To address this issue, this method constructs a priori knowledge base consisting of annotated text sentences from a dataset. For a given input text sentence, relevant text descriptions are selected from the priori knowledge base to enrich the single text input.

[0039] The specific construction method of the prior knowledge base is as follows: In the commonly used datasets for text-to-image generation tasks, one image corresponds to multiple manually annotated sentences that describe the image from different angles. Although these manual annotations describe the same image, they focus on different aspects and describe the image from different angles. Using multiple manual annotations can more comprehensively describe an image and enrich the semantic information of a single input sentence. For an image I i , and its corresponding manual annotation is Where N represents the image I i The number of corresponding text description sentences in the dataset, t i,k For image I i The corresponding k-th text annotation. This method combines all text description sentences in the dataset into a priori knowledge base. For a given text input, different description sentences for the same image are used as supplements to the input sentence to enrich the input semantic information. The specific process is as follows:

[0040] Given input text Select 6 and Different manually annotated sentences As a supplement to the text,

[0041]

[0042] Among them, i is the label of the i-th image, n q is the nth image corresponding to the i-th image q The number of manually labeled labels, j is the number of manually labeled labels input.

[0043] 203: Extract the nouns in the input sentence, use each noun as a condition, retrieve the relevant common sense triples in the ConceptNet external common sense database and filter the common sense triples;

[0044] ConceptNet is an external commonsense knowledge graph, where natural language words and phrases are connected by edges with labels (indicating edge type) and weights (indicating edge credibility). It is a commonsense knowledge base based on the most basic things humans know. ConceptNet is essentially a triple: <start node, relationship label, end node>. Using a noun from an extracted input sentence as the start node, the ConceptNet commonsense database is searched for relevant commonsense triples, providing knowledge not present in the prior knowledge base.

[0045] To extract nouns from input sentences, this algorithm uses the Stanford CoreNLP toolkit, which can tag the part of speech of each word in a sentence. Part-of-speech tagging can produce a variety of results, such as nouns, verbs, prepositions, and pronouns. Words with a noun part of speech are selected as the starting point for extracting common sense.

[0046] After obtaining all nouns in the input sentence, the ConceptNet common sense database is searched for related common sense triples, starting with the noun as the starting node. During the common sense triple search, the number of common sense triples corresponding to each starting node can be set to k. The retrieved common sense triples consist of <start node, relationship label, end node>. Each common sense triple is assigned a confidence weight, representing its credibility. The set of common sense triples corresponding to each noun is sorted in descending order of confidence.

[0047] ConceptNet is a massive knowledge graph containing millions of common sense assertions expressed in natural language. Therefore, using such a vast knowledge base as ConceptNet to retrieve common sense inevitably introduces common sense triplets that are not highly relevant to the input sentence. For image generation tasks, these retrieved irrelevant common sense triplets can be interpreted as noise, negatively impacting image generation. Therefore, common sense filtering plays a crucial role in image generation tasks, and an algorithm must be designed to filter out this irrelevant common sense noise.

[0048] This method designs a two-level noise filtering algorithm to filter out irrelevant knowledge triples retrieved in a hierarchical manner. The following describes the noise filtering method for each level in detail:

[0049] (1) The first level of noise filtering only retains common sense triples whose start and end nodes are both in the dataset text dictionary.

[0050] ConceptNet contains millions of common sense assertions expressed in natural language, and common sense knowledge is highly divergent. Even if the noun in the input sentence is used as the starting node for the search, a large amount of irrelevant common sense knowledge will still be retrieved, seriously interfering with the image generation task. If the end node of a common sense triple has never appeared in the dataset text dictionary, then this common sense triple basically cannot contribute valuable information to image generation and can be understood as noise. Therefore, the first filtering step of this method is to directly remove common sense triples whose start and end nodes do not appear in the dataset text dictionary.

[0051] This process can be expressed as follows:

[0052] K t =<ω s -R-ω e >#(2)

[0053] Ω={ω1,ω2.ω3,…,ω n}#(3)

[0054] K1={K t |ω s ∈Ω,ω e ∈Ω}#(4)

[0055] For formula (2), K t represents a common sense triple, ω s represents the starting node of the common sense triple, R represents the relationship label, ω e Represents the end node of a common sense triple.

[0056] For formula (3), Ω represents the dictionary of the dataset, n represents the size of the dictionary, that is, the number of words contained in the dictionary, ω i Represents the i-th word in the dictionary.

[0057] Formula (4) represents the first-level screening process, K1 represents the set of common sense triples retained after the first screening, and the screening condition is that the start node and end node of the common sense triple are both in the dataset dictionary.

[0058] (2) The second level of noise filtering only retains the top k common sense triplets ranked by the cosine similarity scores between the common sense triplet feature vector and the target image feature vector, and the cosine similarity scores between the common sense triplet feature vector and the input sentence feature vector.

[0059] After obtaining the first-level filtered common sense triple set K1 through the above step (1), although the start and end nodes of the common sense triples are in the dataset dictionary, the vocabulary in the dictionary is complex and numerous, and the common sense triples may still be irrelevant to the input sentence. Therefore, it is necessary to perform a second-level filtering on the common sense triple set K1 to continue filtering out common sense triples that are semantically irrelevant to the input sentence. Cosine similarity uses the cosine value of the angle between two vectors in the vector space as a measure of the size of the difference between two individuals. The closer the cosine value is to 1, the closer the angle is to 0 degrees, that is, the more similar the two vectors are, which is called "cosine similarity". By calculating the cosine similarity between the common sense triple feature vector and the target image feature vector, and the cosine similarity between the common sense triple feature vector and the input sentence feature vector, and then weighting and ranking the two sets of similarity scores, the top k common sense triples are obtained as the result of the second-level filtering.

[0060] This process can be expressed as follows:

[0061]

[0062]

[0063] In formula (5), u and u represent two eigenvectors. The cosine similarity of vectors u and v is obtained by calculating the dot product of u and v and their respective modulo values.

[0064] For formula (6), K represents the set of common sense triplet feature vectors after the first level of filtering, the number of elements in the set is m, I represents the feature vector of the target image, and T represents the feature vector of the input sentence. i represents the i-th common sense triple in the set of common sense triples after the first level of filtering. By substituting i from 1 to m, we can calculate the weighted similarity score corresponding to each common sense triple. The top k common sense triples with the highest scores are taken as the results of the secondary screening.

[0065] 204: Parsing the input description sentence through a scene graph to obtain several scene triples consisting of <subject-verb-object> and using them as structured knowledge to better grasp the complex scene relationships in the image;

[0066] Compared to vivid and concrete images, natural language is somewhat abstract. For textual statements containing complex scene relationships, such as multiple objects in an image with connections between them, such as their positional relationships and the subject-object relationship, the generator struggles to fully and clearly understand the scene described in abstract natural language. This method alleviates this problem by extracting a scene graph. A scene graph is a structured representation consisting of several scene triples consisting of <subject-verb-object>.

[0067] The predicate in a scene triple describes the scene relationship between the subject and object, and is a fine-grained scene unit. The extracted scene triples serve as structured knowledge, specifically segmenting multi-object scene relationships in an image. This helps the generator more clearly understand complex scene relationships within an image, addressing the inherent abstractness of natural language and the difficulty in understanding multi-object scene relationships.

[0068] This method uses the natural language processing tool Standford Enhanced Dependency Parser provided by Stanford University [9] Parse the input sentence into several triples. This natural language processing tool is a pre-trained model and can be used directly. For example, if you input "A man in a helmet rode past the pub on a motorbike," after scene parsing, it will output several structured scene triplets, such as "man in helmet," "man on motorbike," and "man past pub."

[0069] 205: Encode the additional text extracted from the prior knowledge base, the common sense triples extracted from the external knowledge base, and the scenario triples parsed from the input sentence respectively to obtain the feature vectors of the corresponding knowledge;

[0070] The encoding of prior knowledge, external knowledge, and scene knowledge still uses the text encoder in step 201, and the obtained knowledge feature vectors are all 256-dimensional feature vectors, which are used to provide common sense guidance for the upsampling knowledge fusion block of the generative adversarial network.

[0071] 206: Input the original sentence and knowledge description into the generator of the generative adversarial network to generate an image;

[0072] The present invention establishes a semantic mapping relationship between the knowledge description and the generated image area by adding rich common sense knowledge descriptions step by step and one by one to the upsampling fusion module, enhances the visual authenticity of the generated image area, and finally synthesizes a high-quality image by fusing all image areas, thereby realizing the generation and conversion from input text to high-quality, highly semantically consistent images. Each upsampling knowledge fusion block adds the following knowledge description feature vectors step by step and one by one: 1 additional text knowledge feature vector, 1 scene triple knowledge feature vector, and 1 common sense triple feature vector fused by attention. In the forward upsampling generation process, different knowledge description features are added to each upsampling fusion block, so that the upsampling fusion block pays attention to the image area related to the knowledge description, establishes a semantic mapping relationship between the knowledge description and the generated image area, strengthens the image region, and enhances the visual authenticity of the generated image area.

[0073] like Figure 3 As shown in the example, the input text description is "Two warcrafts on an aircraft carrier prepare to take off during a military exercise." Each upsampling fusion block incorporates different knowledge descriptions to enhance the image region. For example, the common sense knowledge description "Aircraft carriers have long runways" establishes a semantic mapping relationship with the long runways of aircraft carriers. This allows the upsampling fusion block to focus on the image region of the aircraft carrier's long runways, enhancing the visual realism of the generated image region. The generator finally synthesizes a high-quality image by fusing all image regions, achieving the generation and conversion of input text into high-quality, semantically consistent images.

[0074] The following is a detailed description of the upsampling knowledge fusion blocks in the generator: Each upsampling knowledge fusion block consists of one upsampling operation and two common-sense-guided affine transformation blocks. The affine transformation blocks perform channel-level scaling and biasing on the intermediate visual feature variables based on the input text constraints and gradually added common-sense knowledge descriptions. During the forward image generation process, the text constraint information and knowledge information are continuously integrated to ultimately generate a semantically consistent image.

[0075] The internal implementation of the common sense guided affine transformation block is introduced in detail below.

[0076] Step 1: Input the feature vector e of the sentence s and the scene triplet feature vector e st Splicing is performed, and the spliced ​​feature vectors are respectively predicted by the multi-layer perceptron to obtain the scaling parameters γ required for the affine transformation c and bias parameter β c , the scaling parameter γ c and bias parameter β c This is used to perform channel-level scaling and biasing on the intermediate visual features output by the upsampling knowledge fusion block. The predicate in the scene triple describes the scene relationship between the subject and object. It is a fine-grained scene unit that helps the generator more clearly understand complex scene relationships in images, addressing the shortcomings of natural language abstraction and the difficulty in understanding multi-object scene relationships.

[0077] Step 2: After filtering by the secondary noise filtering algorithm, each common sense triplet has different semantic similarities with the input sentence. It is necessary to perform reasonable and effective feature fusion on these k common sense triples to increase the weight of common sense triples with higher semantic similarity. To this end, this method designs a common sense feature fusion method based on the attention mechanism. This process can be expressed as follows:

[0078] F attn (es ,e cs )=α1e cs1 +α1e cs1 +…+α m e csm #(7)

[0079]

[0080] For formula (7), e s Represents the feature vector of the input sentence, e cs Represents the set of k common sense triples feature vectors corresponding to each conditional noun, e csi represents the i-th common sense triplet feature vector in the set of k common sense triplet feature vectors, α i Represents the weight of the i-th common sense feature triplet.

[0081] Formula (8) represents the calculation method of the weight of the i-th common sense feature triple, where e s ·e csi Represents e s Eigenvectors and e csi Dot product of eigenvectors.

[0082] The fusion feature vector F of the common sense triple based on the attention mechanism is calculated by formula (7): attn (e s ,e cs ), and the additional text vector e in the prior knowledge base text Splicing is performed, and an additional correction bias parameter β is provided to the affine change through the multi-layer perceptron c ′, additional textual knowledge describes the image from different angles. Different texts have different focuses and description angles, which can describe an image more comprehensively. Common sense triples can provide knowledge information that does not exist in the prior knowledge base, further compensating for the shortcoming of limited information content of a single text, and thus can provide a corrected and more accurate bias.

[0083] Next, a common sense-guided channel-wise affine transformation is performed on the intermediate visual variables as follows:

[0084] γ c =L γ (e s +e st ), β c =L β (e s +e st )#(9)

[0085] β c ′=L β (Fattn (e s ,e cs )+e text )#(10)

[0086]

[0087] Among them, x chw is a feature on a channel of the intermediate visual variable, e s is the feature vector of the input text, e st is the feature vector of the scene triplet, e text is the additional textual knowledge feature vector, F attn (e s ,e cs ) is the common sense triplet feature vector based on the attention mechanism fusion, γ c and β c is the affine transformation at x chw Scaling parameters and bias parameters on channel features. L γ (·) and L β (·) represent the prediction scaling parameter γ c and bias parameter β c Multilayer Perceptron. λ is a hyperparameter that controls the weight of the correction bias.

[0088] After upsampling and two common-sense-guided affine transformations, the intermediate visual variables serve as input to the next upsampling knowledge fusion block. The forward image generation process of these six upsampling knowledge fusion blocks continuously integrates text constraint information and rich knowledge description information. The final upsampling knowledge fusion block, carrying the continuously integrated text constraint information and knowledge description information, passes through a convolutional layer to ultimately generate a semantically consistent image.

[0089] 206: The discriminator takes the synthesized image and the corresponding text vector as input to calculate the adversarial loss and update the parameters of the generator network; the discriminator is used to identify the authenticity of the generated image and the semantic consistency with the input text. The discriminator consists of 6 consecutive downsampling blocks, each of which is a Resblock

[10] , used to extract vector features of the generated image. The image features extracted by 6 consecutive downsampling blocks are concatenated with the input text features. The concatenated feature vector is passed through two convolutional layers to calculate the adversarial loss.

[0090] Below are the loss functions corresponding to the discriminator and generator:

[0091] Discriminator loss function:

[0092]

[0093] Among them, s is the input text sentence, is a text sentence that does not match the image, which is used to allow the discriminator to better judge the semantic consistency between the generated image and the input sentence. x is the real image corresponding to the input sentence s, is the image generated by the generator. k and p are for DF-GAN

[11] The algorithm performs gradient penalties on real images with matching sentences, which can guide the generator to generate more realistic images.

[0094] Generator loss function:

[0095]

[0096] Among them, s is the input text sentence, is the image generated by the generator.

[0097] 207: The generator and discriminator are optimized alternately, and the generator finally generates the image with the best quality.

[0098] The generator and discriminator are two independent neural networks that optimize each other through adversarial loss, guiding each other in the right direction for parameter optimization. The discriminator's goal is to verify the authenticity of generated images. Each optimization step places higher demands on the generator, forcing it to produce higher-quality images. Based on the discriminator's feedback, the generator updates its own parameters through backpropagation to generate better images. The goal is to prevent the discriminator from distinguishing between real and synthetic images. The two compete and reinforce each other. Through continuous alternating optimization, the generator and discriminator eventually reach a dynamic equilibrium, achieving optimal performance for both. The discriminator is unable to distinguish between the authenticity of the generator's images, and the generator, optimized by the discriminator, produces the highest-quality images.

[0099] Example 3

[0100] A common sense-guided text-to-image generation device includes a processor and a memory, wherein the memory stores program instructions, and the processor calls the program instructions stored in the memory to cause the device to perform the following method steps:

[0101] Build a priori knowledge base and select text descriptions related to the input sentence to enrich the single text input;

[0102] Extract the nouns in the input sentence, use each noun as a condition, search for relevant common sense triples in the ConceptNet external common sense database, and filter the common sense triples;

[0103] The input description sentence is parsed into a scene graph to obtain several scene triples consisting of <subject-verb-object> and use them as structured knowledge;

[0104] Input the original sentence and knowledge description into the generator of the generative adversarial network to generate an image;

[0105] The discriminator uses the synthesized image and the corresponding text vector as input to calculate the adversarial loss and update the parameters of the generator network. The generator and discriminator are optimized alternately, and the generator ultimately generates the best quality image, which is used in the field of intelligent news content generation.

[0106] Among them, the text description related to the selection and input sentence is used to enrich the single text input:

[0107] Given input text Select 6 and Different manually annotated sentences As a supplement to the text,

[0108]

[0109] Among them, i is the label of the i-th image, n q is the nth image corresponding to the i-th image q The number of manually annotated labels is 0, and j is the number of manually annotated labels. The following steps are used to retrieve the relevant common sense triples in the ConceptNet external common sense database using each noun as a condition: Using the noun as the starting node, retrieve the relevant common sense triples in the ConceptNet common sense database. The number of common sense triples corresponding to each starting node is set to j. The retrieved common sense triples are composed of <start node, relationship label, end node>. Each common sense triple has a confidence weight representing the credibility of the common sense triple. The set of common sense triples corresponding to each noun is sorted in descending order of confidence.

[0110] Furthermore, the common sense triples are filtered as follows:

[0111] 1) Only retain common sense triples whose start and end nodes are in the dataset text dictionary;

[0112] 2) Only the top k common sense triplets are retained, ranked by the cosine similarity scores between the common sense triplet feature vector and the target image feature vector, and the cosine similarity scores between the common sense triplet feature vector and the input sentence feature vector. Among them, only the common sense triplets whose start and end nodes are both in the dataset text dictionary are retained:

[0113] K t =<ω s -R-ω e >

[0114] Ω={ω1,ω2.ω3,…,ω n}

[0115] K1={K t |ω s ∈Ω,ω e ∈Ω}

[0116] Among them, K t represents a common sense triple, ω s represents the starting node of the common sense triple, R represents the relationship label, ω e represents the end node of the common sense triple; Ω represents the dictionary of the dataset, n represents the size of the dictionary, ω i represents the i-th word in the dictionary; K1 represents the set of common sense triples retained after the first screening.

[0117] Furthermore, only the cosine similarity scores between the common sense triplet feature vector and the target image feature vector, and the cosine similarity scores between the common sense triplet feature vector and the input sentence feature vector are retained, and the top k common sense triplet are ranked weightedly as follows:

[0118]

[0119]

[0120] Among them, u and v represent two feature vectors, K represents the set of common sense triple feature vectors after the first level of filtering, the number of elements in the set is m, I represents the feature vector of the target image, T represents the feature vector of the input sentence, and K i represents the i-th common sense triple in the common sense triple set after the first level of filtering. By inserting i from 1 to m, the weighted similarity score corresponding to each common sense triple can be calculated. The top k common sense triples with the highest scores are taken as the results of the secondary screening.

[0121] Among them, the generator includes: upsampling knowledge fusion block,

[0122] Each upsampling knowledge fusion block adds one additional text knowledge feature vector, one scene triplet knowledge feature vector, and one common sense triplet feature vector fused with attention step by step. During the forward upsampling generation process, different knowledge description features are added to each upsampling fusion block, so that the upsampling fusion block focuses on the image area related to the knowledge description, establishes a semantic mapping relationship between the knowledge description and the generated image area, strengthens the image region by region, and enhances the visual authenticity of the generated image area.

[0123] The common sense triplet feature vector based on attention mechanism fusion is calculated using the following formula:

[0124] F attn (e s ,e cs)=α1e cs1 +α1e cs1 +…+α m e csm

[0125]

[0126] Among them, e s Represents the feature vector of the input sentence, e cs Represents the set of k common sense triples feature vectors corresponding to each conditional noun, e csi represents the i-th common sense triplet feature vector in the set of k common sense triplet feature vectors, α i represents the weight of the i-th common sense feature triplet; e s ·e csi Represents e s Eigenvectors and e csi dot product of eigenvectors;

[0127] The common sense-guided channel-level affine transformation is performed on the intermediate visual variables as follows:

[0128] γ c =L γ (e s +e tri ), β c =L β (e s +e st )

[0129] β c ′=L β (F attn (e s ,e cs )+e text )

[0130]

[0131] Among them, x chw is a feature on a channel of the intermediate visual variable, e s is the feature vector of the input text, e st is the feature vector of the scene triplet, e text is the additional textual knowledge feature vector, F attn (e s ,e cs ) is the common sense triplet feature vector based on the attention mechanism fusion, γ c and β c is the affine transformation at x chw Scaling parameters and bias parameters on channel features. L γ (·) and Lβ (·) represent the prediction scaling parameter γ c and bias parameter β c Multilayer Perceptron. λ is a hyperparameter that controls the weight of the correction bias.

[0132] It should be noted here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0133] The execution subjects of the above-mentioned processor 1 and memory 2 can be computers, single-chip microcomputers, microcontrollers and other devices with computing functions. In specific implementation, the embodiment of the present invention does not limit the execution subject and it is selected according to the needs of actual application.

[0134] Data signals are transmitted between the memory 2 and the processor 1 via the bus 3 , which will not be described in detail in the embodiment of the present invention.

[0135] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored program, and when the program is running, controls the device where the storage medium is located to execute the method steps in the above embodiment.

[0136] The computer-readable storage medium includes but is not limited to a flash memory, a hard disk, a solid-state drive, and the like.

[0137] It should be noted here that the description of the readable storage medium in the above embodiment corresponds to the description of the method in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0138] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present invention are generated in whole or in part.

[0139] The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted via computer-readable storage media. Computer-readable storage media can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. Available media can include magnetic media or semiconductor media, etc.

[0140] References:

[0141] [1]Andrew Brock,Jeff Donahue,and Karen Simonyan,“Large scale GANtraining for high fidelity natural image synthesis,”in ICLR,2019.

[0142] [2]Justin Johnson,Agrim Gupta,and Li Fei-Fei,“Image generation fromscene graphs,”in CVPR,2018,pp.1219–1228.

[0143] [3]Mehdi Mirza and Simon Osindero,“Conditional generative adversarialnets,”CoRR,vol.abs / 1411.1784,2014.

[0144] [4]Han Zhang,Tao Xu,and Hongsheng Li,“Stackgan:Text to photo-realistic image synthesis with stacked generative adversarial networks,”inICCV,2017,pp.5908–5916.

[0145] [5]Tao Xu,Pengchuan Zhang,Qiuyuan Huang,Han Zhang,Zhe Gan,XiaoleiHuang,and Xiaodong He,“Attngan:Fine-grained text to image generation withattentional generative adversarial networks,”in CVPR,2018,pp.1316–1324.

[0146] [6]Minfeng Zhu,Pingbo Pan,Wei Chen,and Yi Yang,“DM-GAN:dynamic memorygenerative adversarial networks for text-to-image synthesis,”in CVPR,2019,pp.5802–5810.

[0147] [7]Yang Y,Wang L,Xie D,et al.Multi-sentence auxiliary adversarialnetworks for fine-grained text-to-image synthesis.IEEE Transactions on ImageProcessing,2021

[0148] [8]Cheng J,Wu F,Tian Y,et al.“RiFeGAN:Rich feature generation fortext-to-image synthesis from prior knowledge,”in CVPR.2020,pp.10911-10920.

[0149] [9]Sebastian Schuster,Ranjay Krishna,Angel Chang,Li Fei-Fei,andChristopher D Manning,“Generating semantically precise scene graphs fromtextual descriptions for improved image retrieval,”in VL@EMNLP,2015,pp.70–80.

[0150]

[10] Kaiming He,Xiangyu Zhang,Shaoqing Ren,and Jian Sun.2016.Deepresidual learning for image recognition.In Proceedings of the IEEE conferenceon computer vision and pattern recognition.770–778.

[0151]

[11] Ming Tao,Hao Tang,Fei Wu,Xiaoyuan Jing,Bing-Kun Bao,andChangsheng Xu,“DF-GAN:A simple and effective baseline for text-to-imagesynthesis,”in CVPR,2022,pp.16494–16504.

[0152] Those skilled in the art will understand that the accompanying drawings are only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0153] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A common sense-guided text-to-image generation method, characterized in that: The method comprises: Build a priori knowledge base and select text descriptions related to the input sentence to enrich the single text input; Extract the nouns in the input sentence, use each noun as a condition, search for relevant common sense triples in the ConceptNet external common sense database, and filter the common sense triples; The input description sentence is parsed through scene graph to obtain several scene triples consisting of <subject-verb-object> and use them as structured knowledge; Input the original sentence and knowledge description into the generator of the generative adversarial network to generate an image; The discriminator uses the synthesized image and the corresponding text vector as input to calculate the adversarial loss and update the parameters of the generator network. The generator and discriminator are optimized alternately, and the generator ultimately produces the highest-quality image, which is used for intelligent image generation of news content. The generator includes: an upsampling knowledge fusion block, Each upsampling knowledge fusion block adds one additional text knowledge feature vector, one scene triplet knowledge feature vector, and one common sense triplet feature vector fused with attention step by step. During the forward upsampling generation process, different knowledge description features are added to each upsampling fusion block. The common sense triplet feature vector based on attention mechanism fusion is calculated using the following formula: F attn (e s ,e cs )=α1e cs1 +α2e cs2 +…+a m e csm Among them, e s Represents the feature vector of the input sentence, e cs Represents the set of k common sense triples feature vectors corresponding to each conditional noun, e csi represents the i-th common sense triplet feature vector in the set of k common sense triplet feature vectors, α i represents the weight of the i-th common sense feature triplet; e s ·e csi Represents e s Eigenvectors and e csi The dot product of the feature vectors; the channel-level affine transformation guided by common sense is performed on the intermediate visual variables as follows: γ c =L γ (and s +e tri ),β c =L β (and s +e st ) β c ′=L β (F attn (And s ,And cs )+e text ) Among them, x chw is a feature on a channel of the intermediate visual variable, e s is the feature vector of the input text, e st is the feature vector of the scene triplet, e text is the additional textual knowledge feature vector, F attn (e s ,e cs ) is the common sense triplet feature vector based on the attention mechanism fusion, γ c and β c is the affine transformation at x chw Scaling parameters and bias parameters on channel features, L γ (·) and L β (·) represent the prediction scaling parameter γ c and bias parameter β c Multilayer perceptron, λ is a hyperparameter that controls the weight of the correction bias.

2. The common sense-guided text-to-image generation method according to claim 1, characterized in that: The text description associated with the selection and input statement to enrich a single text input is: Given input text Select 6 and Different manually annotated sentences As a supplement to the text, Among them, i is the label of the i-th image, n q is the nth image corresponding to the i-th image q The manually labeled number, j is the input manually labeled number, and k is the number of common sense corresponding to each starting node.

3. The common sense-guided text-to-image generation method according to claim 1, characterized in that: The above-mentioned retrieval of the relevant common sense triples in the ConceptNet external common sense database based on each noun is as follows: Taking the noun as the starting node, retrieve the relevant common sense triples in the ConceptNet common sense library. Set the number of common sense corresponding to each starting node to k. The retrieved common sense triples are composed of <start node, relationship label, end node>. Each common sense triple has a confidence weight, which represents the credibility of this common sense triple. The set of common sense triples corresponding to each noun is arranged in descending order of confidence.

4. The common sense-guided text-to-image generation method according to claim 1, characterized in that: The filtering of common sense triples is specifically as follows: 1) Only retain common sense triples whose start and end nodes are in the dataset text dictionary; 2) Only the top k common sense triplets are retained based on the weighted cosine similarity scores between the common sense triplet feature vector and the target image feature vector, and the cosine similarity scores between the common sense triplet feature vector and the input sentence feature vector.

5. The common sense-guided text-to-image generation method according to claim 4, characterized in that: The common sense triples that only retain the start and end nodes in the dataset text dictionary are: K t =<ω s -R-ω e > Ω={ω1,ω2.ω3,…,ω n } K1={K t |oh s ∈Ω,ω e ∈Ω} Among them, K t represents a common sense triple, ω s represents the starting node of the common sense triple, R represents the relationship label, ω e represents the end node of the common sense triple; Ω represents the dictionary of the dataset, n represents the size of the dictionary, ω i represents the i-th word in the dictionary; K1 represents the set of common sense triples retained after the first screening.

6. The common sense-guided text-to-image generation method according to claim 4, characterized in that: The top k common sense triplets are only retained based on the cosine similarity scores between the common sense triplet feature vector and the target image feature vector, and the cosine similarity scores between the common sense triplet feature vector and the input sentence feature vector. Among them, u and v represent two feature vectors, K represents the set of common sense triple feature vectors after the first level of filtering, the number of elements in the set is m, I represents the feature vector of the target image, T represents the feature vector of the input sentence, and K i represents the i-th common sense triple in the common sense triple set after the first level of filtering. By inserting i from 1 to m, the weighted similarity score corresponding to each common sense triple can be calculated. The top k common sense triples with the highest scores are taken as the results of the secondary screening.

7. A common sense-guided text-to-image generation device, characterized in that: The device comprises: a processor and a memory, wherein the processor calls program instructions stored in the memory to enable the device to execute the method steps according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to perform the method steps according to any one of claims 1 to 6.

Images