A method, system and device for generating image based on text data

Abstract

The present application relates to the field of image generation technology, and specifically to a method, system and device for image generation based on text data, the method comprising: obtaining a text data set of an image to be generated; constructing a color orientation index for each word according to the importance weight and sentiment color score of each word and its neighboring words in the text data set; determining the importance index of each word according to the difference in the color orientation index between each word in the text data set and all words in the paragraph in which it is located and the similarity of the word vector, combined with the frequency of each word in the text data set in which it is located; weighting each word in the text data set according to the importance index, using the weighted text data set as the input of a generative adversarial network, and generating a corresponding image. The present application aims to focus on the meaning expressed by the core words so that the generated image can more accurately reflect the theme and content of the text data.

Description

Technical Field

[0001] The present application relates to the technical field of image generation, and in particular to a method, system and device for image generation based on text data. Background Art

[0002] With the rapid development of artificial intelligence technology, methods for generating images based on text have received widespread attention. This method can generate images that meet the description through natural language description, greatly expanding the possibilities of human expression and creation. Traditional image generation usually relies on manual drawing or image editing software, but these methods require users to have high professional skills and the generation process takes a long time. In recent years, image generation models based on deep learning, such as generative adversarial networks and diffusion models, have made significant progress in the field of computer vision, making it possible to automatically generate high-quality images.

[0003] However, existing text-based image generation methods still face many challenges in practical applications. How to accurately understand and interpret complex text description languages ​​has become a major problem. Image generation models based on deep learning can show good results under simple semantic structures. However, when the target information or scene described in the text is relatively complex, due to information loss or inability to fully understand the text semantic information, the image generation results often have problems such as target errors, making it difficult to accurately reflect the subject and content of the text data. Summary of the invention

[0004] In order to solve the above technical problems, the purpose of this application is to provide a method, system and device for generating images based on text data. The technical solutions adopted are as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for generating an image based on text data, the method comprising the following steps:

[0006] Get the text dataset of the image to be generated;

[0007] According to the importance weight and sentiment color score of each word and its neighboring words in the text dataset, a color orientation index for each word is constructed;

[0008] Determine the importance index of each word based on the difference in color orientation index between each word in the text dataset and all words in the paragraph it is in, as well as the similarity of word vectors, combined with the frequency of each word in the text dataset it is in.

[0009] Each word in the text dataset is weighted according to the importance index, and the weighted text dataset is used as the input of the generative adversarial network to generate the corresponding image.

[0010] Preferably, the vocabulary includes nouns, adjectives, verbs and other vocabulary, and their corresponding importance weights are set in descending order according to the arrangement order.

[0011] Preferably, the sentiment color score is obtained by using the SnowNLP sentiment analysis tool on the words in the text data set.

[0012] Preferably, the construction process of the color orientation index is: calculating the importance weight and emotional color score of any neighboring word of each word for forward fusion results; taking the average level of the fusion results of all neighboring words of each word as the color orientation index of each word.

[0013] Preferably, before forward fusion of the importance weight and the emotional color score, the difference between the emotional color score and two endpoints of its range is calculated, and the difference is used to replace the emotional color score and fused with the importance weight.

[0014] Preferably, the method for obtaining the word vector is: using a context embedding model for the text data set to generate a word vector for each vocabulary word.

[0015] Preferably, the method for determining the importance index includes:

[0016] Determine the scene relevance index of each word based on the difference in color orientation index between each word in the text dataset and all words in the paragraph where it is located and the similarity of word vectors;

[0017] The importance index of each word is determined according to the scene relevance index of each word and the frequency of its occurrence in the text dataset.

[0018] Preferably, the method for determining the scene relevance index includes:

[0019] For any word in the paragraph where each word in the text dataset is located, calculate the similarity of the word vector between each word and any word;

[0020] Calculating the difference in color orientation index between each of the words and any of the words;

[0021] The similarities and differences are reversely fused, and the scene relevance index is determined according to the reverse fusion result of all the words in the paragraph where each word is located.

[0022] In a second aspect, an embodiment of the present application provides an image generation system based on text data, the system comprising:

[0023] A text data acquisition module, used to obtain a text data set of an image to be generated;

[0024] The text data analysis module is used to construct a color orientation index for each word according to the importance weights and sentiment color scores of each word and its neighboring words in the text data set; the importance index of each word is determined according to the difference in the color orientation index between each word in the text data set and all words in the paragraph in which it is located and the similarity of the word vectors, combined with the frequency of each word appearing in the text data set in which it is located;

[0025] The image generation module is used to weight each word in the text dataset according to the importance index, and use the weighted text dataset as the input of the generative adversarial network to generate the corresponding image.

[0026] In a third aspect, an embodiment of the present application further provides an image generation device based on text data, wherein the device executes any of the steps of the above-mentioned image generation method based on text data.

[0027] It can be seen from the above embodiments that the method, system and device for generating an image based on text data provided by the embodiments of the present application have at least the following beneficial effects:

[0028] The present application collects a text data set of an image to be generated, and constructs a color orientation index by analyzing the importance weights and emotional color scores of the words in the text data set. The beneficial effect is that the color orientation index takes into account the part of speech and emotional expression of the neighboring words around each word. According to the color orientation index of the word, the emotional color orientation that the user wants to express can be correctly conveyed, and then the theme elements and emotional color of the text description data can be better grasped when generating images; secondly, based on the word vector relationship between words in the same paragraph and the frequency of occurrence of the words in the text data set, and combined with the color orientation index of the words, an importance index is constructed. The beneficial effect is that it takes into account the potential semantic relationship between different words when they describe the same scene in the same paragraph in the text data set. Different weights are given to each word in the text data set according to the importance index, which can more accurately represent the importance of each word in the text data set, and then when generating images, the meaning expressed by the core words can be focused on, so that the generated image can more accurately reflect the theme and content of the text data. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 A flowchart of a method for generating an image based on text data provided by an embodiment of the present application;

[0031] Figure 2 A flow chart of a method for determining a scene relevance index provided in one embodiment of the present application;

[0032] Figure 3 A flow chart of a method for determining an importance index provided in one embodiment of the present application;

[0033] Figure 4 A schematic diagram of the structure of an image generation system based on text data provided for one embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to further explain the technical means and effects adopted by the present application to achieve the predetermined invention purpose, the following is a detailed description of the text data-based image generation method, system and device proposed in the present application, its specific implementation, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0035] Unless otherwise specified and limited, terms such as "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a circuit structure, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such articles or devices. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the article or device including the element. In addition, the term "and\or" used herein includes any and all combinations of one or more related listed items. All technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of this application.

[0036] The following is a detailed description of a specific scheme of an image generation method, system and device based on text data provided by the present application in conjunction with the accompanying drawings.

[0037] See also Figure 1 , which shows a flowchart of a method for generating an image based on text data provided by an embodiment of the present application, the method comprising the following steps:

[0038] The first step is to obtain the text dataset of the image to be generated.

[0039] In the advertising industry, the technology of generating images from text has gradually become an innovative trend, bringing new possibilities for brand display and marketing activities. The advertising text description data not only carries the creative intent of the advertisement, but also contains brand information, product features and the emotional needs of the target audience.

[0040] This embodiment takes the text data generation image of the advertising industry as an example to obtain an advertising text description data set. The obtained advertising text description data set is preprocessed using the Harbin Institute of Technology LTP toolkit, including vocabulary, part-of-speech tagging, stop word deletion, etc. The preprocessed advertising text description data set is recorded as text data set Q, wherein the use of the Harbin Institute of Technology LTP toolkit to preprocess text data is a well-known technology, and this application will not repeat its specific implementation process.

[0041] At this point, the text dataset Q of the image to be generated can be obtained through the above method.

[0042] In the second step, a color-oriented index is constructed for each word based on the importance weights and sentiment scores of each word and its neighboring words in the text dataset.

[0043] In the process of generating images using text description data, part of speech is crucial in determining the composition of the image. For example, when a user inputs "blue sneakers", the generated image should use "sneakers" as the main element and highlight the modifying feature "blue". Therefore, in this scenario, "sneakers" as the main element of the image should be given a higher basic weight, while the adjective "blue" affects the choice of color through characteristic features and should be given additional weight in specific contexts.

[0044] Secondly, each keyword has its own unique emotional color, and emotional color also plays an important role in the image generation process. For example, if the text description data entered by the user is "happy summer vacation", the generated image should show a bright and pleasant atmosphere. Therefore, accurate analysis of the emotional color of the keyword can correctly convey the emotions that the user wants to express, thereby enhancing the color of the advertising image.

[0045] In the text description language, nouns are often used to determine the subject elements of the generated image, while adjectives and adverbs pay more attention to the emotional expression of the modified words, which determines the attribute characteristics of the subject elements in the generated image, and verbs mainly describe the actions of the subject elements. In addition, the text description language may also have other words. Since the subject elements in the generated image are mainly determined by these parts of speech, this embodiment unifies other words except names, adjectives and verbs into one category for analysis.

[0046] Therefore, in this embodiment, the importance weights of nouns, adjectives, verbs and other words are set to 2, 1.5, 1.2 and 0.1 respectively, and the importance weights of nouns, adjectives, verbs and other words are set in order from large to small. In other implementation methods of the embodiments of the present application, the implementers can set them by themselves.

[0047] For the preprocessed text dataset Q, obtain K neighboring words around each word, including itself, to form a neighboring word set A for each word. In this embodiment, K is 7. According to the emotional color and part-of-speech features of the words in the neighboring word set A, the color orientation of the word is expressed.

[0048] In this embodiment, the SnowNLP sentiment analysis tool is used to perform sentiment analysis on the words in the text dataset Q to obtain the sentiment color score of each word in the text dataset Q. , emotional color score The value range of is [0,1], where 0 represents an extremely negative emotion and 1 represents an extremely positive emotion. The SnowNLP sentiment analysis tool is a well-known technology, and this application will not repeat its specific calculation process.

[0049] Accordingly, the present application aims to construct a color-oriented index for each word according to the importance weight and sentiment color score of each word and its neighboring words in a text dataset.

[0050] Preferably, in this embodiment, the importance weight and the emotional color score of any neighboring word of each word are calculated to perform forward fusion, and the average level of the fusion results of all neighboring words of each word is used as the color orientation index of each word. Before the importance weight and the emotional color score are forward fused, the difference between the emotional color score and the two endpoints of its range is calculated, and the difference is used instead of the emotional color score to be fused with the importance weight.

[0051] It can be understood that forward fusion is a fusion method such as addition and multiplication between data. The specific forward fusion method is determined by the implementer according to the actual situation, and this application does not impose any special restrictions.

[0052] In one implementation of the present application, taking the i-th word in the text data set Q as an example, the color orientation index of the i-th word is The expression is: ; In the formula, Represents a text dataset Middle The color orientation index of the words, Represents a text dataset Middle The number of neighboring words around a word, Represents a text dataset Middle The words around The importance weights of neighboring words, Represents a text dataset Middle The words around The sentiment color scores of neighboring words.

[0053] It should be understood that the above formula obtains the text dataset Each target word is surrounded by The part of speech and emotional color characteristics of neighboring words, so that the calculated color orientation index reflects the text data set The combined influence of the emotional color and part-of-speech characteristics of each word in its context makes the color orientation of each analysis more obvious. Middle The words around When the neighboring words show extremely positive or negative sentiment, the calculated The larger the text data set, the Middle The words around Different neighboring words are given different weights, so that the calculated text dataset Middle Color Orientation Index of Words The larger the value is, the more attention will be paid to the emotional expression of the word in the process of converting text to image, so that the theme elements and emotional color of the text description data can be better grasped when generating images.

[0054] The third step is to determine the importance index of each word based on the difference in color orientation index between each word in the text dataset and all words in its paragraph and the similarity of word vectors, combined with the frequency of each word appearing in its text dataset.

[0055] In the text description data, some words have potential semantic relationships. In a specific scenario, different words describe the same scenario, which makes some words highly scene-related. For example, when the user inputs the text description data "beach vacation, sunny, palm trees swaying", the generated image should show a beach scene with palm trees under the sun. In this scenario, there is a high scene relevance between "sunshine", "beach" and "palm trees".

[0056] Furthermore, different descriptive words for the same scene usually appear in adjacent sentences in the same paragraph. Therefore, for the text dataset Q, it is necessary to analyze the correlation between different words in the same paragraph in the text dataset Q.

[0057] The BERT model is a pre-trained language model based on the Transformer structure. It learns rich language representations through unsupervised training on large-scale text data. When generating word embeddings, BERT can capture the contextual information of words and generate dynamic word vectors. Therefore, this example uses the text dataset As input, use the BERT model for the set Each word in generates a specific word vector based on the context information The BERT model is a well-known technology, and its specific implementation process will not be described here.

[0058] In addition, in the process of generating text data images, different words appear at different frequencies in the text description data, so the distribution of words in the text data also reflects the importance of the words in certain cases. Specifically, some high-frequency words that appear repeatedly in the text description data are usually the core content that the entire text description data wants to express, and special attention should be paid to this type of data in the process of generating images.

[0059] Therefore, for the text dataset Document frequency statistics show how often each word appears in the text dataset , and then calculate the overall distribution characteristics of words in the text description data.

[0060] Preferably, in this embodiment, the scene relevance index of each word is determined based on the difference in color orientation index between each word in the text data set and all words in its paragraph and the similarity of word vectors; the importance index of each word is determined based on the scene relevance index of each word and the frequency of its occurrence in the text data set.

[0061] Among them, the method for determining the scene relevance index includes: for any word in the paragraph where each word in the text data set is located, calculating the similarity of the word vector between each word and any word; calculating the difference in color orientation index between each word and any word; reversely fusing the similarity and the difference, and determining the scene relevance index based on the reverse fusion result of all words in the paragraph where each word is located.

[0062] In the embodiment of the present application, the flow chart of the method for determining the scene relevance index is as shown in the attached Figure 2 shown.

[0063] It can be understood that reverse fusion is a fusion method such as subtraction and division between data. The specific reverse fusion method is determined by the implementer according to the actual situation, and this application does not impose any special restrictions.

[0064] In one implementation of the present application, the importance index of a word The expression is: ; In the formula, Represents a text dataset Middle The importance index of each word, Represents a text dataset Middle The frequency of occurrence of a word, e is a natural constant, Represents a text dataset Middle The scene relevance index of the vocabulary.

[0065] in, ; In the formula, Represents a collection of text data The number of words in the paragraph where the i-th word is located, and Represents text datasets respectively Middle words and word vectors for the vocabulary; Represents a computational text dataset Middle words and The cosine similarity between the word vectors of the words, and Represents text datasets respectively Middle words and The color orientation index of the vocabulary; is the parameter adjustment factor. In order to prevent the denominator from being 0, The value is 0.01.

[0066] In this embodiment, when calculating the words and When measuring the similarity between the word vectors of the first word, the cosine similarity is used. The calculation process of the cosine similarity is a well-known technology and will not be described in detail. In other implementations of the present application, the Pearson correlation coefficient and the inverse of the DTW distance can also be used to characterize the first word vector. words and The similarity between the word vectors of the words.

[0067] According to the above analysis, if two words describe the same scene in the text description data, then the two words should have a strong correlation in the context of the text description data, which makes the cosine similarity between the word vectors of the two words The larger the value, the greater the color orientation index between the two words when they describe the same scene characteristics. There is also a high degree of consistency, which makes the difference between the color orientation index calculated between the two words The smaller the text dataset is, the smaller the final calculation Middle The scene relevance index between the word and other words in the same paragraph The bigger.

[0068] At the same time, text datasets Middle The higher the frequency of a word in the text description data, the greater the possibility that the word is a core word, and the text data set calculated is Middle Importance index of each word The larger the value, the more likely the vocabulary is in the text dataset. The higher the importance.

[0069] In the embodiment of the present application, the flow chart of the method for determining the importance index is as shown in the attached Figure 3 shown.

[0070] The fourth step is to weight each word in the text dataset according to the importance index, use the weighted text dataset as the input of the generative adversarial network, and generate the corresponding image.

[0071] After the above analysis and calculation, we get the text data set The importance index of each word in the text dataset reflects the importance of The importance of each word in the text description. When generating images based on text data, the words with larger importance index are more helpful for the model to understand the subject content expressed by the text description data.

[0072] In this embodiment, a generative adversarial network model is used to convert the user's text description data into corresponding images.

[0073] Specifically, for the text dataset For each word in , weight the corresponding word according to the importance index, and record the weighted text dataset as The weighted text dataset is used as the input of the generative adversarial neural network, so that the generated images can more accurately reflect the theme and content of the text description.

[0074] See also Figure 4 , Figure 4 is a structural diagram of a text data-based image generation system provided in an embodiment of the present application. In this embodiment, each unit included in the terminal is used to execute each step in an embodiment corresponding to a text data-based image generation method. Figure 4 The system includes: a text data acquisition module, a text data analysis module, and an image generation module.

[0075] A text data acquisition module, used to obtain a text data set of an image to be generated;

[0076] The text data analysis module is used to construct a color orientation index for each word according to the importance weights and sentiment color scores of each word and its neighboring words in the text data set; the importance index of each word is determined according to the difference in the color orientation index between each word in the text data set and all words in the paragraph in which it is located and the similarity of the word vectors, combined with the frequency of each word appearing in the text data set in which it is located;

[0077] The image generation module is used to weight each word in the text dataset according to the importance index, and use the weighted text dataset as the input of the generative adversarial network to generate the corresponding image.

[0078] Based on the same inventive concept as the above method, an embodiment of the present application further provides an image generation device based on text data, and the device executes the steps of any one of the above-mentioned image generation methods based on text data.

[0079] The various embodiments in the present application are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0080] It should be noted that, unless otherwise specified and limited, terms such as "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a circuit structure, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such article or device. In the absence of further restrictions, an element defined by the sentence "including one..." does not exclude the existence of other identical elements in the article or device including the element. In addition, the term "and\or" used herein includes any and all combinations of one or more related listed items.

[0081] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not invented by the present application.

[0082] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A method for generating an image based on text data, characterized in that: The method comprises the following steps: Get the text dataset of the image to be generated; According to the importance weight and sentiment color score of each word and its neighboring words in the text dataset, a color orientation index for each word is constructed; Determine the importance index of each word based on the difference in color orientation index between each word in the text dataset and all words in the paragraph it is in, as well as the similarity of word vectors, combined with the frequency of each word in the text dataset it is in. Weight each word in the text dataset according to the importance index, use the weighted text dataset as the input of the generative adversarial network, and generate the corresponding image; The construction process of the color orientation index is as follows: calculating the importance weight and emotional color score of any neighboring word of each word for positive fusion result; taking the average level of the fusion result of all neighboring words of each word as the color orientation index of each word; Before forward fusion of the importance weight and the emotional color score, the difference between the emotional color score and the two end points of its range is calculated, and the difference is used to replace the emotional color score and fused with the importance weight.

2. The method for generating an image based on text data as claimed in claim 1, characterized in that: The vocabulary includes nouns, adjectives, verbs and other vocabulary, and the corresponding importance weights are set in descending order according to the arrangement order.

3. The method for generating an image based on text data as claimed in claim 1, characterized in that: The sentiment color score is obtained by using the SnowNLP sentiment analysis tool on the words in the text dataset.

4. The method for generating an image based on text data as claimed in claim 1, characterized in that: The method for obtaining the word vector is: using a context embedding model for the text data set to generate a word vector for each word.

5. The method for generating an image based on text data as claimed in claim 1, characterized in that: The method for determining the importance index includes: Determine the scene relevance index of each word based on the difference in color orientation index between each word in the text dataset and all words in the paragraph where it is located and the similarity of word vectors; The importance index of each word is determined according to the scene relevance index of each word and the frequency of its occurrence in the text dataset.

6. The method for generating an image based on text data as claimed in claim 5, characterized in that: The method for determining the scene relevance index includes: For any word in the paragraph where each word in the text dataset is located, calculate the similarity of the word vector between each word and any word; Calculating the difference in color orientation index between each of the words and any of the words; The similarities and differences are reversely fused, and the scene relevance index is determined according to the reverse fusion result of all the words in the paragraph where each word is located.

7. An image generation system based on text data, characterized in that: To implement the method for generating an image based on text data as claimed in claim 1, the system comprises: A text data acquisition module is used to obtain a text data set of an image to be generated; The text data analysis module is used to construct a color orientation index for each word according to the importance weights and sentiment color scores of each word and its neighboring words in the text data set; the importance index of each word is determined according to the difference in the color orientation index between each word in the text data set and all words in the paragraph in which it is located and the similarity of the word vectors, combined with the frequency of each word appearing in the text data set in which it is located; The image generation module is used to weight each word in the text dataset according to the importance index, and use the weighted text dataset as the input of the generative adversarial network to generate the corresponding image.

8. An image generation device based on text data, the device executing the steps of the image generation method based on text data as claimed in any one of claims 1 to 6.

Images