Visual data generation method and device, electronic equipment and storage medium

Abstract

The invention relates to a visual data generation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a text cue word; wherein the text prompt word comprises a plurality of sentence segments, and each sentence segment has a pre-allocated attention weight; encoding the text cue word into an initial text embedding vector; according to the attention weight of each sentence segment, performing weighting processing on each dimension of the initial text embedding vector to obtain a target text embedding vector; and based on the target text embedding vector, generating visual data matched with the text cue word. According to the scheme, no extra training or fine adjustment is needed, a large amount of time and resources are saved, the sequence and structure of the original text cue words do not need to be disturbed, text unintention is avoided, and the situation of complex text cue words can be effectively handled.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for generating visual data. Background Art

[0002] The text-to-graph model can directly generate images based on user-entered text (or text prompts) that capture the concepts described in the text. However, sometimes, users want to emphasize a specific section within the text prompt, so that the model's generated results will be more biased towards the emphasized portion. This emphasis is equivalent to shifting the model's attention to the emphasized portion of the text.

[0003] There are two approaches in the related art that can achieve this goal. The first is to fine-tune the model's understanding of specific concepts, while the other is to implicitly adjust the encoding of text prompts by rearranging them through prompt engineering. However, the first approach requires additional training and fine-tuning, which consumes a lot of time and resources. The second approach disrupts the order and structure of the original text prompts, resulting in text distortion. Furthermore, neither approach is effective for complex text prompts. Summary of the Invention

[0004] In order to solve or partially solve the problems existing in the related art, the present application provides a visual data generation method, device, electronic device and storage medium, which does not require any additional training and fine-tuning, saving a lot of time and resources, and does not need to disrupt the original order and structure of text prompt words, avoiding text frustration, and can effectively deal with situations with complex text prompt words.

[0005] The first aspect of the present application provides a method for generating visual data, comprising:

[0006] Acquire a text prompt word; wherein the text prompt word includes a plurality of segments, each segment having a pre-assigned attention weight;

[0007] Encoding the text prompt word into an initial text embedding vector;

[0008] According to the attention weights of the respective sentence segments, weighted processing is performed on the dimensions of the initial text embedding vector to obtain a target text embedding vector;

[0009] Based on the target text embedding vector, visual data matching the text prompt word is generated. In one embodiment, before encoding the text prompt word into the initial text embedding vector, the method further includes:

[0010] Sequentially dividing each sentence segment in the text prompt word into word units to obtain a plurality of word units;

[0011] The initial text embedding vector includes an initial word-unit embedding vector of each word-unit; encoding the text prompt word into the initial text embedding vector includes:

[0012] Each word is encoded to obtain an initial word embedding vector for each word.

[0013] In one embodiment, the target text embedding vector includes a target word embedding vector for each word; and the weighted processing of each dimension of the initial text embedding vector according to the attention weight of each sentence segment to obtain the target text embedding vector includes:

[0014] According to the attention weights of the respective sentence segments, the initial word unit embedding vectors of the respective word units are weighted to obtain the target word unit embedding vectors of the respective word units.

[0015] In one embodiment, weighting the initial word-unit embedding vectors of the respective word-units according to the attention weights of the respective sentence segments to obtain target word-unit embedding vectors of the respective word-units includes:

[0016] The target word unit embedding vector of each word unit is calculated based on the value of the initial word unit embedding vector of each word unit and the attention weight of the corresponding sentence segment.

[0017] In one embodiment, generating visual data matching the text prompt word based on the target text embedding vector includes:

[0018] Get preset noise;

[0019] Performing feature fusion on the target word element embedding vector of each word element and the preset noise to obtain word element fusion features of each word element;

[0020] Based on the word-gram fusion features of the respective word-grams, visual data matching the text prompt word is generated.

[0021] In one embodiment, obtaining a text prompt word includes:

[0022] Get user input data;

[0023] A text prompt word is obtained according to the user's input data.

[0024] In one embodiment, obtaining a text prompt word according to the user's input data includes:

[0025] Obtaining an initial text prompt word according to the user's input data;

[0026] The initial text prompt word is divided into segments, and the attention weight of each segment is determined to obtain a text prompt word including multiple segments and the attention weight of each segment.

[0027] A second aspect of the present application provides a visual data generating device, comprising:

[0028] A text prompt word acquisition module, configured to acquire text prompt words; wherein the text prompt words include a plurality of sentence segments, each of which has a pre-assigned attention weight;

[0029] An encoding processing module, configured to encode the text prompt word into an initial text embedding vector;

[0030] A weighted processing module, configured to perform weighted processing on each dimension of the initial text embedding vector according to the attention weights of each sentence segment to obtain a target text embedding vector;

[0031] A visual data generation module is used to generate visual data matching the text prompt word based on the target text embedding vector.

[0032] In one embodiment, before encoding the text prompt word into an initial text embedding vector, the apparatus further comprises:

[0033] A word unit segmentation module is used to sequentially segment each sentence in the text prompt word into word units to obtain multiple word units;

[0034] The initial text embedding vector includes the initial word embedding vector of each word; the encoding processing module includes:

[0035] The encoding processing submodule is used to encode each word unit to obtain the initial word unit embedding vector of each word unit.

[0036] In one embodiment, the target text embedding vector includes a target word embedding vector of each word; and the weighting processing module includes:

[0037] The weighted processing submodule is used to perform weighted processing on the initial word unit embedding vector of each word unit according to the attention weight of each sentence segment to obtain the target word unit embedding vector of each word unit.

[0038] In one embodiment, the weighted processing submodule includes:

[0039] A weighted processing unit is used to calculate the target word unit embedding vector of each word unit based on the value of the initial word unit embedding vector of each word unit and the attention weight of the corresponding sentence segment.

[0040] In one embodiment, the visual data generation module includes:

[0041] A preset noise acquisition submodule is used to acquire the preset noise;

[0042] A feature fusion submodule, configured to perform feature fusion on the target word element embedding vector of each word element and the preset noise to obtain word element fusion features of each word element;

[0043] The visual data generation submodule is used to generate visual data matching the text prompt word based on the word-gram fusion features of the respective word-grams.

[0044] In one embodiment, the text prompt word acquisition module includes:

[0045] Input data acquisition submodule, used to obtain user input data;

[0046] The input data processing submodule is used to obtain text prompt words according to the user's input data.

[0047] In one embodiment, the input data processing submodule includes:

[0048] An input data processing unit, configured to obtain an initial text prompt word according to the user's input data;

[0049] The initial text prompt word processing unit is used to divide the initial text prompt word into segments and determine the attention weight of each segment to obtain a text prompt word including multiple segments and the attention weight of each segment.

[0050] A third aspect of the present application provides an electronic device, including:

[0051] processor; and

[0052] The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method described above.

[0053] A fourth aspect of the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method described above.

[0054] The technical solution provided by this application may have the following beneficial effects:

[0055] The solution provided by the present application is to obtain a text prompt word; wherein the text prompt word includes multiple segments, each of which has a pre-assigned attention weight; encode the text prompt word into an initial text embedding vector; according to the attention weight of each segment, weight the dimensions of the initial text embedding vector to obtain a target text embedding vector; based on the target text embedding vector, generate visual data matching the text prompt word. The present application directly weights the dimensions of the initial text embedding vector of the text prompt word according to the attention weight pre-assigned to each segment, so as to directly control the strength of attention to the content of the text prompt word from the bottom layer, so that the generated visual data is more consistent with the effect to be presented by the text prompt word, thereby not requiring any additional training and fine-tuning, saving a lot of time and resources, and not needing to disrupt the original order and structure of the text prompt word, avoiding text frustration, and the present application can accurately assign the corresponding attention size to the content of the text prompt word through partition weighting, thereby effectively dealing with the situation of complex text prompt words.

[0056] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail the exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0058] Figure 1 is a flow chart of a method for generating visual data according to an embodiment of the present application;

[0059] Figure 2 is another flowchart of the method for generating visual data shown in an embodiment of the present application;

[0060] Figure 3 Schematic diagram of the data processing platform shown in the embodiment of the present application;

[0061] Figure 4A-4B This is an image of a brown puppy generated by the Wensheng graph model;

[0062] Figure 5A-5B This is the “Apples in the tree” image generated by the Wensheng graph model;

[0063] Figure 6 It is an image generated by the text-generated graph model under complex text prompt words;

[0064] Figure 7 is a structural diagram of a visual data generating device shown in an embodiment of the present application;

[0065] Figure 8 It is a structural diagram of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The following describes embodiments of the present application in more detail with reference to the accompanying drawings. Although the accompanying drawings illustrate embodiments of the present application, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0067] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0068] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0069] In the field of AI (artificial intelligence) image generation, a key branch is generating images from text, also known as text-to-image, or more commonly, AI painting. This technology can directly generate images depicting the concepts described in user-entered text (or text prompts). However, sometimes users want to emphasize a specific section within the text prompt, so that the model's generated results will favor the emphasized portion.

[0070] There are two methods in the related art to achieve the above purpose. The first is to change the model's own understanding of specific concepts through fine-tuning, and the other is to implicitly adjust the encoding results of text prompt words by re-arranging text prompt words through prompt word engineering.

[0071] In the first method, in order to strengthen the model's understanding of specific concepts, it is done by fine-tuning the large model using sample data containing specific concepts. This method uses new sample data to train on the basis of the pre-trained model in order to guide the model to focus more on the concepts contained in the fine-tuning data to strengthen the model's cognition.

[0072] However, the first approach requires both collecting sample data and training the model, consuming significant computing power and time. Furthermore, fine-tuning a model often alters the weights of the underlying model, impacting its capabilities and making it highly unstable. Furthermore, the sample data used for fine-tuning the model is often ambiguous to the user, making it difficult to collect sample data that precisely reinforces specific concepts and, consequently, to target the areas they truly need to emphasize. Consequently, the first approach is costly and struggles to precisely direct the model's attention to specific concepts.

[0073] Given the difficulty of fine-tuning, the second approach is more intuitive and feasible for the average user: rearranging text prompts through prompt word engineering. In this second approach, emphasis can be achieved by shifting the word order of the content to be emphasized forward, as the content closer to the front should be the main subject of the description. Alternatively, the emphasis can be achieved by repeatedly mentioning the content in the text. Therefore, this second approach aligns with common human language habits and has a low barrier to use and learning.

[0074] However, the second method implicitly guides the model's attention allocation to the text by reorganizing the language of the text prompts. Since it is not explicit, users cannot determine the level of attention paid to each word. In addition, this method is extremely dependent on the arrangement of the text prompts. In most cases, deliberately emphasizing a concept in the text prompts will disrupt the original order and structure of the text prompts, causing the text to lose its meaning. This disruption is particularly prominent in the case of longer text prompts, often causing the images generated by the model to not match the text prompts, or even causing the image to lose control. Furthermore, this method has weak expressive power. When the user wants to emphasize a concept that the model has a shallow understanding of, or wants to emphasize multiple concepts at the same time, no matter how the text prompts are arranged, it will not be able to effectively emphasize it.

[0075] As can be seen, in related technologies, strengthening the expression of specific content in the model is mostly achieved by fine-tuning or rearranging the text prompt words. However, both methods are highly unstable and either require a lot of time and resources or disrupt the order and structure of the original text prompt words. Both methods are also unsuitable for complex situations and almost fail when the text prompt words become complex.

[0076] In response to the above problems, an embodiment of the present application provides a visual data generation method, which directly weights the dimensions of the initial text embedding vector of the text prompt word according to the attention weight pre-assigned to each sentence segment, so as to directly control the strength of attention to the content of the text prompt word from the bottom layer, so that the generated visual data is more in line with the effect to be presented by the text prompt word, thereby not requiring any additional training and fine-tuning, saving a lot of time and resources, and not needing to disrupt the original order and structure of the text prompt word, avoiding text frustration. In addition, the present application can accurately assign corresponding attention size to the content of the text prompt word through partition weighting, thereby effectively dealing with the situation of complex text prompt words.

[0077] The technical solutions of the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0078] Figure 1 It is a flowchart of the visual data generation method shown in an embodiment of the present application.

[0079] See also Figure 1 , the visual data generation method of the present application comprises:

[0080] S110, obtaining a text prompt word; wherein the text prompt word includes multiple segments, and each segment has a pre-assigned attention weight.

[0081] Embodiments of the present application can be applied to a data processing platform that can obtain input data entered by a user. The input data can refer to the content required to generate visual data. For example, the input data can include the object category, color, quantity, environment, weather, and other content in the visual data to be generated. The data processing platform can accept input data in different languages, such as Chinese or English. The visual data can include at least one of static image data and dynamic video data.

[0082] Optionally, the process of obtaining text prompt words includes: obtaining user input data, and obtaining text prompt words based on the user input data. The types of the user input data include: text type, voice type, image type, or video type. In response to the type of the user input data being text type, the text type data can be directly used as the text prompt words; in response to the type of the user input data being voice type, image type, or video type, the text prompt words are obtained by extracting content from the voice type data, image type data, or video type data. Through this optional embodiment, the user's usage threshold is lowered, making it easier for the user to use, and the user can fully input the required information so that the generation of visual data matching the text prompt words better meets the user's needs.

[0083] Optionally, the process of obtaining a text prompt word based on user input data includes: obtaining an initial text prompt word based on the user input data, dividing the initial text prompt word into segments, and determining an attention weight for each segment, thereby obtaining a text prompt word including multiple segments and an attention weight for each segment. In an optional embodiment, the process of determining the attention weight for each segment includes: obtaining an attention weight for each segment based on user input, or obtaining an attention weight for each segment based on the segment type of each segment. In an optional embodiment, the segment type of a segment can be determined by the semantics or text content of the segment, for example: if the semantics of the segment indicates information about the subject object (such as character image features, character personality features, or character action features) or lens information (such as horizontal angle, overhead angle, lens from near to far, or composition features), then the segment type of the segment is a high-weight segment type; if the semantics of the segment indicates background information (such as sky color features, material features of edge ornaments, etc.), then the segment type of the segment is a low-weight segment type; or, if the text content of the segment is matched to contain text of a preset high-weight segment type, then the segment type of the segment is determined to be a high-weight segment type; if the text content of the segment is matched to contain text of a preset low-weight segment type, then the segment type of the segment is determined to be a low-weight segment type, and so on. Through the above optional embodiments, the user's usage threshold is also lowered, and based on this process, the attention weight that makes the generation of visual data more accurate is quickly obtained.

[0084] A segment is the smallest unit that can be used independently. For example, if the initial text prompt is "a brown puppy," it can be divided into three segments: "a," "brown," and "puppy." Another example is if the initial text prompt is "Apples in the tree," it can be divided into four segments: "Apples," "in," "the," and "tree."

[0085] In one example, obtaining the attention weight of each segment based on user input may include: after dividing multiple segments, obtaining the attention weight assigned by the user to each segment. In this way, the attention weight of each segment can be determined by the user. For example, if the user wants to emphasize the content of a certain segment in the text prompt, a higher attention weight can be assigned to the segment; if the user wants to downplay the content of a certain segment in the text prompt, a lower attention weight can be assigned to the segment.

[0086] In another example, obtaining the attention weight of each segment based on the segment type of each segment may include: after dividing a plurality of segments, identifying whether each segment is a segment of the target segment type, and when a segment is identified as a segment of the target segment type, assigning a corresponding attention weight to the segment. In this way, the attention weight of each segment can be automatically assigned by the data processing platform. The target segment type may include at least one of a high-weight segment type and a low-weight segment type. For example, when the segment type of a segment is identified as a high-weight segment type, the data processing platform may determine the segment as an emphasized segment and assign a higher attention weight to the emphasized segment; when the segment type of a segment is identified as a low-weight segment type, the data processing platform may determine the segment as a weakened segment and assign a lower attention weight to the weakened segment. An emphasized segment may refer to a segment that needs to be emphasized in a text prompt word, and a weakened segment may refer to a segment that needs to be weakened in a text prompt word.

[0087] S120, encoding the text prompt word into an initial text embedding vector.

[0088] The data processing platform may include a text encoder, which may be used to convert text into a vector for representation. Therefore, the data processing platform may input a text prompt word into the text encoder so that the text prompt word may be encoded into an initial text embedding vector through the text encoder.

[0089] The types of text encoders may include, but are not limited to, CLIP (Contrastive Language-Image Pre-training) models and T5 Encoder (Transfer Text-to-Text Transformer) models.

[0090] S130: Perform weighted processing on each dimension of the initial text embedding vector according to the attention weight of each sentence segment to obtain the target text embedding vector.

[0091] The data processing platform can directly weight each dimension of the initial text embedding vector output by the text encoder according to the attention weight pre-assigned to each sentence segment, so as to obtain the weighted target text embedding vector (text embedding).

[0092] S140, generating visual data matching the text prompt word based on the target text embedding vector.

[0093] Optionally, the data processing platform may include a text graph model based on a diffusion model structure. The text graph model based on the diffusion model structure can be used to convert a vector into an image or video for representation. Therefore, the data processing platform can input the target text embedding vector into the text graph model based on the diffusion model structure as a conditioning mechanism, so that visual data matching the text prompt word can be generated based on the target text embedding vector through the text graph model, wherein the visual data may include at least one of static image data and dynamic video data.

[0094] Among them, the types of Wensheng graph models may include but are not limited to: UNet (Convolutional Networks for Biomedical Image Segmentation, deep learning segmentation network), UviT (A ViT Backbone for Diffusion Models, ViT backbone for diffusion models).

[0095] In the embodiment of the present application, since the target text embedding vector is a vector obtained after weighted processing of the text prompt word, and the attention weight used in the weighted processing is assigned by the user himself or automatically assigned by the data processing platform for the target sentence type, it is possible to directly control the attention of the text graph model to the content of the text prompt word from the bottom layer, that is, to increase the attention of the text graph model to the sentence segments with high attention weights, or to reduce the attention of the text graph model to the sentence segments with low attention weights, so that the visual data generated by the text graph model based on the target text embedding vector is more in line with the effect to be presented by the text prompt word, thereby not requiring any additional training and fine-tuning, saving a lot of time and resources, and not needing to disrupt the original order and structure of the text prompt words, avoiding text frustration.

[0096] In an embodiment of the present application, by dividing a number of segments and assigning each segment its own attention weight, the segments with higher attention weights will be able to obtain more attention weights when weighting the initial text embedding vector partitions. This can control the importance of each segment in the entire text prompt word, thereby systematically and accurately controlling the attention of the text prompt word content in a complex text prompt word environment.

[0097] As can be seen from this example, the solution provided by this application obtains text prompt words; wherein the text prompt words include multiple segments, each of which has a pre-assigned attention weight; the text prompt words are encoded into an initial text embedding vector; according to the attention weight of each segment, each dimension of the initial text embedding vector is weighted to obtain a target text embedding vector; based on the target text embedding vector, visual data matching the text prompt words is generated. This application directly weights each dimension of the initial text embedding vector of the text prompt words according to the attention weights pre-assigned to each segment, so as to directly control the strength of attention to the content of the text prompt words from the bottom layer, so that the generated visual data is more in line with the effect to be presented by the text prompt words, thereby not requiring any additional training and fine-tuning, saving a lot of time and resources, and not needing to disrupt the original order and structure of the text prompt words, avoiding text frustration, and this application can accurately assign corresponding attention size to the content of the text prompt words through partition weighting, thereby effectively dealing with the situation of complex text prompt words.

[0098] Figure 2 This is another flowchart of the visual data generation method shown in an embodiment of the present application.

[0099] See also Figure 2 , the visual data generation method of the present application comprises:

[0100] S210, obtaining a text prompt word; wherein the text prompt word includes a plurality of segments, and each segment has a pre-assigned attention weight.

[0101] Figure 3 This is a schematic diagram of the architecture of the data processing platform shown in the embodiment of the present application. Figure 3 As shown, the data processing platform obtains a text prompt word s, wherein the text prompt word s may include m sentence segments: s 1 ,s 2 ,...,s m , m is an integer and m≥1. Each sentence segment has a pre-assigned attention weight: w 1 ,w 2 ,...,w m .

[0102] In one embodiment, obtaining a text prompt word may include:

[0103] Obtain user input data; obtain text prompt words based on the user input data.

[0104] The user input data may be of text, voice, image, or video type. If the user input data is of text type, the text data may be directly used as a text prompt. If the user input data is of voice, image, or video type, the text prompt may be obtained by extracting content from the voice, image, or video data. This optional embodiment lowers the user's threshold for use, facilitating user usage and enabling users to fully input their desired information, thereby enabling the generation of visual data matching the text prompt to better meet their needs.

[0105] In one embodiment, obtaining a text prompt word based on user input data may include:

[0106] An initial text prompt word is obtained according to the user's input data; the initial text prompt word is divided into segments, and the attention weight of each segment is determined to obtain a text prompt word including multiple segments and the attention weight of each segment.

[0107] Among them, the process of determining the attention weight of each sentence segment may include: obtaining the attention weight of each sentence segment according to user input, or obtaining the attention weight of each sentence segment according to the sentence type of each sentence segment. Specifically, the sentence type of the sentence segment can be determined by the semantics or text content of the sentence segment, for example: if the semantics of the sentence segment indicates the information of the subject object (such as character image characteristics, character personality characteristics or character action characteristics) or lens information (such as horizontal angle, overhead angle, lens from near to far or composition characteristics), then the sentence type of the sentence segment is a high-weighted sentence type. If the semantics of the sentence segment indicates background information (such as sky color characteristics, material characteristics of edge ornaments, etc.), then the sentence type of the sentence segment is a low-weighted sentence type; or, if the text content of the sentence segment contains a text of a preset high-weighted sentence type, then the sentence type of the sentence segment is determined to be a high-weighted sentence type, and if the text content of the sentence segment is matched to contain a text of a preset low-weighted sentence type, then the sentence type of the sentence segment is determined to be a low-weighted sentence type, and so on. Through the above optional embodiments, the user's usage threshold is also lowered, and based on this process, the attention weight that makes the generation of visual data more accurate is quickly obtained.

[0108] In one example, obtaining the attention weight of each segment based on user input may include: after dividing multiple segments, obtaining the attention weight assigned by the user to each segment. In this way, the attention weight of each segment can be determined by the user. For example, if the user wants to emphasize the content of a certain segment in the text prompt, a higher attention weight can be assigned to the segment; if the user wants to downplay the content of a certain segment in the text prompt, a lower attention weight can be assigned to the segment.

[0109] In another example, obtaining the attention weight of each segment according to the segment type of each segment may include: after dividing a plurality of segments, identifying whether each segment is a segment of the target segment type, and when a segment is identified as a segment of the target segment type, assigning a corresponding attention weight to the segment. In this way, the attention weight of each segment can be automatically assigned by the data processing platform. The target segment type may include at least one of a high-weight segment type and a low-weight segment type. For example, when the segment type of a segment is identified as a high-weight segment type, the data processing platform may determine the segment as an emphasized segment and assign a higher attention weight to the emphasized segment; when the segment type of a segment is identified as a low-weight segment type, the data processing platform may determine the segment as a weakened segment and assign a lower attention weight to the weakened segment.

[0110] In addition, except for emphasized and weakened segments, other segments in the text prompt words are ordinary segments. Ordinary segments have default attention weights, so the data processing platform does not need to assign corresponding attention weights to ordinary segments.

[0111] Therefore, the segments in the text prompt word may include at least one of an emphasized segment, an ordinary segment and a weakened segment, wherein the emphasized segment may refer to the segment that needs to be emphasized in the text prompt word, the weakened segment may refer to the segment that needs to be weakened in the text prompt word, and the ordinary segment may refer to the other segments in the text prompt word except the emphasized segment and the weakened segment, wherein the attention weight of the emphasized segment is greater than the attention weight of the ordinary segment and the attention weight of the weakened segment.

[0112] S220 , dividing each sentence segment in the text prompt word into word units in order to obtain a plurality of word units.

[0113] The data processing platform may include a tokenizer, which can be used to split a patch into tokens for representation, such as Figure 3As shown, after obtaining the text prompt word, the data processing platform can input the text prompt word into the word segmenter so that the word segmenter can sequentially divide each sentence segment in the text prompt word into word units, wherein each sentence segment corresponds to at least one word unit, so multiple word units can be obtained. Specifically, for a certain sentence segment s i ,i=1,2,...,m,the word segmenter can i Divided into n i A sequence of tokens: n i is an integer and n i ≥1.

[0114] By dividing each sentence into lemmas in sequence, we can split words and roots in natural language into smaller language units, so that the subsequent text graph model can better understand and process the text content. Therefore, a lemma can be referred to as the smallest unit that carries the semantic information of the text.

[0115] As an example, for the "puppy" segment, "small" is an adjective and "dog" is a noun, both of which are the smallest units that carry text semantic information. Therefore, the word segmenter can divide the "puppy" segment into two tokens, namely "small" and "dog". As another example, for the "Apples" segment, "App" is the abbreviation of Application and "les" is the abbreviation of LinearExpenditure system. Both of which are the smallest units that carry text semantic information. Therefore, the word segmenter can divide the "Apples" segment into two tokens, namely "App" and "les". As another example, for the "tree" segment, "tree" is the smallest unit that carries text semantic information. Therefore, the word segmenter can divide the "tree" segment into one token, namely "tree".

[0116] It should be noted that, depending on the different word segmenters, some of the word units in the text prompt word may be special word units, which may include at least one of the following: the start word unit, the end word unit, the unknown word unit, and the placeholder. The start word unit may refer to the sentence start identifier, for example <go>or <sos>; The terminal token can refer to the end-of-sentence identifier, such as <eos>;Unknown words can refer to low-frequency words or words that are not in the vocabulary, such as <unk>Placeholders can refer to identifiers used to dynamically insert values, parameters, or formats, such as <%v>. Since these special tokens do not represent any words in the text prompt, they do not participate in the final semantic presentation. Therefore, the attention weights of special tokens can be fixed to the default 1.0.

[0117] S230: Encode each word to obtain an initial word embedding vector for each word.

[0118] like Figure 3 As shown, the n corresponding to each sentence segment output by the word segmenter is obtained i After the word-unit sequence is obtained, the data processing platform can connect the word-unit sequences of all the segments end to end, and then input the entire sequence into the text encoder to encode the initial text embedding vector of the text prompt word. Specifically, the text encoder can encode each word-unit to obtain the initial word-unit embedding vector of each word-unit. In order to facilitate distinction, for a certain word-unit In this embodiment of the present application, the word The initial word embedding vector of After getting the initial word embedding vectors of all words After that, the embodiment of the present application can embed the initial word of each word into the vector By stacking them in sequence, we can get the initial text embedding vector of the text prompt word. Therefore, the initial text embedding vector of the text prompt word and the initial word embedding vector of each word are There is a position correspondence between them, that is, the initial text embedding vector can include the initial word unit embedding vector of each word unit, and each dimension of the initial text embedding vector corresponds to the initial word unit embedding vector of a word unit.

[0119] S240, performing weighted processing on the initial word-unit embedding vector of each word-unit according to the attention weight of each sentence segment to obtain the target word-unit embedding vector of each word-unit.

[0120] The data processing platform can directly embed the initial word element vector of each word element output by the text encoder according to the attention weight pre-assigned to each sentence segment. Perform weighted processing to obtain the weighted target word embedding vector for each word.

[0121] In order to facilitate the distinction, for a word In this embodiment of the present application, the word The weighted target word embedding vector is marked as The target word embedding vector after all word weights are obtained After that, the embodiment of the present application can embed the weighted target word of each word into the vector By stacking them in sequence, we can get the target text embedding vector of the text prompt word. Therefore, the target text embedding vector of the text prompt word and the target word embedding vector of each word are There is also a positional correspondence between them, that is, the target text embedding vector can include the target word embedding vector of each word, and each dimension of the target text embedding vector corresponds to the target word embedding vector of a word.

[0122] In one embodiment, weighting the initial word-unit embedding vector of each word-unit according to the attention weight of each sentence segment to obtain the target word-unit embedding vector of each word-unit may include:

[0123] The target word embedding vector of each word is calculated by multiplying the value of the initial word embedding vector of each word with the attention weight of the corresponding sentence segment.

[0124] The size of the vector modulus is proportional to the level of attention paid to the corresponding word-unit; that is, the size of the vector modulus is related to the contribution of the vector modulus to the visual data generation process. For example, if the initial text prompt is "a brown puppy," the vector moduli of the two words "small" and "dog" may be relatively small. Therefore, when subsequently input into the text-graph model, the text-graph model pays less attention to both "small" and "dog," causing it to easily overlook "small" and "dog." As a result, the text-graph model does not fully understand the concepts of "small" and "dog." As a result, the visual data generated by the text-graph model lacks a clear representation of "puppy," for example, resembling a puppy, a kitten, a wolf, or some other small animal.

[0125] To this end, the data processing platform can embed the initial word element vector of each word element output by the text encoder according to the attention weight pre-assigned to each sentence segment. Perform weighted processing to adjust the size of the vector module of each word, thereby adjusting the attention of the text graph model to each word. Specifically, the data processing platform can embed the initial word of each word into the vector The value of and the attention weight w of the corresponding sentence i Multiply them together to get the target word embedding vector for each word. Figure 3 As shown, for i = 1, 2, ..., m, j = 1, 2, ..., n i , assignment Weighted vector That is the word The final target word embedding vector.

[0126] In one example, assuming that the attention weight of the "puppy" sentence is 1.2, the initial word embedding vector values ​​of the two words "small" and "dog" can be multiplied by 1.2 to obtain the target word embedding vectors of the two words "small" and "dog", respectively, thereby enhancing the attention of the text graph model to "small" and "dog", making the text graph model pay more attention to "small" and "dog", and allowing the text graph model to better understand the concepts of "small" and "dog", so that "puppy" is more obvious in the visual data generated by the text graph model.

[0127] S250: Obtain preset noise.

[0128] The training of the Wensheng graph model can be divided into two parts: the forward process and the reverse process. The forward process is to generate random noise x by adding Gaussian noise ε to the original visual data x0. T , that is, the purpose of the forward process is to transform the original visual data x0 into random noise x T , and the purpose of the reverse process is to convert the random noise x T Restore the original visual data x0, that is, the reverse process is to predict the noise ε θ , and then calculate the predicted noise ε θ The loss value of Gaussian noise ε is used to update the Wensheng graph model, thereby realizing the random noise x T Gradually remove the predicted noise ε θ , and finally restore the original visual data x0. When the loss value is less than the preset threshold, the trained text graph model is obtained.

[0129] Therefore, after obtaining the weighted target text embedding vector, the embodiment of the present application can randomly generate a noise matrix x with a noise map T As preset noise.

[0130] S260, performing feature fusion on the target word-unit embedding vector of each word-unit and the preset noise to obtain the word-unit fusion feature of each word-unit.

[0131] In the embodiment of the present application, the target word of each word can be embedded into the vector and preset noise x T Input into the trained text graph model so that the target word of each word can be embedded into the vector through the text graph model With preset noise x T Perform feature fusion to obtain the word-unit fusion features of each word.

[0132] S270: Generate visual data matching the text prompt word based on the word-gram fusion features of each word-gram.

[0133] Through the word-unit fusion features of each word, the text graph model can be guided to predict the noise ε in the direction of the text prompt word θ , then the Wensheng graph model will preset the noise x T Gradually remove the predicted noise ε θ Since the text-based graph model pays more attention to the word-merge fusion features with higher attention weights and less attention to the word-merge fusion features with lower attention weights, the visual data generated by the text-based graph model based on the word-merge fusion features of each word can be more consistent with the effect that the text prompt words want to present.

[0134] The visual data may include at least one of static image data and dynamic video data. In the embodiment of the present application, the following example is described using static image data:

[0135] In one example, assuming that the initial text prompt word obtained based on the user's input data is: "a brown puppy", the initial text prompt word can be divided into three segments: "a", "brown", and "puppy". Since the text graph model may have a weak understanding of the "puppy" segment, that is, the text graph model pays less attention to the "puppy" segment, the text graph model generates an image of "a brown puppy", and the "puppy" in the image is not clearly shown, such as Figure 4A As shown, Figure 4A This is the image of "a brown puppy" generated by the Wensheng graph model without increasing the attention weight of the "puppy" segment. It can be seen that Figure 4A The "puppy" in the image is not obvious enough, for example, it looks like a puppy, a kitten, a wolf, or other small animals. In order to make the "puppy" in the image more obvious, the "puppy" segment can be used as an emphasis segment, so that a higher attention weight is assigned to the "puppy" segment. For example, the attention weight of the "puppy" segment can be set from the default 1.0 to a higher value, such as 1.2. Since "one" and "brown" are obvious enough in the image, the "one" and "brown" segments are used as ordinary segments, so that the "one" and "brown" segments are assigned default attention weights. For example, the attention weights of the "one" and "brown" segments can be maintained at the default 1.0, thereby obtaining a text prompt word including multiple segments and the attention weights of each segment. Through this operation, the attention of the text graph model to the "puppy" segment can be enhanced, so as to increase the attention of the text graph model to the "puppy" segment, so that the image data generated by the text graph model can better meet the effect that the user wants to present, such as Figure 4B As shown, Figure 4B This is the image of "a brown puppy" generated by the Wensheng graph model when the attention weight of the "puppy" segment is increased. It can be seen that Figure 4B The "puppy" in the film is more obvious.

[0136] In another example, assuming that the initial text prompt word obtained based on the user's input data is: "Apples in the tree", the initial text prompt word can be divided into four segments: "Apples", "in", "the", and "tree". Since the text graph model may have a stronger understanding of the "in" segment, that is, the text graph model pays more attention to the "in" segment, the text graph model generates an image of "Apples in the tree". All apples in the image are hanging on the tree. This may result in some apples being ripe, but some apples not being ripe, such as Figure 5A As shown, Figure 5A This is the image of "Apples in the tree" generated by the text graph model without reducing the attention weight of the "in" segment. When the user wants to highlight some apples falling under the tree in the image, the apples falling under the tree indicate that the apples are ripe, so as to present the effect of abundant fruits and a good harvest. In order to achieve this effect, the "in" segment can be used as a weakened segment, and the other segments can be used as ordinary segments. Therefore, a lower attention weight can be assigned to the "in" segment. For example, the attention weight of the "in" segment can be set from the default 1.0 to a lower value, such as 0.8, and the default attention weights can be assigned to the "Apples", "the" and "tree" segments. For example, the attention weights of the "Apples", "the" and "tree" segments can be maintained at the default 1.0, thereby obtaining a text prompt word including multiple segments and the attention weights of each segment. Through this operation, the attention of the text graph model to the "in" segment can be weakened, so as to reduce the attention of the text graph model to the "in" segment, so that the image data generated by the text graph model is more in line with the effect that the user wants to present, such as Figure 5B As shown, Figure 5B This is the "Apples in the tree" image generated by the Wensheng graph model when the attention weight of the "in" segment is reduced. It can be seen that Figure 5B If some apples fall from the tree, it can create an effect of abundant fruits and a good harvest.

[0137] In another example, suppose that the initial text prompt words obtained according to the user's input data are: "Red lanterns are hung in the bamboo forest in the park, pink plum blossoms are in full bloom next to it, and there is a pavilion in the distance. The sun shines on the winding park path, presenting a warm feeling." It can be seen that the initial text prompt words are very complex. Therefore, the embodiment of the present application can divide the initial text prompt words into several sentence segments: "in the park", "bamboo forest", "hanging", "red", "lantern", "next to", "blooming", "pink", "plum blossoms", "in the distance is", "a", "pavilion", "sunlight", "sprinkled on", "winding", "park", "path", "presenting", "warm", "feeling", and each sentence segment can be assigned with different The attention weight of each sentence is obtained, thereby obtaining a text prompt word including multiple sentences and the attention weight of each sentence segment. The sentence segment with a higher attention weight will be able to obtain more attention weights when the initial text embedding vector partition is weighted. In this way, the user can control the importance of each sentence segment in the entire text prompt word, thereby accurately controlling the attention of the text prompt word content in an orderly manner in the environment of complex text prompt words. For example, if the user wants to highlight elements such as "bamboo forest", "lantern", "plum blossom", and "path" in the image, a higher attention weight can be assigned to the "bamboo forest", "lantern", "plum blossom", and "path" segments, and the other segments can maintain the default attention weight. In this way, the image data generated by the text graph model is more in line with the effect that the user wants to present, such as Figure 6 As shown, Figure 6 The image generated by the Wenshengtu model shows "red lanterns hanging in the bamboo forest in the park, pink plum blossoms blooming next to it, a pavilion in the distance, and sunlight shining on the winding park path, creating a warm feeling." Figure 6 To highlight elements such as "bamboo forest", "lantern", "plum blossom", and "path", the embodiment of the present application weights the partitions of the text prompt words, thereby providing accurate and stable allocation of attention weights when the text prompt words are very complex.

[0138] As can be seen from this example, the solution provided by this application can enhance the attention of the text graph model to the sentence segment from the bottom layer by increasing the attention weight of the sentence segment to be emphasized, so as to increase the attention of the text graph model to the sentence segment, and by reducing the attention weight of the sentence segment to be weakened, the attention of the text graph model to the sentence segment can be weakened from the bottom layer, so as to reduce the attention of the text graph model to the sentence segment, so that the visual data generated by the text graph model is more consistent with the effect to be presented by the text prompt word, thereby not requiring any additional training and fine-tuning, saving a lot of time and resources, and avoiding disrupting the order and structure of the original text prompt word. Further, by dividing the initial text prompt word into several segments and assigning each segment a respective attention weight, a text prompt word including multiple segments and the attention weights of each segment is obtained. The segments with higher attention weights will be able to obtain more attention weights when the initial text embedding vector partitions are weighted. In this way, the user can control the importance of each segment in the entire text prompt word, thereby systematically and accurately controlling the attention of the text prompt word content in the environment of complex text prompt words.

[0139] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides a visual data generating device, an electronic device, a computer-readable storage medium and corresponding embodiments.

[0140] Figure 7 It is a structural diagram of the visual data generating device shown in an embodiment of the present application.

[0141] See also Figure 7 , the present application provides a visual data generating device, which may include:

[0142] The text prompt word acquisition module 710 is used to acquire text prompt words; wherein the text prompt words include multiple segments, each of which has a pre-assigned attention weight;

[0143] An encoding processing module 720 is used to encode the text prompt word into an initial text embedding vector;

[0144] A weighted processing module 730 is used to perform weighted processing on each dimension of the initial text embedding vector according to the attention weight of each sentence segment to obtain a target text embedding vector;

[0145] The visual data generation module 740 is used to generate visual data matching the text prompt word based on the target text embedding vector.

[0146] In one embodiment, before encoding the text prompt word into the initial text embedding vector, the apparatus may further include:

[0147] A word unit segmentation module is used to sequentially segment each sentence in the text prompt word into word units to obtain multiple word units;

[0148] The initial text embedding vector includes the initial word embedding vector of each word; the encoding processing module 720 may include:

[0149] The encoding processing submodule is used to encode each word unit to obtain the initial word unit embedding vector of each word unit.

[0150] In one embodiment, the target text embedding vector includes a target word embedding vector for each word; the weighting processing module 730 may include:

[0151] The weighted processing submodule is used to perform weighted processing on the initial word unit embedding vector of each word unit according to the attention weight of each sentence segment to obtain the target word unit embedding vector of each word unit.

[0152] In one embodiment, the weighted processing submodule may include:

[0153] The weighted processing unit is used to calculate the target word unit embedding vector of each word unit based on the value of the initial word unit embedding vector of each word unit and the attention weight of the corresponding sentence segment.

[0154] In one embodiment, the visual data generation module 740 may include:

[0155] A preset noise acquisition submodule is used to acquire the preset noise;

[0156] The feature fusion submodule is used to fuse the target word embedding vector of each word with the preset noise to obtain the word fusion feature of each word;

[0157] The visual data generation submodule is used to generate visual data matching the text prompt words based on the word-gram fusion features of each word-gram.

[0158] In one embodiment, the text prompt word acquisition module 710 may include:

[0159] Input data acquisition submodule, used to obtain user input data;

[0160] The input data processing submodule is used to obtain text prompt words according to the user's input data.

[0161] In one embodiment, the input data processing submodule may include:

[0162] An input data processing unit, configured to obtain an initial text prompt word based on the user's input data;

[0163] The initial text prompt word processing unit is used to divide the initial text prompt word into segments and determine the attention weight of each segment to obtain a text prompt word including multiple segments and the attention weight of each segment.

[0164] As can be seen from this example, the solution provided by this application obtains text prompt words; wherein the text prompt words include multiple segments, each of which has a pre-assigned attention weight; the text prompt words are encoded into an initial text embedding vector; according to the attention weight of each segment, each dimension of the initial text embedding vector is weighted to obtain a target text embedding vector; based on the target text embedding vector, visual data matching the text prompt words is generated. This application directly weights each dimension of the initial text embedding vector of the text prompt words according to the attention weights pre-assigned to each segment, so as to directly control the strength of attention to the content of the text prompt words from the bottom layer, so that the generated visual data is more in line with the effect to be presented by the text prompt words, thereby not requiring any additional training and fine-tuning, saving a lot of time and resources, and not needing to disrupt the original order and structure of the text prompt words, avoiding text frustration, and this application can accurately assign corresponding attention size to the content of the text prompt words through partition weighting, thereby effectively dealing with the situation of complex text prompt words.

[0165] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0166] Figure 8 It is a structural diagram of an electronic device shown in an embodiment of the present application.

[0167] See also Figure 8 , the electronic device 800 includes a memory 810 and a processor 820.

[0168] The processor 820 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0169] The memory 810 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 820 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that retains stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (e.g., a magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (e.g., a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all instructions and data required by the processor during operation. In addition, the memory 810 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. In some embodiments, the memory 810 may include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0170] The memory 810 stores executable codes. When the executable codes are processed by the processor 820 , the processor 820 may execute part or all of the above-mentioned methods.

[0171] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0172] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device, the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0173] The embodiments of the present application have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to the technology in the market, or to enable other persons skilled in the art to understand the embodiments disclosed herein.< / unk> < / eos> < / sos> < / go>

Claims

1. A method for generating visual data, characterized in that: include: Acquire a text prompt word; wherein the text prompt word includes a plurality of sentence segments, each of which has a pre-assigned attention weight; Encoding the text prompt word into an initial text embedding vector; According to the attention weights of the respective sentence segments, weighted processing is performed on the dimensions of the initial text embedding vector to obtain a target text embedding vector; Based on the target text embedding vector, visual data matching the text prompt word is generated.

2. The method according to claim 1, characterized in that Before encoding the text prompt word into an initial text embedding vector, the method further includes: Sequentially dividing each sentence segment in the text prompt word into word units to obtain a plurality of word units; The initial text embedding vector includes the initial word-element embedding vector of each word-element; encoding the text prompt word into the initial text embedding vector includes: Each word unit is encoded to obtain an initial word unit embedding vector of each word unit.

3. The method according to claim 2, characterized in that The target text embedding vector includes the target word-element embedding vector of each word-element; and the weighted processing of each dimension of the initial text embedding vector according to the attention weights of each sentence segment to obtain the target text embedding vector includes: According to the attention weights of the various sentence segments, the initial word unit embedding vectors of the various word units are weighted to obtain the target word unit embedding vectors of the various word units.

4. The method according to claim 3, characterized in that The step of performing weighted processing on the initial word unit embedding vectors of the word units according to the attention weights of the sentence segments to obtain the target word unit embedding vectors of the word units includes: According to the value of the initial word unit embedding vector of each word unit and the attention weight of the corresponding sentence segment, the target word unit embedding vector of each word unit is calculated.

5. The method according to claim 3, characterized in that: The step of generating visual data matching the text prompt word based on the target text embedding vector comprises: Get preset noise; Performing feature fusion of the target word unit embedding vector of each word unit and the preset noise to obtain word unit fusion features of each word unit; Based on the word-gram fusion features of the respective word-grams, visual data matching the text prompt word is generated.

6. The method according to any one of claims 1 to 5, characterized in that: The obtaining of text prompt words includes: Get user input data; A text prompt word is obtained according to the input data of the user.

7. The method according to claim 6, characterized in that The obtaining of a text prompt word according to the user's input data includes: Obtaining an initial text prompt word according to the user's input data; The initial text prompt word is divided into segments, and the attention weight of each segment is determined to obtain a text prompt word including multiple segments and the attention weight of each segment.

8. A visual data generating device, characterized in that: include: A text prompt word acquisition module, used to acquire text prompt words; wherein the text prompt words include a plurality of sentence segments, each of which has a pre-assigned attention weight; An encoding processing module, used for encoding the text prompt word into an initial text embedding vector; A weighted processing module, used to perform weighted processing on each dimension of the initial text embedding vector according to the attention weights of each sentence segment to obtain a target text embedding vector; A visual data generation module is used to generate visual data matching the text prompt word based on the target text embedding vector.

9. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 7.

Images