Visual data generation method and apparatus, electronic device, and storage medium

Abstract

The application relates to a visual data generation method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining a text prompt word; wherein the text prompt word comprises multiple sentence segments, and each sentence segment has a pre-assigned attention weight; encoding the text prompt word into an initial text embedding vector; performing weighted processing on each dimension of the initial text embedding vector according to the attention weight of each sentence segment to obtain a target text embedding vector; and generating visual data matched with the text prompt word based on the target text embedding vector. The scheme provided in the application can save a large amount of time and resources without any additional training and fine-tuning, can avoid disturbing the original order and structure of the text prompt word, can avoid text loss, and can effectively deal with the situation of a complex text prompt word.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a visual data generation method and device, electronic equipment and storage medium. BACKGROUND

[0002] A text-to-image model can directly generate an image with a concept described by a user input text (or text prompt). However, sometimes, the user wants to emphasize the content of a certain sentence segment in the text prompt so that the generation result of the model is more biased towards the emphasized part. This emphasis is equivalent to transferring the attention of the model to the emphasized part.

[0003] There are two methods in the related art to achieve the above purpose. The first method is to change the understanding of the model itself for a specific concept through fine-tuning, and the second method is to rearrange the text prompt to implicitly adjust the encoding result of the text prompt. However, the first method requires additional training and fine-tuning, resulting in a large consumption of time and resources, and the second method requires disturbing the original order and structure of the text prompt, resulting in text loss. Moreover, both methods cannot effectively deal with complex text prompts. SUMMARY

[0004] To solve or partially solve the problems in the related art, the present application provides a visual data generation method, device, electronic equipment and storage medium, which neither requires any additional training and fine-tuning to save a large amount of time and resources, nor disturbs the original order and structure of the text prompt to avoid text loss, and can effectively deal with complex text prompts.

[0005] The first aspect of the present application provides a visual data generation method, comprising:

[0006] obtaining a text prompt; wherein the text prompt comprises a plurality of sentence segments, each sentence segment having a pre-assigned attention weight;

[0007] encoding the text prompt into an initial text embedding vector;

[0008] weighting each dimension of the initial text embedding vector according to the attention weight of each sentence segment to obtain a target text embedding vector;

[0009] generating visual data matching the text prompt based on the target text embedding vector. In an embodiment, before the text prompt is encoded into an initial text embedding vector, the method further comprises:

[0010] performing token division on each sentence segment in the text prompt in order to obtain a plurality of tokens;

[0011] The initial text embedding vector includes initial token embedding vectors of each token; and the encoding of the text prompt into an initial text embedding vector includes:

[0012] The encoding of each token is processed to obtain the initial token embedding vector of each token.

[0013] In an embodiment, the target text embedding vector includes target token embedding vectors of each token; and the weighting of each dimension of the initial text embedding vector according to the attention weight of each sentence segment to obtain a target text embedding vector includes:

[0014] The initial token embedding vector of each token is weighted according to the attention weight of each sentence segment to obtain the target token embedding vector of each token.

[0015] In an embodiment, the weighting of the initial token embedding vector of each token according to the attention weight of each sentence segment to obtain the target token embedding vector of each token includes:

[0016] The target token embedding vector of each token is calculated according to the value of the initial token embedding vector of each token and the attention weight of the corresponding sentence segment.

[0017] In an embodiment, the generation of visual data matching the text prompt based on the target text embedding vector includes:

[0018] Obtaining a preset noise;

[0019] The target token embedding vector of each token is fused with the preset noise to obtain a token fusion feature of each token;

[0020] The visual data matching the text prompt is generated based on the token fusion feature of each token.

[0021] In an embodiment, the obtaining of the text prompt includes:

[0022] Obtaining input data of a user;

[0023] The text prompt is obtained according to the input data of the user.

[0024] In an embodiment, the obtaining of the text prompt according to the input data of the user includes:

[0025] An initial text prompt is obtained according to the input data of the user;

[0026] The initial text prompt is segmented, and attention weights of each segment are determined, to obtain a text prompt including multiple segments and the attention weights of each segment.

[0027] The second aspect of the application provides a visual data generation device, comprising:

[0028] A text prompt acquisition module is configured to acquire a text prompt, wherein the text prompt includes multiple segments, and each segment has a pre-assigned attention weight.

[0029] An encoding processing module is configured to encode the text prompt into an initial text embedding vector.

[0030] A weighting processing module is configured to perform weighting processing on each dimension of the initial text embedding vector according to the attention weights of each segment, to obtain a target text embedding vector.

[0031] A visual data generation module is configured to generate visual data matching the text prompt based on the target text embedding vector.

[0032] In an embodiment, before the text prompt is encoded into an initial text embedding vector, the device further comprises:

[0033] A wordpiece segmentation module is configured to sequentially segment each segment in the text prompt into wordpieces, to obtain multiple wordpieces.

[0034] The initial text embedding vector includes an initial wordpiece embedding vector of each wordpiece; and the encoding processing module includes:

[0035] An encoding processing submodule is configured to perform encoding processing on each wordpiece, to obtain the initial wordpiece embedding vector of each wordpiece.

[0036] In an embodiment, the target text embedding vector includes a target wordpiece embedding vector of each wordpiece; and the weighting processing module includes:

[0037] A weighting processing submodule is configured to perform weighting processing on the initial wordpiece embedding vector of each wordpiece according to the attention weight of the corresponding segment, to obtain the target wordpiece embedding vector of each wordpiece.

[0038] In an embodiment, the weighting processing submodule includes:

[0039] A weighting processing unit is configured to calculate the target wordpiece embedding vector of each wordpiece according to the value of the initial wordpiece embedding vector of each wordpiece and the attention weight of the corresponding segment.

[0040] In an embodiment, the visual data generation module includes:

[0041] a preset noise obtaining sub-module, configured to obtain a preset noise;

[0042] a feature fusion sub-module, configured to perform feature fusion on the target token embedding vector of each token and the preset noise to obtain token fusion features of the each token;

[0043] a visual data generation sub-module, configured to generate visual data matching the text prompt word based on the token fusion features of the each token.

[0044] In an embodiment, the text prompt word obtaining module comprises:

[0045] an input data obtaining sub-module, configured to obtain input data of a user;

[0046] an input data processing sub-module, configured to obtain a text prompt word according to the input data of the user.

[0047] In an embodiment, the input data processing sub-module comprises:

[0048] an input data processing unit, configured to obtain an initial text prompt word according to the input data of the user;

[0049] an initial text prompt word processing unit, configured to perform sentence segmentation on the initial text prompt word and determine attention weights of each sentence segment to obtain a text prompt word comprising a plurality of sentence segments and the attention weights of each sentence segment.

[0050] The third aspect of the present application provides an electronic device, comprising:

[0051] a processor; and

[0052] a memory having executable code stored thereon, when the executable code is executed by the processor, the processor performs the method as described above.

[0053] The fourth aspect of the present application provides a computer-readable storage medium having executable code stored thereon, when the executable code is executed by the processor of an electronic device, the processor performs the method as described above.

[0054] The technical solutions provided by the present application can include the following beneficial effects:

[0055] The solution provided by the present application is to obtain a text prompt word; wherein the text prompt word includes multiple segments, each of which has a pre-assigned attention weight; encode the text prompt word into an initial text embedding vector; according to the attention weight of each segment, weight the dimensions of the initial text embedding vector to obtain a target text embedding vector; based on the target text embedding vector, generate visual data matching the text prompt word. The present application directly weights the dimensions of the initial text embedding vector of the text prompt word according to the attention weight pre-assigned to each segment, so as to directly control the strength of attention to the content of the text prompt word from the bottom layer, so that the generated visual data is more consistent with the effect to be presented by the text prompt word, thereby not requiring any additional training and fine-tuning, saving a lot of time and resources, and not needing to disrupt the original order and structure of the text prompt word, avoiding text frustration, and the present application can accurately assign the corresponding attention size to the content of the text prompt word through partition weighting, thereby effectively dealing with the situation of complex text prompt words.

[0056] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail the exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0058] Figure 1 is a flow chart of a method for generating visual data according to an embodiment of the present application;

[0059] Figure 2 is another flowchart of the method for generating visual data shown in an embodiment of the present application;

[0060] Figure 3 Schematic diagram of the data processing platform shown in the embodiment of the present application;

[0061] Figures 4A-4B This is an image of a brown puppy generated by the Wensheng graph model;

[0062] Figures 5A-5B This is the “Apples in the tree” image generated by the Wensheng graph model;

[0063] Figure 6 It is an image generated by the text-generated graph model under complex text prompt words;

[0064] Figure 7 is a structural diagram of a visual data generating device shown in an embodiment of the present application;

[0065] Figure 8 FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0066] Embodiments of the present application will be described in more detail with reference to the accompanying drawings. Although embodiments of the present application are shown in the drawings, it should be understood that the present application can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0067] The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the present application and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0068] It will be understood that, although the terms "first," "second," "third," etc. can be used herein to describe various information, these terms are not intended to denote a particular order or hierarchy. These terms are used only to distinguish one from another. For example, a first information can be termed a second information, and, similarly, a second information can be termed a first information, without departing from the scope of the present application. As such, features defined with "first," "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically defined.

[0069] In the field of AI (Artificial Intelligence) image generation technology, an important branch is to generate images from text, i.e., text-to-image, or colloquially, AI painting, which can directly generate images with the concept described in the text according to the user input text (or text prompt word). However, sometimes the user wants to emphasize the content of a certain sentence in the text prompt word so that the generation result of the model is more biased towards the emphasized part.

[0070] There are two methods in the related art to achieve the above purpose, the first is to change the understanding of the model itself to a specific concept through fine-tuning, and the other is to rearrange the text prompt word through prompt word engineering to implicitly adjust the encoding result of the text prompt word.

[0071] In the first method, in order to strengthen the model's understanding of a specific concept, the model is fine-tuned using sample data containing the specific concept. This method strengthens the model's cognition by training the pre-trained model using new sample data to guide the model to pay more attention to the concept contained in the fine-tuning data.

[0072] However, the first method not only requires collecting sample data but also training the model, consuming a large amount of computing power and time. Secondly, fine-tuning a model usually changes the weights of the base model itself, affecting the model's own ability and being very unstable. Moreover, the sample data used to fine-tune the model is ambiguous to the user in most cases, and it is difficult for the user to collect sample data that accurately strengthens a specific concept, making it difficult to target the part that the user really needs to emphasize. Therefore, the first method requires high cost and it is difficult to accurately guide the model's attention to a specific concept.

[0073] Considering the difficulty of the fine-tuning method, the second method is more intuitive and feasible for ordinary users, that is, rearranging the text prompt words through prompt word engineering. In the second method, the content emphasized in the sentence can be rearranged by moving the content to be emphasized to the front, because the content at the front should be the subject of the description, and the content to be emphasized can also be mentioned multiple times in the text. Therefore, the second method conforms to human language habits, and the threshold for use and learning is low.

[0074] However, the second method implicitly guides the model's attention allocation relationship to the text by rearranging the language of the text prompt words, and since it is not explicit, the user cannot determine the attention level of each word. In addition, this approach relies heavily on the arrangement of text prompt words, and in most cases, deliberately emphasizing a concept in the text prompt words will disrupt the original order and structure of the text prompt words, resulting in text failure. This disruption is particularly pronounced in longer text prompt words, often leading to images generated by the model that do not match the text prompt words, and even image out-of-control situations. Furthermore, this approach has weak expression capabilities, and when the user wants to emphasize a concept that the model understands poorly, or wants to emphasize multiple concepts at the same time, no matter how the text prompt words are arranged, the emphasis cannot be effectively emphasized.

[0075] As can be seen, in related technologies, the expression of specific content in the model is mostly achieved by fine-tuning or rearranging the text prompt words. However, both methods have high instability, and they either require a large amount of time and resources or disrupt the original order and structure of the text prompt words, and both methods cannot be applied to complex situations, and when the text prompt words tend to be complex, these methods are almost ineffective.

[0076] To solve the above problems, the embodiment of the present application provides a visual data generation method, which directly performs weighting processing on each dimension of the initial text embedding vector of the text prompt word according to the attention weight pre-assigned to each sentence segment, so as to directly control the attention strength to the content of the text prompt word from the bottom layer, so that the generated visual data is more in line with the effect to be presented by the text prompt word, thereby saving a lot of time and resources without any additional training and fine-tuning, and without disturbing the original order and structure of the text prompt word, avoiding text frustration, and the present application can accurately assign the corresponding attention size to the content of the text prompt word through zoning weighting, thereby effectively dealing with the situation of complex text prompt word.

[0077] The technical solutions of the embodiments of the present application are described in detail below with reference to the drawings.

[0078] Figure 1 is a flowchart of the visual data generation method shown in the embodiments of the present application.

[0079] Referring to Figure 1 The visual data generation method of the present application comprises:

[0080] S110, obtaining a text prompt word; wherein the text prompt word comprises a plurality of sentence segments, and each sentence segment has a pre-assigned attention weight.

[0081] The embodiments of the present application can be applied to a data processing platform, which can obtain input data input by a user, and the input data can refer to the content required to be included in the generated visual data, for example, the input data can include object categories, colors, quantities, environments, weather, etc. in the generated visual data. The data processing platform can accept input data of different language types, such as Chinese or English input data. The visual data can include at least one of static image data and dynamic video data.

[0082] Optionally, the process of obtaining the text prompt word comprises: obtaining the input data of the user, and obtaining the text prompt word according to the input data of the user. The type of the input data of the user includes: text type, voice type, image type or video type. In response to the type of the input data of the user being the text type, the data of the text type can be directly taken as the text prompt word; in response to the type of the input data of the user being the voice type, the image type or the video type, the text prompt word is obtained by performing content extraction on the data of the voice type, the image type or the video type. Through the optional embodiment, the use threshold of the user is reduced, the use of the user is facilitated, and then the user can fully input the demand information, so that the visual data generated according to the text prompt word is more in line with the demand of the user.

[0083] Optionally, the process of obtaining the text prompt according to the input data of the user comprises: obtaining an initial text prompt according to the input data of the user, performing sentence segmentation on the initial text prompt, and determining the attention weight of each sentence segment to obtain the text prompt comprising a plurality of sentence segments and the attention weight of each sentence segment. In an optional embodiment, the process of determining the attention weight of each sentence segment comprises: obtaining the attention weight of each sentence segment according to the input of the user, or obtaining the attention weight of each sentence segment according to the sentence segment type of each sentence segment. In an optional embodiment, the sentence segment type of a sentence segment can be determined by the semantic or text content of the sentence segment. For example, if the semantic of a sentence segment indicates information of a subject object (such as a character image feature, a character personality feature, or a character action feature) or shot information (such as a level view angle, a low-angle view, a shot from near to far, or a composition feature), it indicates that the sentence segment type of the sentence segment is a high-weight sentence segment type. If the semantic of a sentence segment indicates background information (such as a sky color feature, a material feature of an edge ornament, etc.), it indicates that the sentence segment type of the sentence segment is a low-weight sentence segment type. Or, if the text content of a sentence segment matches a preset high-weight sentence segment type, it is determined that the sentence segment type of the sentence segment is a high-weight sentence segment type. If the text content of a sentence segment matches a preset low-weight sentence segment type, it is determined that the sentence segment type of the sentence segment is a low-weight sentence segment type, and so on. Through the above optional embodiments, the use threshold of the user is also reduced, and the attention weight that makes the generated visual data more accurate is quickly obtained based on the process.

[0084] wherein the sentence segment can refer to the smallest unit that can be used independently. As an example, assuming that the initial text prompt is "a brown puppy", it can be divided into 3 sentence segments, which are "a", "brown", and "puppy" respectively. As another example, assuming that the initial text prompt is "Apples in the tree", it can be divided into 4 sentence segments, which are "Apples", "in", "the", and "tree" respectively.

[0085] In an example, obtaining the attention weight of each sentence segment according to the input of the user can comprise: after dividing the plurality of sentence segments, obtaining the attention weight assigned by the user for each sentence segment. In this way, the attention weight of each sentence segment can be determined by the user. For example, if the user wants to emphasize the content of a sentence segment in the text prompt, the user can assign a higher attention weight to the sentence segment. If the user wants to weaken the content of a sentence segment in the text prompt, the user can assign a lower attention weight to the sentence segment.

[0086] In another example, obtaining the attention weight of each sentence segment according to the sentence segment type of each sentence segment can include: after the plurality of sentence segments are divided, identifying whether each sentence segment is a sentence segment of a target sentence segment type, and when a certain sentence segment is identified as a sentence segment of the target sentence segment type, assigning the corresponding attention weight to the sentence segment. In this way, the attention weight of each sentence segment can be automatically assigned by the data processing platform. The target sentence segment type can include at least one of a high-weight sentence segment type and a low-weight sentence segment type. For example, when the sentence segment type of a certain sentence segment is identified as a high-weight sentence segment type, the data processing platform can determine the sentence segment as an emphasized sentence segment and assign a higher attention weight to the emphasized sentence segment; when the sentence segment type of a certain sentence segment is identified as a low-weight sentence segment type, the data processing platform can determine the sentence segment as a weakened sentence segment and assign a lower attention weight to the weakened sentence segment. The emphasized sentence segment can refer to a sentence segment that needs to be emphasized in the text prompt, and the weakened sentence segment can refer to a sentence segment that needs to be weakened in the text prompt.

[0087] S120, encoding the text prompt into an initial text embedding vector.

[0088] The data processing platform can include a text encoder, which can be used to convert text into a vector to represent, so the data processing platform can input the text prompt into the text encoder, so that the text prompt can be encoded into an initial text embedding vector by the text encoder.

[0089] The type of the text encoder can include but is not limited to a CLIP (Contrastive Language-Image Pre-training) model, a T5 Encoder (Transfer Text-to-Text Transformer) model.

[0090] S130, weighting processing each dimension of the initial text embedding vector according to the attention weight of each sentence segment to obtain a target text embedding vector.

[0091] The data processing platform can directly weight process each dimension of the initial text embedding vector output by the text encoder according to the attention weight pre-assigned to each sentence segment, so as to obtain a weighted target text embedding.

[0092] S140, generating visual data matching the text prompt based on the target text embedding vector.

[0093] Optionally, the data processing platform can include a text-to-image model based on a diffusion model structure, which can be used to convert vectors into images or videos for representation, so the data processing platform can input the target text embedding vector as a conditioning input into the text-to-image model based on the diffusion model structure, so that the visual data matching the text prompt can be generated by the text-to-image model based on the target text embedding vector, wherein the visual data can include at least one of static image data and dynamic video data.

[0094] The type of text-to-image model can include but is not limited to UNet (Convolutional Networks for Biomedical Image Segmentation), UviT (A ViT Backbone for Diffusion Models).

[0095] In the embodiments of the present application, since the target text embedding vector is a vector obtained by weighting the text prompt, and the attention weight used in the weighting process is assigned by the user or automatically assigned by the data processing platform for the sentence segment of the target sentence segment type, the attention strength of the text-to-image model to the content of the text prompt can be directly controlled from the bottom, that is, the attention of the text-to-image model to the sentence segment with high attention weight is increased, or the attention of the text-to-image model to the sentence segment with low attention weight is reduced, so that the visual data generated by the text-to-image model based on the target text embedding vector is more consistent with the effect to be presented by the text prompt, thereby saving a lot of time and resources without any additional training and fine-tuning, and without disturbing the original order and structure of the text prompt, avoiding text loss.

[0096] In the embodiments of the present application, by dividing a plurality of sentence segments and assigning each sentence segment with its own attention weight, the sentence segment with higher attention weight will obtain more attention weight when weighting the initial text embedding vector, so that the importance of each sentence segment in the entire text prompt can be controlled, thereby accurately controlling the attention of the text prompt content in a complex text prompt environment.

[0097] As can be seen from the example, the scheme provided by the application obtains a text prompt word; wherein the text prompt word includes multiple sentence segments, and each sentence segment has a pre-allocated attention weight; encodes the text prompt word into an initial text embedding vector; performs weighted processing on each dimension of the initial text embedding vector according to the attention weight of each sentence segment to obtain a target text embedding vector; and generates visual data matched with the text prompt word based on the target text embedding vector. According to the pre-allocated attention weight of each sentence segment, the application directly performs weighted processing on each dimension of the initial text embedding vector of the text prompt word, so as to directly control the attention strength of the content of the text prompt word from the bottom layer, so that the generated visual data is more in line with the effect presented by the text prompt word, thereby saving a lot of time and resources without any additional training and fine-tuning, and without disturbing the original order and structure of the text prompt word, avoiding text loss, and the application can accurately allocate the corresponding attention size to the content of the text prompt word through partition weighting, thereby effectively dealing with the situation of complex text prompt words.

[0098] Figure 2 is another flowchart of the visual data generation method shown in the embodiments of the application.

[0099] Referring to Figure 2 , the visual data generation method of the application comprises:

[0100] S210, obtaining a text prompt word; wherein the text prompt word includes multiple sentence segments, and each sentence segment has a pre-allocated attention weight.

[0101] Figure 3 is an architecture diagram of the data processing platform shown in the embodiments of the application, as Figure 3 shown, the data processing platform obtains a text prompt word s, wherein the text prompt word s can include m sentence segments: s 1 ,s 2 ,...,s m , m is an integer and m≥1. Wherein each sentence segment has a pre-allocated attention weight: w 1 ,w 2 ,...,w m .

[0102] In an embodiment, obtaining a text prompt word can include:

[0103] obtaining input data of a user; and obtaining a text prompt word according to the input data of the user.

[0104] The type of the input data of the user includes a text type, a voice type, an image type, or a video type. In response to the type of the input data of the user being the text type, the data of the text type is directly taken as the text prompt word; in response to the type of the input data of the user being the voice type, the image type, or the video type, the text prompt word is obtained by performing content extraction on the data of the voice type, the data of the image type, or the data of the video type. Through the optional embodiment, the use threshold of the user is reduced, the use of the user is facilitated, and then the user can fully input the demand information, so that the visual data matched with the text prompt word is more in line with the demand of the user.

[0105] In an embodiment, obtaining the text prompt word according to the input data of the user can include:

[0106] obtaining an initial text prompt word according to the input data of the user, performing sentence segment division on the initial text prompt word, and determining an attention weight of each sentence segment to obtain the text prompt word including a plurality of sentence segments and the attention weight of each sentence segment.

[0107] The process of determining the attention weight of each sentence segment can include: obtaining the attention weight of each sentence segment according to the input of the user, or obtaining the attention weight of each sentence segment according to a sentence segment type of each sentence segment. Specifically, the sentence segment type of the sentence segment can be determined by the semantic or text content of the sentence segment, for example: if the semantic of the sentence segment indicates information of a subject object (such as a character image feature, a character personality feature, or a character action feature) or shot information (such as a level view angle, a low-angle view angle, a shot from near to far, or a composition feature), it indicates that the sentence segment type of the sentence segment is a high-weight sentence segment type; if the semantic of the sentence segment indicates background information (such as a sky color feature, a material feature of an edge ornament, etc.), it indicates that the sentence segment type of the sentence segment is a low-weight sentence segment type; or if the text content of the sentence segment matches a text of a preset high-weight sentence segment type, it is determined that the sentence segment type of the sentence segment is the high-weight sentence segment type; if the text content of the sentence segment matches a text of a preset low-weight sentence segment type, it is determined that the sentence segment type of the sentence segment is the low-weight sentence segment type, and so on. Through the above optional embodiment, the use threshold of the user is also reduced, and the attention weight that makes the generated visual data more accurate is quickly obtained based on the process.

[0108] In an example, obtaining the attention weight of each patch according to the user input can include: after the plurality of patches are divided, obtaining the attention weight assigned to each patch by the user. In this way, the attention weight of each patch can be determined by the user. For example, if the user wants to emphasize the content of a patch in the text prompt, the user can assign a higher attention weight to the patch; if the user wants to weaken the content of a patch in the text prompt, the user can assign a lower attention weight to the patch.

[0109] In another example, obtaining the attention weight of each patch according to the patch type of each patch can include: after the plurality of patches are divided, identifying whether each patch is a patch of a target patch type, and assigning a corresponding attention weight to the patch when it is identified that the patch is a patch of the target patch type. In this way, the attention weight of each patch can be automatically assigned by the data processing platform. The target patch type can include at least one of a high-weight patch type and a low-weight patch type. For example, when it is identified that the patch type of a patch is a high-weight patch type, the data processing platform can determine the patch as an emphasized patch and assign a higher attention weight to the emphasized patch; when it is identified that the patch type of a patch is a low-weight patch type, the data processing platform can determine the patch as a weakened patch and assign a lower attention weight to the weakened patch.

[0110] In addition, in addition to the emphasized patch and the weakened patch, other patches in the text prompt are ordinary patches, and the ordinary patch has a default attention weight, so the data processing platform does not need to assign a corresponding attention weight to the ordinary patch.

[0111] Therefore, the patches in the text prompt can include at least one of an emphasized patch, an ordinary patch, and a weakened patch, wherein the emphasized patch can refer to a patch that needs to be emphasized in the text prompt, the weakened patch can refer to a patch that needs to be weakened in the text prompt, and the ordinary patch can refer to a patch other than the emphasized patch and the weakened patch in the text prompt, wherein the attention weight of the emphasized patch > the attention weight of the ordinary patch > the attention weight of the weakened patch.

[0112] S220, performing tokenization on each patch in the text prompt in order to obtain a plurality of tokens.

[0113] The data processing platform can include a tokenizer, which can be used to split a patch into tokens for representation, such as Figure 3As shown, after obtaining the text prompt, the data processing platform can input the text prompt to a tokenizer to divide each sentence segment in the text prompt into tokens in sequence by the tokenizer, where each sentence segment corresponds to at least one token, and thus a plurality of tokens can be obtained. Specifically, for a certain sentence segment s i , i = 1, 2, …, m, the tokenizer can divide the sentence segment s i into a sequence of n i tokens: n i is an integer and n i ≥ 1.

[0114] By dividing each sentence segment into tokens in sequence, words and roots in natural language can be split into smaller language units, so that the subsequent text-to-image model can better understand and process the text content, and thus a token can refer to the smallest unit carrying semantic information of the text.

[0115] As an example, for the “dog” sentence segment, “small” is an adjective and “dog” is a noun, both of which are the smallest units carrying semantic information of the text, and thus the tokenizer can divide the “dog” sentence segment into two tokens, i.e., “small” and “dog”. As another example, for the “Apples” sentence segment, “App” is an abbreviation of Application (application) and “les” is an abbreviation of Linear Expenditure system (linear expenditure system model), both of which are the smallest units carrying semantic information of the text, and thus the tokenizer can divide the “Apples” sentence segment into two tokens, i.e., “App” and “les”. As yet another example, for the “tree” sentence segment, “tree” is the smallest unit carrying semantic information of the text, and thus the tokenizer can divide the “tree” sentence segment into one token, i.e., “tree”.

[0116] It should be noted that depending on the tokenizer, some tokens in the text prompt can be special tokens, which can include at least one of a start token, an end token, an unknown token, and a placeholder. The start token can refer to a sentence start identifier, for example <go>or <sos>; end-of-sentence token can refer to a sentence end identifier, e.g. <eos>; unknown tokens can refer to low frequency words or words not in the vocabulary, e.g. <unk>Placeholders can refer to identifiers used to dynamically insert values, parameters, or formats, such as <%v>. Since these special tokens do not represent any words in the text prompt, they do not participate in the final semantic presentation. Therefore, the attention weights of special tokens can be fixed to the default 1.0.

[0117] S230: Encode each word to obtain an initial word embedding vector for each word.

[0118] like Figure 3 As shown, the n corresponding to each sentence segment output by the word segmenter is obtained i After the word-unit sequence is obtained, the data processing platform can connect the word-unit sequences of all the segments end to end, and then input the entire sequence into the text encoder to encode the initial text embedding vector of the text prompt word. Specifically, the text encoder can encode each word-unit to obtain the initial word-unit embedding vector of each word-unit. In order to facilitate distinction, for a certain word-unit In this embodiment of the present application, the word The initial word embedding vector of After getting the initial word embedding vectors of all words After that, the embodiment of the present application can embed the initial word of each word into the vector By stacking them in sequence, we can get the initial text embedding vector of the text prompt word. Therefore, the initial text embedding vector of the text prompt word and the initial word embedding vector of each word are There is a position correspondence between them, that is, the initial text embedding vector can include the initial word unit embedding vector of each word unit, and each dimension of the initial text embedding vector corresponds to the initial word unit embedding vector of a word unit.

[0119] S240, performing weighted processing on the initial word-unit embedding vector of each word-unit according to the attention weight of each sentence segment to obtain the target word-unit embedding vector of each word-unit.

[0120] The data processing platform can directly embed the initial word element vector of each word element output by the text encoder according to the attention weight pre-assigned to each sentence segment. Perform weighted processing to obtain the weighted target word embedding vector for each word.

[0121] In order to facilitate the distinction, for a word In this embodiment of the present application, the word The weighted target word embedding vector is marked as The target word embedding vector after all word weights are obtained After that, the embodiment of the present application can embed the weighted target word of each word into the vector By stacking them in sequence, we can get the target text embedding vector of the text prompt word. Therefore, the target text embedding vector of the text prompt word and the target word embedding vector of each word are There is also a positional correspondence between them, that is, the target text embedding vector can include the target word embedding vector of each word, and each dimension of the target text embedding vector corresponds to the target word embedding vector of a word.

[0122] In one embodiment, weighting the initial word-unit embedding vector of each word-unit according to the attention weight of each sentence segment to obtain the target word-unit embedding vector of each word-unit may include:

[0123] The target word embedding vector of each word is calculated by multiplying the value of the initial word embedding vector of each word with the attention weight of the corresponding sentence segment.

[0124] The size of the vector modulus is proportional to the level of attention paid to the corresponding word-unit; that is, the size of the vector modulus is related to the contribution of the vector modulus to the visual data generation process. For example, if the initial text prompt is "a brown puppy," the vector moduli of the two words "small" and "dog" may be relatively small. Therefore, when subsequently input into the text-graph model, the text-graph model pays less attention to both "small" and "dog," causing it to easily overlook "small" and "dog." As a result, the text-graph model does not fully understand the concepts of "small" and "dog." As a result, the visual data generated by the text-graph model lacks a clear representation of "puppy," for example, resembling a puppy, a kitten, a wolf, or some other small animal.

[0125] To this end, the data processing platform can embed the initial word element vector of each word element output by the text encoder according to the attention weight pre-assigned to each sentence segment. Perform weighted processing to adjust the size of the vector module of each word, thereby adjusting the attention of the text graph model to each word. Specifically, the data processing platform can embed the initial word of each word into the vector The value of and the attention weight w of the corresponding sentence i Multiply them together to get the target word embedding vector for each word. Figure 3 As shown, for i = 1, 2, ..., m, j = 1, 2, ..., n i , assignment Weighted vector That is the word The final target word embedding vector.

[0126] In an example, assuming that the attention weight of the "dog" sentence segment is 1.2, the values of the initial wordpiece embedding vectors of the two wordpieces "small" and "dog" can be multiplied by 1.2 respectively to obtain the target wordpiece embedding vectors of the two wordpieces "small" and "dog", so as to enhance the attention of the text-to-image model to "small" and "dog", so that the text-to-image model pays more attention to "small" and "dog", and the text-to-image model better understands the concepts of "small" and "dog", and thus the "dog" in the visual data generated by the text-to-image model is more obvious.

[0127] S250, obtaining the preset noise.

[0128] The training of the text-to-image model can include two parts: a forward process and a reverse process. The forward process is to generate random noise x T by continuously adding Gaussian noise ε to the original visual data x0, and the purpose of the forward process is to change the original visual data x0 into random noise x T , and the purpose of the reverse process is to restore the original visual data x0 from the random noise x T , that is, the reverse process is to predict the noise ε θ from the random noise x θ , and then calculate the loss value of the predicted noise ε T and the Gaussian noise ε, so as to update the text-to-image model by using the loss value, so as to gradually remove the predicted noise ε θ from the random noise x T , and finally restore the original visual data x0. When the loss value is less than a preset threshold, the trained text-to-image model is obtained.

[0129] Therefore, after obtaining the weighted target text embedding vector, the embodiment of the present application can randomly generate a noise matrix x T as the preset noise.

[0130] S260, performing feature fusion on the target wordpiece embedding vector of each wordpiece and the preset noise to obtain the wordpiece fusion feature of each wordpiece.

[0131] The embodiment of the present application can input the target wordpiece embedding vector of each wordpiece and the preset noise x T to the trained text-to-image model, so that the target wordpiece embedding vector of each wordpiece and the preset noise x T can be fused by the text-to-image model to obtain the wordpiece fusion feature of each wordpiece.

[0132] S270, generating visual data matched with the text prompt word based on the wordpiece fusion feature of each wordpiece.

[0133] Through the wordpiece fusion feature of each wordpiece, the text-to-image model can be guided to predict the noise ε in the direction of the text prompt word θ Then the text-to-image model will preset the noise x T Gradually remove the predicted noise ε θ Since the text-to-image model pays more attention to the wordpiece fusion features with higher attention weights and less attention to the wordpiece fusion features with lower attention weights, the visual data generated by the text-to-image model based on the wordpiece fusion features of each wordpiece can better conform to the effect to be presented by the text prompt word.

[0134] Among them, the visual data can include at least one of static image data and dynamic video data, and the static image data is used for the following example illustration in the embodiments of the present application:

[0135] In an example, assuming that the initial text prompt word obtained according to the input data of the user is "a brown small dog", the initial text prompt word can be divided into three sentence segments: "a", "brown", and "small dog". Since the text-to-image model may have weaker understanding ability for the "small dog" sentence segment, i.e., the text-to-image model has lower attention for the "small dog" sentence segment, the text-to-image model generates an "a brown small dog" image, and the "small dog" in the image is not obvious enough, as shown in Figure 4A , Figure 4A is the "a brown small dog" image generated by the text-to-image model without increasing the attention weight of the "small dog" sentence segment. As can be seen, Figure 4A , the "small dog" in the image is not obvious enough, for example, it looks like a small dog, a small cat, a small wolf, or other small animals. In order to make the "small dog" in the image more obvious, the "small dog" sentence segment can be used as an emphasized sentence segment, so that a higher attention weight is assigned to the "small dog" sentence segment, for example, the attention weight of the "small dog" sentence segment can be set to a higher value, for example, 1.2, from the default 1.0. Since the "a" and "brown" in the image are obvious enough, the "a" and "brown" sentence segments are used as normal sentence segments, so that the default attention weight is assigned to the "a" and "brown" sentence segments, for example, the attention weight of the "a" and "brown" sentence segments can be maintained at the default 1.0, so as to obtain a text prompt word including multiple sentence segments and the attention weight of each sentence segment. Through this operation, the attention of the text-to-image model to the "small dog" sentence segment can be enhanced, so as to increase the attention of the text-to-image model to the "small dog" sentence segment, so that the image data generated by the text-to-image model can better conform to the effect to be presented by the user, as shown in Figure 4B , Figure 4B is the "a brown small dog" image generated by the text-to-image model with increased attention weight of the "small dog" sentence segment. As can be seen, Figure 4B , the "small dog" in the image is more obvious.

[0136] In another example, assuming that the initial text prompt obtained according to the input data of the user is "Apples in the tree", the initial text prompt can be divided into four sentence segments: "Apples", "in", "the", and "tree". Since the text-to-image model can have a stronger understanding of the "in" sentence segment, i.e., the text-to-image model has a higher attention on the "in" sentence segment, the text-to-image model generates an "Apples in the tree" image in which all the apples are hanging on the tree, which can have a situation that some apples are ripe, but some apples are not ripe, as shown in Figure 5A . Figure 5A is an "Apples in the tree" image generated by the text-to-image model without reducing the attention weight of the "in" sentence segment. When the user wants to highlight some apples falling under the tree in the image, the apples falling under the tree indicate that the apples are ripe, so as to present a bountiful harvest effect. In order to achieve this effect, the "in" sentence segment can be a weakened sentence segment, and other sentence segments can be normal sentence segments, so a lower attention weight can be assigned to the "in" sentence segment, for example, the attention weight of the "in" sentence segment can be set from the default 1.0 to a lower value, for example, 0.8, and the "Apples", "the", and "tree" sentence segments can be assigned the default attention weight, for example, the attention weights of the "Apples", "the", and "tree" sentence segments can be maintained at the default 1.0, thereby obtaining a text prompt including multiple sentence segments and the attention weight of each sentence segment. Through this operation, the attention of the text-to-image model on the "in" sentence segment can be weakened, so as to reduce the focus of the text-to-image model on the "in" sentence segment, so that the image data generated by the text-to-image model is more in line with the effect that the user wants to present, as shown in Figure 5B . Figure 5B is an "Apples in the tree" image generated by the text-to-image model with reducing the attention weight of the "in" sentence segment. As can be seen, Figure 5B some apples fall under the tree, which can present a bountiful harvest effect.

[0137] In yet another example, assuming that the initial text prompt obtained according to the input data of the user is "There are red lanterns hanging in the bamboo forest in the park, pink plum blossoms blooming beside it, a pavilion in the distance, sunlight shining on the winding park path, presenting a warm feeling", it can be seen that the initial text prompt is very complex, and therefore, the embodiment of the present application can divide the initial text prompt into a plurality of sentence segments: "in the park", "bamboo forest", "hanging", "red", "lanterns", "beside", "blossoming", "pink", "plum blossoms", "in the distance", "a", "pavilion", "sunlight", "shining on", "winding", "park", "path", "presenting", "warm", "feeling", and can assign each sentence segment a respective attention weight, thereby obtaining a text prompt including a plurality of sentence segments and the attention weight of each sentence segment, and the sentence segment with a higher attention weight will be able to obtain more attention weight when weighting the initial text embedding vector, so that the user can control the importance of each sentence segment in the entire text prompt, thereby accurately controlling the attention of the text prompt content in a complex text prompt environment, for example, the user wants to highlight "bamboo forest", "lanterns", "plum blossoms", "path" and other elements in the image, and therefore, the "bamboo forest", "lanterns", "plum blossoms", "path" sentence segments can be assigned a higher attention weight, and the other sentence segments can maintain the default attention weight, so that the image data generated by the text-to-image model is more in line with the effect that the user wants to present, as shown in Figure 6 , Figure 6 is the "There are red lanterns hanging in the bamboo forest in the park, pink plum blossoms blooming beside it, a pavilion in the distance, sunlight shining on the winding park path, presenting a warm feeling" image generated by the text-to-image model, it can be seen that Figure 6 "bamboo forest", "lanterns", "plum blossoms", "path" and other elements are highlighted, and therefore, the embodiment of the present application can provide accurate and stable allocation of attention weight when the text prompt is very complex by weighting the text prompt.

[0138] As can be seen from the example, the scheme provided by the application can increase the attention weight of the sentence segment that needs to be emphasized, enhance the attention of the text-to-image model to the sentence segment from the bottom, increase the attention of the text-to-image model to the sentence segment, and reduce the attention weight of the sentence segment that needs to be weakened, weaken the attention of the text-to-image model to the sentence segment from the bottom, and reduce the attention of the text-to-image model to the sentence segment, so that the visual data generated by the text-to-image model is more in line with the effect presented by the text prompt, thereby saving a lot of time and resources without any additional training and fine-tuning, and avoiding disturbing the original order and structure of the text prompt. Further, by dividing the initial text prompt into a plurality of sentence segments and assigning each sentence segment a respective attention weight, the text prompt including a plurality of sentence segments and the attention weight of each sentence segment is obtained. The sentence segment with a higher attention weight will obtain more attention weight when the initial text embedding vector is weighted and divided, so that the user can control the importance of each sentence segment in the entire text prompt, thereby accurately controlling the attention of the text prompt content in a complex text prompt environment.

[0139] Corresponding to the foregoing application function implementation method embodiments, the application further provides a visual data generation apparatus, an electronic device, a computer readable storage medium, and corresponding embodiments.

[0140] Figure 7 FIG. 1 is a structural schematic diagram of a visual data generation apparatus according to an embodiment of the application.

[0141] Referring to Figure 7 The visual data generation apparatus provided by the application can include:

[0142] The text prompt acquisition module 710 is configured to acquire a text prompt; wherein the text prompt includes a plurality of sentence segments, and each sentence segment has a pre-assigned attention weight;

[0143] The encoding processing module 720 is configured to encode the text prompt into an initial text embedding vector;

[0144] The weighting processing module 730 is configured to weight each dimension of the initial text embedding vector according to the attention weight of each sentence segment, to obtain a target text embedding vector;

[0145] The visual data generation module 740 is configured to generate visual data matching the text prompt based on the target text embedding vector.

[0146] In an embodiment, before the text prompt is encoded into the initial text embedding vector, the apparatus can further include:

[0147] The word division module is configured to divide each sentence segment in the text prompt word in sequence to obtain a plurality of words.

[0148] The initial text embedding vector includes an initial word embedding vector of each word.

[0149] The encoding processing submodule is configured to encode each word to obtain an initial word embedding vector of each word.

[0150] In an embodiment, the target text embedding vector includes a target word embedding vector of each word; and the weighting processing module 730 can include:

[0151] The weighting processing submodule is configured to weight the initial word embedding vector of each word according to the attention weight of the corresponding sentence segment to obtain a target word embedding vector of each word.

[0152] In an embodiment, the weighting processing submodule can include:

[0153] The weighting processing unit is configured to calculate the target word embedding vector of each word according to the value of the initial word embedding vector of each word and the attention weight of the corresponding sentence segment.

[0154] In an embodiment, the visual data generation module 740 can include:

[0155] The preset noise acquisition submodule is configured to acquire a preset noise.

[0156] The feature fusion submodule is configured to fuse the target word embedding vector of each word with the preset noise to obtain a word fusion feature of each word.

[0157] The visual data generation submodule is configured to generate visual data matched with the text prompt word based on the word fusion feature of each word.

[0158] In an embodiment, the text prompt word acquisition module 710 can include:

[0159] The input data acquisition submodule is configured to acquire input data of a user.

[0160] The input data processing submodule is configured to obtain a text prompt word according to the input data of the user.

[0161] In an embodiment, the input data processing submodule can include:

[0162] The input data processing unit is configured to obtain an initial text prompt word according to the input data of the user.

[0163] The initial text prompt word processing unit is configured to perform sentence segmentation on the initial text prompt word, and determine attention weights of each sentence segment, to obtain a text prompt word including a plurality of sentence segments and the attention weights of each sentence segment.

[0164] As can be seen from the example, the scheme provided by the present application acquires a text prompt word; wherein the text prompt word includes a plurality of sentence segments, and each sentence segment has a pre-allocated attention weight; encodes the text prompt word into an initial text embedding vector; performs weighted processing on each dimension of the initial text embedding vector according to the attention weights of each sentence segment, to obtain a target text embedding vector; and generates visual data matched with the text prompt word based on the target text embedding vector. The present application directly performs weighted processing on each dimension of the initial text embedding vector of the text prompt word according to the pre-allocated attention weight of each sentence segment, so as to directly control the attention strength of the content of the text prompt word from the bottom layer, so that the generated visual data is more in line with the effect to be presented by the text prompt word, thereby saving a large amount of time and resources without any additional training and fine-tuning, and without disturbing the original order and structure of the text prompt word, avoiding text loss, and the present application can accurately allocate the corresponding attention size to the content of the text prompt word through partition weighting, thereby effectively dealing with the situation of complex text prompt words.

[0165] As to the apparatus in the above-mentioned embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0166] Figure 8 FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application.

[0167] Referring to Figure 8 The electronic device 800 includes a memory 810 and a processor 820.

[0168] The processor 820 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic components, discrete hardware components, or the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0169] The memory 810 can include various types of storage units such as a system memory, a read-only memory (ROM), and a permanent storage device. Among them, the ROM can store static data or instructions required by the processor 820 or other modules of the computer. The permanent storage device can be a rewritable storage device. The permanent storage device can be a non-volatile storage device that does not lose stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device employs a mass storage device (e.g., a magnetic or optical disk, a flash memory) as a permanent storage device. In some other embodiments, the permanent storage device can be a removable storage device (e.g., a floppy disk, an optical drive). The system memory can be a readable and writable storage device or a volatile readable and writable storage device such as a dynamic random access memory. The system memory can store some or all of the instructions and data required by the processor during runtime. In addition, the memory 810 can include a combination of any computer readable storage media, including various types of semiconductor storage chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic disks and / or optical disks. In some embodiments, the memory 810 can include a readable and / or writable removable storage device such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., an SD card, a min SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer readable storage media does not include a carrier wave and an instantaneous electronic signal transmitted by wireless or wired transmission.

[0170] The executable code stored on the memory 810 can cause the processor 820 to perform part or all of the above-mentioned methods when the executable code is processed by the processor 820.

[0171] In addition, the method according to the present application can also be implemented as a computer program or a computer program product including computer program code instructions for performing part or all of the steps of the above-mentioned methods of the present application.

[0172] Alternatively, the present application can also be implemented as a computer readable storage medium (or non-transitory machine readable storage medium or machine readable storage medium) having executable code (or computer program or computer instruction code) stored thereon, which when executed by a processor of an electronic device, causes the processor to perform part or all of the steps of the above-mentioned methods according to the present application.

[0173] Having described various embodiments of the application, it is to be understood that the above description is meant to be illustrative only, and that many modifications and variations of the embodiments described herein are possible. It is therefore to be understood that within the scope of the appended claims, and their equivalents, many alternatives to the embodiments described herein are possible. The selection of terms to be used in the description is not intended to limit the scope of the embodiments described herein, but rather to best explain the principles of the embodiments, practical application, or improvement over the technology in the art, or to enable others skilled in the art to understand the embodiments disclosed herein.< / unk> < / eos> < / sos> < / go>

Claims

1. A method of generating visual data, characterized by, The method comprises the following steps: obtaining a text prompt; wherein the text prompt comprises a plurality of sentence segments, and each sentence segment has a pre-assigned attention weight; performing token division on each sentence segment in the text prompt in sequence to obtain a plurality of tokens; performing encoding processing on each token to obtain an initial token embedding vector of the token; and the initial token embedding vectors of the tokens constitute an initial text embedding vector of the text prompt; performing weighting processing on each dimension of the initial text embedding vector according to the attention weights of the sentence segments to obtain a target text embedding vector; generating visual data matched with the text prompt based on the target text embedding vector; the attention weights of the sentence segments are obtained by the following steps: performing sentence segmentation on an initial text prompt to determine each sentence segment; determining whether each sentence segment is a sentence segment of a target sentence segment type according to the semantics or text content of the sentence segment, and assigning a corresponding attention weight to the sentence segment when it is determined that the sentence segment is a sentence segment of the target sentence segment type; wherein the target sentence segment type comprises at least one of a high-weight sentence segment type and a low-weight sentence segment type; determining the sentence segment type of each sentence segment according to the semantics of the sentence segment, comprising: if the semantics of the sentence segment indicates subject object information or shot information, determining that the sentence segment type of the sentence segment is a high-weight sentence segment type; and if the semantics of the sentence segment indicates background information, determining that the sentence segment type of the sentence segment is a low-weight sentence segment type; or, determining the sentence segment type of each sentence segment according to the text content of the sentence segment, comprising: if the text content of the sentence segment matches the text of a preset high-weight sentence segment type, determining that the sentence segment type of the sentence segment is a high-weight sentence segment type; and if the text content of the sentence segment matches the text of a preset low-weight sentence segment type, determining that the sentence segment type of the sentence segment is a low-weight sentence segment type.

2. The method of claim 1, wherein, the target text embedding vector comprises target token embedding vectors of the tokens; and the weighting processing on each dimension of the initial text embedding vector according to the attention weights of the sentence segments to obtain a target text embedding vector comprises: performing weighting processing on the initial token embedding vectors of the tokens according to the attention weights of the sentence segments to obtain target token embedding vectors of the tokens.

3. The method of claim 2, wherein, the weighting processing on the initial token embedding vectors of the tokens according to the attention weights of the sentence segments to obtain target token embedding vectors of the tokens comprises: calculating the target token embedding vectors of the tokens according to the values of the initial token embedding vectors of the tokens and the attention weights of the corresponding sentence segments.

4. The method of claim 2, wherein, the generating of visual data matched with the text prompt based on the target text embedding vector comprises: obtaining a preset noise; performing feature fusion on the target token embedding vectors of the tokens and the preset noise to obtain token fusion features of the tokens; generating visual data matched with the text prompt based on the token fusion features of the tokens.

5. The method according to any one of claims 1 to 4, characterized in that, The obtaining the text prompt word comprises: obtaining input data of a user; obtaining a text prompt word according to the input data of the user.

6. The method of claim 5, wherein, The obtaining the text prompt word according to the input data of the user comprises: obtaining an initial text prompt word according to the input data of the user; performing sentence segmentation on the initial text prompt word and determining attention weights of each sentence segment to obtain a text prompt word comprising a plurality of sentence segments and the attention weights of each sentence segment.

7. A visual data generating apparatus characterized by comprising: Comprise: a text prompt word obtaining module configured to obtain a text prompt word; wherein the text prompt word comprises a plurality of sentence segments, and each sentence segment has a pre-allocated attention weight; a word unit dividing module configured to perform word unit division on each sentence segment in the text prompt word in sequence to obtain a plurality of word units; an encoding processing module configured to perform encoding processing on each word unit to obtain an initial word unit embedding vector of the word unit; the initial word unit embedding vectors of the word units constitute an initial text embedding vector of the text prompt word; a weighting processing module configured to perform weighting processing on each dimension of the initial text embedding vector according to the attention weights of each sentence segment to obtain a target text embedding vector; a visual data generating module configured to generate visual data matching the text prompt word based on the target text embedding vector; The attention weights of each sentence segment are obtained by: performing sentence segmentation on an initial text prompt word to determine each sentence segment; according to the semantics or text content of each sentence segment, determining whether the sentence segment is a sentence segment of a target sentence segment type, and when a sentence segment of the target sentence segment type is identified, assigning the sentence segment a corresponding attention weight; wherein the target sentence segment type comprises at least one of a high-weight sentence segment type and a low-weight sentence segment type; According to the semantics of each sentence segment, the sentence segment type of each sentence segment is determined, comprising: if the semantics of the sentence segment indicates subject object information or shot information, the sentence segment type of the sentence segment is determined as a high-weight sentence segment type; if the semantics of the sentence segment indicates background information, the sentence segment type of the sentence segment is determined as a low-weight sentence segment type; Alternatively, according to the text content of each sentence segment, the sentence segment type of each sentence segment is determined, comprising: if the text content of the sentence segment contains a preset high-weight sentence segment type text, the sentence segment type of the sentence segment is determined as a high-weight sentence segment type; if the text content of the sentence segment contains a preset low-weight sentence segment type text, the sentence segment type of the sentence segment is determined as a low-weight sentence segment type.

8. An electronic device, comprising: Comprise: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method of any one of claims 1-6.

9. A computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method of any one of claims 1-6.

Images