Image-text generation model training method, visual text image generation method and device

Abstract

The invention provides an image-text generation model training method, a visual text image generation method and a visual text image generation device, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes such as AIGC content generation based on artificial intelligence. The image-text generation model training method comprises the following steps: inputting a sample image and an instruction into a to-be-trained image-text generation model, and generating an attention map of prediction noise, a prediction image and a noise-added image by the to-be-trained image-text generation model; generating a mask image according to the sample image, and calculating a total loss function by using attention maps of the mask image, the prediction image, the prediction noise and the noise-added image; and training the image-text generation model to be trained based on the total loss function.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, especially to technical fields such as computer vision, deep learning, and large models, and can be applied to scenarios such as AIGC content generation based on artificial intelligence. Background Art

[0002] Visual text images, also known as text images, are images containing text content and have a wide range of application scenarios. For example, users have a wide range of needs for drawing books, road signs, posters, advertisements, and other scenes. The production of visual text images requires that the required text content be correctly spelled on the basis of a reasonable drawing of the image background, and that the font, color, and other attributes of the text be kept in harmony with the image background. Summary of the invention

[0003] The present invention provides a method for training a picture-text generation model, a method, a device, a equipment and a storage medium for generating a visual text image.

[0004] According to one aspect of the present disclosure, a method for training a graph-text generation model is provided, for training a graph-text generation model, comprising:

[0005] Inputting the sample image and the instruction into the image-text generation model to be trained, and generating the attention map of the predicted noise, the predicted image and the noisy image by the image-text generation model to be trained;

[0006] Generate a mask image according to the sample image, and calculate a total loss function using the mask image, the predicted image, the predicted noise, and the attention map of the noisy image;

[0007] Based on the total loss function, the image-text generation model to be trained is trained.

[0008] According to another aspect of the present disclosure, a method for generating a visual text image is provided, comprising:

[0009] The instruction is input into a picture-text generation model, and the picture-text generation model generates a visual text image based on the instruction; wherein the picture-text generation model is trained using the training method provided in the present disclosure.

[0010] According to another aspect of the present disclosure, a device for training a graph-text generation model is provided, for training a graph-text generation model, comprising:

[0011] A first input module, used for inputting a sample image and an instruction into a to-be-trained image-text generation model, and generating an attention map of predicted noise, predicted image and noisy image by the to-be-trained image-text generation model;

[0012] A loss function calculation module, used to generate a mask image according to the sample image, and calculate a total loss function using the mask image, the predicted image, the predicted noise, and the attention map of the noisy image;

[0013] The training module is used to train the image-text generation model to be trained based on the total loss function.

[0014] According to another aspect of the present disclosure, there is provided a visual text image generating device, comprising:

[0015] The second input module is used to input the instruction into the image-text generation model, and the image-text generation model generates a visual text image based on the instruction; wherein the image-text generation model is trained using the training device provided by the present disclosure.

[0016] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0020] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0021] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0022] The present disclosure proposes a method for training a picture-text generation model, which can train a picture-text generation model; and proposes a method for generating a visual text image using the picture-text generation model. When using the model to generate a visual text image, only a prompt needs to be input without specifying the specific position of the text in the visual text image, thereby eliminating the limitation of needing to accurately specify the text position, while also enhancing the image generation quality.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0025] Figure 1 is a schematic diagram of an application scenario according to an embodiment of the present disclosure;

[0026] Figure 2 is a flowchart of an implementation method of a graph-text generation model training method according to an embodiment of the present disclosure;

[0027] Figure 3 It is a schematic diagram of the architecture and training method of the image-text generation model according to an embodiment of the present disclosure;

[0028] Figure 4 is a schematic diagram of a method for generating a visual text image according to an embodiment of the present disclosure;

[0029] Figure 5 is a structural schematic diagram of a graph-text generation model training device 500 according to an embodiment of the present disclosure;

[0030] Figure 6 is a structural schematic diagram of a graph-text generation model training device 600 according to an embodiment of the present disclosure;

[0031] Figure 7 is a structural schematic diagram of a visual text image generating device 700 according to an embodiment of the present disclosure;

[0032] Figure 8 A schematic block diagram of an example electronic device 800 is shown that may be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0034] The “and / or” in the embodiments of the present disclosure indicates that three relationships may exist. For example, A and / or B may indicate the following three situations: A exists alone, A and B exist at the same time, and B exists alone. The term “at least one” herein indicates any combination of at least two of any one or more of a plurality of types. For example, at least one of A, B, and C may indicate any one or more elements selected from the set consisting of A, B, and C. The terms “first” and “second” herein refer to and distinguish between multiple similar technical terms, and do not mean to limit the order or to limit the meaning to only two. For example, the first feature and the second feature refer to two types / two features. The first feature may be one or more, and the second feature may also be one or more.

[0035] The disclosed embodiment proposes a method for training a picture-text generation model, which can be used to train a picture-text generation model. By training the model, visual text images can be generated.

[0036] Visual text images, also known as text images, are images containing text content and have a wide range of application scenarios. For example, users have a wide range of needs for drawing books, road signs, posters, advertisements, and other scenes. The production of visual text images requires that the required text content be correctly spelled on the basis of a reasonable drawing of the image background, and that the font, color, and other attributes of the text be kept in harmony with the image background.

[0037] At present, the main methods of producing visual text images include computer font production and artificial intelligence generated content (AIGC) model production. When using the AIGC model to produce visual text images, the user needs to specify the location of the text, and then the model generates the visual text image according to the conditions specified by the user. This method has at least the following disadvantages:

[0038] First, since the text position is pre-specified by the user, it is difficult for the user to meet the requirements for image diversity and creativity;

[0039] Second, since the image layout is pre-specified, it is easy to cause a sense of separation between the text and the image background in the image generated by the model, affecting the aesthetics of the image.

[0040] Based on this, the disclosed embodiment proposes a method for training a text-image generation model, which can be used to train a text-image generation model. The text-image generation model does not require the user to specify the position of the text in the image. The user only needs to input an instruction (prompt) to the trained text-image generation model, and the instruction includes image description information and text content that needs to be displayed in the image. The text-image generation model can automatically generate a visual text image based on the instruction. Since the user does not need to specify the position of the text in the image, the model can reasonably place the text according to the image background, improve the diversity and creativity of the image, and it is not easy to be separated between the text and the image background, so it can improve the aesthetics of the image.

[0041] Figure 1 is a schematic diagram of an application scenario according to an embodiment of the present disclosure, such as Figure 1 As shown, the schematic diagram of the application scenario of the embodiment of the present disclosure may include but is not limited to a model training device 110 and an image generating device 120, and the model training device 110 and the image generating device 120 may communicate with each other through any type of wired or wireless network. Specifically, the model training device 110 can be used to train a graphic generation model, and send the trained graphic generation model to the image generating device 120; the image generating device 120 can load and install the graphic generation model, and after receiving the instruction (prompt) input by the user, input the instruction into the graphic generation model, and generate a visual text image by the graphic generation model. Among them, the instruction may include text instructions in multiple languages, and the text instruction contains image description information and text content that needs to be displayed in the image. For example, the instruction includes "a cute little bear holding a sign with the red words "Happy Everyday" on the sign."

[0042] The model training device 110 involved in the embodiment of the present disclosure may include a server or a server group. The image generation device 120 involved in the embodiment of the present disclosure includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, game consoles, e-book readers, multimedia playback devices, wearable devices and other electronic devices. In addition, the embodiment of the present disclosure does not specifically limit the number of image generation devices 120. For example, the application scenario diagram of the embodiment of the present disclosure may include one or more behavior image generation devices 120.

[0043] Figure 2 It is a flowchart of an implementation method of a graph-text generation model training method according to an embodiment of the present disclosure, including:

[0044] S210, inputting the sample image and the instruction into the image-text generation model to be trained, and generating an attention map of the predicted noise, the predicted image and the noisy image by the image-text generation model to be trained;

[0045] S220, generating a mask image according to the sample image, and calculating a total loss function using the mask image, the predicted image, the predicted noise, and the attention map of the noisy image;

[0046] S230: Based on the total loss function, the image-text generation model to be trained is trained.

[0047] It can be seen from the above training method that when training the image-text generation model, sample images and prompts are used to enable the model to generate attention maps of predicted noise, predicted images, and noisy images, and the total loss function is calculated based on the generated content and sample images, so as to adjust the model parameters according to the total loss function. During the training process, there is no need to enter the specified text position. Therefore, when using the trained image-text generation model to generate visual text images, there is no need for the user to enter the text position, thereby eliminating the limitation of the need to accurately specify the text position. At the same time, the model can set appropriate text according to the image situation, thereby enhancing the image generation quality.

[0048] In some embodiments, the image-text generation model uses a text-conditioned latent diffusion model (LDM), which includes an encoder, a decoder, and a UNet network (U-shaped network), and the text-conditioned LDM model has the creative generation capability of generating images from text. After the user inputs the instruction (prompt) through encoding, it is input into the text-conditioned LDM model as a condition, and the LDM model obtains the final generated image through the diffusion process.

[0049] During the model training process, the image is subjected to "data diffusion processing", that is, Gaussian noise is gradually added to the image; through a series of continuous time steps (time step), Gaussian noise of different intensities is added to the image obtained in the previous step at each time step. As the time step increases, the noise intensity gradually increases, and the details of the image are gradually masked by the noise. In the end, the image is almost completely covered by random noise, but it still retains extremely weak traces of the real image. When the noisy image is input into the UNet network, a predicted noise map will be generated, and then the predicted image can be obtained by subtracting the predicted noise map from the noisy image.

[0050] Figure 3 It is a schematic diagram of the architecture and training method of the image-text generation model according to an embodiment of the present disclosure.

[0051] like Figure 3As shown, the main part of the image-text generation model trained by the embodiment of the present disclosure is a diffusion model. In some examples, the diffusion model is a text-conditioned LDM, which includes an encoder 310 (such as VAEε in FIG. 3, representing a VAE encoder), a decoder 320 (such as Figure 3 VAE D in , representing a VAE decoder), a UNet network 330, and a text encoder 340. The diffusion model can generate visual text images based on the instructions (prompt) input by the user; the user does not need to enter the specific location of the visual text (or text), which is an important difference compared to existing solutions.

[0052] like Figure 3 As shown, in order to process the prompt input by the user, the model architecture involved in the embodiment of the present disclosure also includes a mixed granularity input module, which can generate a first vector representation of the instruction based on the user's instruction (prompt), and the first vector representation carries the glyph information of the visual text contained in the instruction (prompt). For example, the prompt input by the user is:

[0053] “Cute Teddy bear holding a sign with text”Happy Days” in red”

[0054] The "Happy Days" in quotation marks in the prompt is the visual text. The first vector representation of the prompt carries the semantic information of the entire instruction and the glyph information of the visual text.

[0055] In some embodiments, the image-text generation model to be trained generates an attention map of predicted noise, predicted image, and noisy image, comprising the following steps:

[0056] (1) generating a first vector representation of a prompt, wherein the first vector representation carries glyph information of a visual text contained in the prompt;

[0057] (2) For sample images (such as Figure 3 Add noise to x0) in order to obtain a noisy image;

[0058] (3) Generate prediction noise (such as Figure 3 ε t ), predicted image (such as Figure 3 Attention map of x'0) and noisy image.

[0059] The embodiment of the present disclosure does not limit the execution order of the above steps (1) and (2). The two steps can be executed one after the other (either step can be executed first) or simultaneously.

[0060] In the above process, the image-text generation model to be trained does not use the text position when performing calculations, but only uses the prompt input by the user and the sample image for calculations; the image-text generation model trained in this way can generate visual text images using only the prompt input by the user, eliminating the restrictions on the text position, thereby improving the capabilities of the image-text generation model.

[0061] The following combination Figure 3 , the above three steps are described in detail:

[0062] First, regarding step (1):

[0063] In some embodiments, step (1) comprises:

[0064] Extract the visual text from the prompt, and perform glyph rendering on the visual text to obtain a glyph map corresponding to the visual text (such as Figure 3 g in k ); performing OCR encoding on the glyph image to obtain a vector representation of the glyph image;

[0065] Determine the initial vector representation of the instruction (prompt);

[0066] The vector representation of the glyph image is used to replace the relevant content in the initial vector representation to obtain a first vector representation; wherein the relevant content includes the vector representation of the visual text corresponding to the glyph image.

[0067] For example, the prompt is "Cute Teddy bear holding a sign with text"Happy Days"inred", the visual text "Happy Days" is extracted from the prompt, and the glyph rendering module (not shown in the figure) performs glyph rendering on the visual text to obtain the glyph graph of the visual text (such as Figure 3 g in k). In one example, the visual text that needs to be displayed in the visual text image is extracted from the prompt by dividing it into two visual texts, including "Happy" and "Days", and the two visual texts are rendered with glyphs respectively to obtain two glyph graphs, namely the glyph graph of "Happy" and the glyph graph of "Days". The glyph graph of the visual text can be considered as an image, which contains the shape of the corresponding visual text; the OCR module 350 is then used to perform OCR encoding on the glyph graph to obtain a vector representation of the glyph graph, which carries the glyph information of the visual text.

[0068] On the other hand, prompt is input into the word segmentation module 360, which segments prompt and generates vector representations of each word segmentation; the vector representations of all word segments of prompt constitute the initial vector representation of the prompt. For example, if the prompt is "Cute Teddy bear holding a sign with text"Happy Days" in red", the initial vector representation of the prompt includes the vector representations of all word segments of the prompt.

[0069] Afterwards, the vector representation of the glyph graph is used to replace the relevant content in the initial vector representation, that is, the vector representation of the visual text corresponding to the glyph graph is replaced. Continuing with the above example, the above prompt has two glyph graphs, namely the glyph graph of "Happy" and the glyph graph of "Days"; the vector representation of "Happy" in the initial vector representation is replaced with the vector representation of the glyph graph of "Happy", and the vector representation of "Days" in the initial vector representation is replaced with the vector representation of the glyph graph of "Days", so as to obtain the first vector representation of the prompt, which carries the semantic features of the prompt and contains the glyph information of the glyph graph of "Happy" and the glyph graph of "Days".

[0070] In some implementations, extracting visual text and performing glyph rendering on the visual text includes:

[0071] According to the preset disassembly granularity, N visual texts are extracted from the instruction (prompt), and the glyphs of each visual text are rendered respectively to obtain N glyph graphs; wherein,

[0072] The decomposition granularity is a single character, and the decomposition granularity corresponds to the language type of the visual text.

[0073] For example, the above example is an English instruction, the content in the quotation marks of the instruction is the visual text, 2 visual texts are extracted from the quotation marks of the instruction, and the 2 visual texts are rendered respectively to obtain 2 glyph images; that is, N=2 in this example.

[0074] In the embodiment of the present disclosure, there are multiple disassembly granularities, and different disassembly granularities correspond to different language types.

[0075] If the prompt contains multiple characters, the decomposition granularity is related to the language type in the quotation marks in the prompt; if the language in the quotation marks is Chinese, the decomposition granularity is a single Chinese character; if the language in the quotation marks is English, the decomposition granularity is a single word.

[0076] The above-mentioned disassembly granularity is only an example. The embodiment of the present disclosure can also set the corresponding disassembly granularity according to the characteristics of different languages. By designing and adopting the corresponding disassembly granularity for various languages, it is possible to extract the visual text with a length suitable for text rendering of the visual text, and to facilitate the use of the glyph image obtained after text rendering to replace the vector representation of the visual text, thereby improving the accuracy of the first vector representation of the generated prompt, and then improving the model effect.

[0077] Second, regarding step (2):

[0078] like Figure 3 As shown, in some examples, the sample image (such as Figure 3 x0 in the input diffusion model VAE encoder 310 (such as Figure 3 The VAE ε in the example is used, and the VAE encoder 310 outputs a latent representation of the sample image. Then, noise is gradually added to the latent representation of the sample image using latent diffusion algorithms to obtain a noisy image.

[0079] Third, regarding step (3):

[0080] The first vector representation of prompt obtained in step (1) is input into the text encoder 340, and the text encoder 340 extracts the feature information of the first vector representation of prompt, and inputs the feature information and the noisy image obtained in step (2) into the UNet network 330 (e.g. Figure 3 ε θ ), the UNet network 330 predicts the added noise, that is, generates the predicted noise; then, the predicted noise is subtracted from the noisy image to obtain the latent representation of the predicted image; and the latent representation of the predicted image is input into the VAE decoder 320 (such as Figure 3In the process of prediction by the diffusion model, an intermediate result is also obtained, namely, the attention map of the noisy image, which can indicate the location of the visual text in the noisy image predicted by the UNet network 330.

[0081] The above introduces the input data, output data, and data processing in the model during the training process of the image-text generation model. Figure 3 M k ), predicted image (such as Figure 3 x'0 in), prediction noise (such as Figure 3 ε t ) and the attention map of the noisy image, the total loss function can be calculated, and then the total loss function can be used to adjust the parameters in the image-text generation model to realize the training of the image-text generation model.

[0082] In some examples, the total loss function is calculated as follows:

[0083] Determine multiple individual loss functions and use them to calculate the total loss function.

[0084] For example, the weighted sum of multiple individual loss functions may be calculated to obtain a total loss function, wherein the weight of each individual loss function is set according to a preset rule.

[0085] Individual loss functions can include: attention loss function (such as Figure 3 In ), OCR recognition loss function (such as Figure 3 In ), local mean square error loss function (such as Figure 3 In ) and diffusion loss function (such as Figure 3 In )

[0086] The following introduces the above single loss functions respectively:

[0087] (1) Attention loss function

[0088] In some examples, the attention loss function is determined using attention maps of the mask image and the noisy image.

[0089] For example, the attention loss function can be determined using the following calculation formula:

[0090] in,

[0091] represents the attention loss function;

[0092] N, represents the number of visual texts in the sample image;

[0093] k, represents the ordinal number of the glyph graph of the visual text, and the value of k ranges from 1 to N;

[0094] represents the cross-attention (CA) model in the UNet network 330 for x t and treatment;

[0095] x t , represents the image after adding noise;

[0096] Represents the vector representation (embedding) corresponding to the k-th visual text;

[0097] M k , represents the mask image corresponding to the k-th visual text.

[0098] The attention loss function can urge the model to learn the correspondence between the visual text in the image and the visual text description in the user's instructions by narrowing the distance between the attention map of the diffusion model and the image mask.

[0099] (2) OCR recognition loss function

[0100] In some examples, a mask image and a predicted image are used to determine a glyph graph included in the predicted image; and an OCR recognition loss function is determined using a glyph graph included in the predicted image and a glyph graph corresponding to a visual text included in a prompt.

[0101] For example, the OCR recognition loss function can be determined using the following calculation formula:

[0102] in,

[0103] Represents the OCR recognition loss function;

[0104] N, represents the number of visual texts in the sample image;

[0105] k, represents the ordinal number of the glyph graph of the visual text, and the value of k ranges from 1 to N;

[0106] represents the weight associated with a time step (t); for example, the larger t is, The bigger;

[0107] CTC() is a loss function calculation formula for the OCR module;

[0108] Represents the predicted image x'0;

[0109] M k , represents the mask image corresponding to the k-th visual text;

[0110] ⊙, represents the element-wise product operation;

[0111] g k , represents the grapheme graph of the k-th visual text.

[0112] The definition of the Connectivity Temporal Classification (CTC) function is: given an input sequence X and a target sequence Y, where X represents the feature sequence of the input image and Y represents the sequence of target characters, CTC (X, Y) is defined as the joint probability of all possible alignments. The CTC loss function considers all possible alignments by summing the alignment probabilities and then taking the negative logarithm to represent the loss. This design allows the model to produce more accurate character sequence outputs during training.

[0113] The OCR recognition loss function uses optical character recognition (OCR) technology to detect text information in the model-generated images, and encourages the model to generate recognizable and correctly spelled visual text.

[0114] (3) Local mean square error loss function

[0115] In some examples, the predicted noise and the noise added to the sample image are used to calculate the mean square error of the noise prediction; the mean square error and the mask image are used to calculate the local mean square error loss function.

[0116] For example, the local mean square error loss function can be determined using the following calculation formula:

[0117] in,

[0118] in,

[0119] N, represents the number of visual texts in the sample image;

[0120] k, represents the ordinal number of the glyph graph of the visual text, and the value of k ranges from 1 to N;

[0121] ω k , represents the weight of the k-th visual text;

[0122] ε θ , represents the UNet model;

[0123] x0, represents the sample image;

[0124] t, represents the time step when adding noise;

[0125] c, represents the first vector representation of prompt;

[0126] ε θ (x0,t,c), represents the prediction noise;

[0127] ε0, represents the noise added to the sample image;

[0128] represents the mean square error of noise prediction;

[0129] M k , represents the mask image corresponding to the k-th visual text;

[0130] Represents the portion of the glyph graph where the visual text is extracted from the mean squared error of the noisy prediction.

[0131] The local mean square error loss function encourages the model to focus on the text area in the image by weighting the text area.

[0132] In some examples, ω k The value of can be equal to the area of ​​the sample image divided by the area of ​​the glyph map of the kth visual text. Since the smaller the area of ​​the visual text, the higher the difficulty of fitting it should be, the local mean square error loss function uses different values ​​of ω k , urging the model to pay more attention to smaller text regions.

[0133] (4) Diffusion loss function

[0134] In some examples, the diffusion loss function is calculated using the predicted noise and the noise added to the sample image.

[0135] The diffusion loss function in the embodiment of the present disclosure adopts the mean square error (MSC) function, which refers to the square difference between all predicted values ​​and true values, and averages them.

[0136] For example, the diffusion loss function can be determined using the following calculation formula:

[0137] in,

[0138] ε θ , represents the UNet model;

[0139] x0, represents the sample image;

[0140] t, represents the time step when adding noise;

[0141] c, represents the first vector representation of prompt;

[0142] ε θ (x0,t,c), represents the prediction noise;

[0143] ε0 represents the noise added to the sample image.

[0144] The total loss function can be calculated using the above-mentioned individual loss functions. In some examples, a weighted sum of multiple individual loss functions is calculated to obtain the total loss function, wherein the weights of the individual loss functions are set according to a preset rule.

[0145] For example, the total loss function can be determined using the following calculation:

[0146] in,

[0147] represents the total loss function;

[0148] Calculate the weighted sum of the above-mentioned attention loss function, OCR recognition loss function and local mean square error loss function, and add the weighted sum to the diffusion loss function to obtain the total loss function. Among them, the weights (i.e., λ1, λ2) used when calculating the weighted sum can be set according to preset rules. The embodiment of the present disclosure can set the weights of each single loss function, so as to achieve focused training of various single capabilities of the model. For example, if the focus is on urging the model to learn the correspondence between the visual text in the image and the description of the visual text in the user's instructions, then increase the weight of the attention loss function (i.e., λ1); if the focus is on training the model to produce more accurate character sequence outputs, then increase the weight of the OCR recognition loss function (i.e., λ2); if the focus is on urging the model to pay attention to the text area in the image, then increase the weight of the local mean square error loss function (i.e., 1-λ1-λ2).

[0149] like Figure 3 As shown, during the training process, the OCR module 350, as well as the encoder 310, decoder 320 and text encoder 340 in the text-conditioned LDM can be parameter-frozen parts, that is, during the training process, the parameters of these parts are not adjusted; the UNet network 330 in the text-conditioned LDM is a learnable part, that is, during the training process, the parameters of this part are adjusted.

[0150] The above introduces the training process of the image-text generation model proposed in the embodiment of the present disclosure.

[0151] By using the trained image-text generation model, we can use the prompt input by the user to generate the corresponding visual text image.

[0152] The disclosed embodiment also proposes a method for generating a visual text image. Figure 4 As shown, the method includes: inputting an instruction (prompt) 410 into a picture-text generation model 420, and the picture-text generation model 420 generates a visual text image 430 based on the instruction; wherein the picture-text generation model 420 is trained using the above-mentioned training method.

[0153] In some embodiments, the image-text generation model 420 includes a mixed granularity input (Mixed GranularityInput) module and a diffusion model; wherein the mixed granularity input model is used to receive a prompt input by a user and generate a first vector representation of the prompt, wherein the first vector representation carries glyph information of the visual text contained in the prompt; and the diffusion model is used to generate a visual text image using the first vector representation.

[0154] The hybrid granular input model includes a glyph rendering module, an OCR module, and a tokenizer module. The glyph rendering module is used to extract one or more visual texts from the prompt, and perform glyph rendering on each visual text to obtain a glyph map of each visual text; the OCR module is used to perform OCR encoding on each glyph map to obtain a vector representation of each glyph map; the tokenizer module is used to segment the prompt, and then generate a vector representation of each segmented word, thereby obtaining an initial vector representation of the prompt; and then the vector representation of the glyph map is used to replace the relevant content in the initial vector representation, so as to obtain the first vector representation of the prompt.

[0155] The diffusion model is a text-conditioned LDM, including an encoder, a decoder, a UNet network, and a text encoder. The diffusion model generates the corresponding visual text image based on the first vector representation of the prompt. In the model inference stage, only the decoder, the UNet network, and the text encoder are used; for example, the first vector representation of the prompt is input into the text encoder, and the text encoder extracts the feature information of the first vector representation of the prompt; the feature information and noise are input into the UNet network, and the UNet network generates the prediction noise; then, the prediction noise is subtracted from the noise input into the UNet network to obtain the hidden representation of the prediction image; and the hidden representation of the prediction image is input into the decoder to obtain the prediction image.

[0156] When the above method is used to generate a visual text image, the user only needs to input a prompt into the image-text generation model without specifying the position of the text in the visual text image. Thus, the image-text generation model can select a suitable text position according to the actual generated image, thereby ensuring the harmony of the text font, color, etc. with the image background, and improving the quality of the visual text image.

[0157] The present disclosure also provides a device for training a graph-text generation model. Figure 5 is a structural schematic diagram of a graph-text generation model training device 500 according to an embodiment of the present disclosure, comprising:

[0158] A first input module 510, used to input a sample image and an instruction into a to-be-trained image-text generation model, and the to-be-trained image-text generation model generates an attention map of predicted noise, predicted image, and noisy image;

[0159] A loss function calculation module 520 is used to generate a mask image according to the sample image, and calculate a total loss function using the mask image, the predicted image, the predicted noise, and the attention map of the noisy image;

[0160] The training module 530 is used to train the image-text generation model to be trained based on the total loss function.

[0161] In some embodiments, the image-text generation model to be trained generates attention maps of predicted noise, predicted image, and noisy image, including:

[0162] generating a first vector representation of the instruction, the first vector representation carrying glyph information of the visual text contained in the instruction;

[0163] Add noise to the sample image to obtain a noisy image;

[0164] Using the noisy image and the first vector representation of the instruction, an attention map of the predicted noise, the predicted image, and the noisy image is generated.

[0165] In some embodiments, the image-text generation model to be trained generates attention maps of predicted noise, predicted image, and noisy image, including:

[0166] generating a first vector representation of the instruction, the first vector representation carrying glyph information of the visual text contained in the instruction;

[0167] Add noise to the sample image to obtain a noisy image;

[0168] Using the noisy image and the first vector representation of the instruction, an attention map of the predicted noise, the predicted image, and the noisy image is generated.

[0169] In some implementations, generating a first vector representation of an instruction includes:

[0170] Extracting visual text from the instruction, performing glyph rendering on the visual text to obtain a glyph graph corresponding to the visual text; extracting features of the glyph graph using an optical character recognition module to obtain a vector representation of the glyph graph;

[0171] Determine the initial vector representation of the instruction;

[0172] The relevant content in the initial vector representation is replaced by the vector representation of the glyph image to obtain a first vector representation; wherein the relevant content includes the vector representation of the visual text corresponding to the glyph image.

[0173] In some implementations, extracting visual text from the instruction and performing glyph rendering on the visual text includes:

[0174] According to the preset decomposition granularity, N visual texts are extracted from the instruction, and glyph rendering is performed on each visual text to obtain N glyph images; wherein the decomposition granularity is a single character, and the decomposition granularity corresponds to the language type of the visual text.

[0175] When the instruction is a Chinese instruction, the decomposition granularity is a single Chinese character; when the instruction is an English instruction, the decomposition granularity is a single word.

[0176] Figure 6 is a structural schematic diagram of a graph-text generation model training device 600 according to an embodiment of the present disclosure. In some implementations, the loss function calculation module 520 includes a single loss function determination module 630 and a total loss function determination module 640;

[0177] The single loss function determination module 630 is used to determine multiple single loss functions, and the determination of multiple single loss functions includes: determining an attention loss function using an attention map of a mask image and a noisy image; and determining a glyph map contained in the predicted image using the mask image and the predicted image; determining an optical character recognition loss function using a glyph map contained in the predicted image and a glyph map corresponding to a visual text contained in an instruction;

[0178] The total loss function determination module 640 is used to calculate the total loss function using multiple individual loss functions.

[0179] In some implementations, the single loss function determination module 630 is further configured to:

[0180] Using the predicted noise and the noise added to the sample image, the mean square error of the noise prediction is calculated;

[0181] Using the mean square error and the mask image, the local mean square error loss function is calculated.

[0182] In some implementations, the single loss function determination module 630 is further configured to:

[0183] The diffusion loss function is calculated using the predicted noise and the noise added to the sample image.

[0184] In some embodiments, the total loss function determination module 640 is used to calculate the weighted sum of multiple individual loss functions to obtain a total loss function;

[0185] Among them, the weight of each single loss function is set according to the preset rules.

[0186] The present disclosure also provides a visual text image generation device. Figure 7 is a schematic diagram of the structure of a visual text image generating device 700 according to an embodiment of the present disclosure, comprising:

[0187] The second input module 710 is used to input instructions into the image-text generation model, and the image-text generation model generates a visual text image based on the instructions; wherein the image-text generation model is trained using the training device provided by the present disclosure.

[0188] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0189] In the technical solution disclosed in the present invention, the acquisition, storage and application of personal information of users involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0190] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0191] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0192] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0193] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0194] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as detection methods. For example, in some embodiments, the detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the detection method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the detection method in any other appropriate manner (e.g., by means of firmware).

[0195] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0196] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0197] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0198] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0199] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0200] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0201] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0202] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a graph-text generation model, the method comprising: Inputting the sample image and the instruction into the image-text generation model to be trained, and generating the attention map of the predicted noise, the predicted image and the noisy image by the image-text generation model to be trained; Generate a mask image according to the sample image, and calculate a total loss function using the mask image, the predicted image, the predicted noise, and an attention map of the noisy image; Based on the total loss function, the image-text generation model to be trained is trained.

2. The method according to claim 1, wherein: The image-text generation model to be trained generates an attention map of predicted noise, predicted image and noisy image, including: generating a first vector representation of the instruction, wherein the first vector representation carries glyph information of the visual text contained in the instruction; Adding noise to the sample image to obtain a noisy image; An attention map of predicted noise, predicted image, and noisy image is generated using the noisy image and the first vector representation of the instruction.

3. The method according to claim 2, wherein: The generating a first vector representation of the instruction comprises: Extracting visual text from the instruction, performing glyph rendering on the visual text to obtain a glyph graph corresponding to the visual text; extracting features of the glyph graph using an optical character recognition module to obtain a vector representation of the glyph graph; determining an initial vector representation of the instruction; The vector representation of the glyph graph is used to replace the relevant content in the initial vector representation to obtain the first vector representation; wherein the relevant content includes the vector representation of the visual text corresponding to the glyph graph.

4. The method according to claim 3, wherein: The step of extracting visual text from the instruction and performing glyph rendering on the visual text includes: According to the preset disassembly granularity, N visual texts are extracted from the instruction, and glyph rendering is performed on each of the visual texts to obtain N glyph images; wherein, The disassembly granularity is a single character, and the disassembly granularity corresponds to a language type of the visual text.

5. The method according to any one of claims 2 to 4, wherein: The method of calculating a total loss function by using the attention map of the mask image, the predicted image, the predicted noise and the noisy image comprises: Determine a plurality of individual loss functions, and calculate the total loss function using the plurality of individual loss functions; Wherein, determining multiple single loss functions includes: determining an attention loss function using the attention maps of the mask image and the noisy image; and, Using the mask image and the predicted image, determine a glyph graph included in the predicted image; using the glyph graph included in the predicted image and a glyph graph corresponding to the visual text included in the instruction, determine an optical character recognition loss function.

6. The method according to claim 5, wherein: Determining a plurality of individual loss functions further comprises: Calculating a mean square error of noise prediction using the predicted noise and the noise added to the sample image; A local mean square error loss function is calculated using the mean square error and the mask image.

7. The method according to claim 5 or 6, wherein: Determining a plurality of individual loss functions further comprises: The diffusion loss function is calculated using the predicted noise and the noise added to the sample image.

8. The method according to any one of claims 5 to 7, wherein: The calculating the total loss function by using the multiple individual loss functions comprises: Calculating a weighted sum of the multiple individual loss functions to obtain the total loss function; Among them, the weight of each of the single loss functions is set according to preset rules.

9. A method for generating a visual text image, the method comprising: The instruction is input into a picture-text generation model, and the picture-text generation model generates a visual text image based on the instruction; wherein the picture-text generation model is trained using the training method described in any one of claims 1-8.

10. A device for training a graph-text generation model, for training a graph-text generation model, the device comprising: A first input module, used for inputting a sample image and an instruction into a to-be-trained image-text generation model, and generating an attention map of predicted noise, predicted image and noisy image by the to-be-trained image-text generation model; A loss function calculation module, used to generate a mask image according to the sample image, and calculate a total loss function using the mask image, the predicted image, the predicted noise and the attention map of the noisy image; A training module is used to train the image-text generation model to be trained based on the total loss function.

11. The device according to claim 10, wherein: The image-text generation model to be trained generates an attention map of predicted noise, predicted image and noisy image, including: generating a first vector representation of the instruction, wherein the first vector representation carries glyph information of the visual text contained in the instruction; Adding noise to the sample image to obtain a noisy image; An attention map of predicted noise, predicted image, and noisy image is generated using the noisy image and the first vector representation of the instruction.

12. The device according to claim 11, wherein The generating a first vector representation of the instruction comprises: Extracting visual text from the instruction, performing glyph rendering on the visual text to obtain a glyph graph corresponding to the visual text; extracting features of the glyph graph using an optical character recognition module to obtain a vector representation of the glyph graph; determining an initial vector representation of the instruction; The vector representation of the glyph graph is used to replace the relevant content in the initial vector representation to obtain the first vector representation; wherein the relevant content includes the vector representation of the visual text corresponding to the glyph graph.

13. The device according to claim 12, wherein: The step of extracting visual text from the instruction and performing glyph rendering on the visual text includes: According to the preset disassembly granularity, N visual texts are extracted from the instruction, and glyph rendering is performed on each of the visual texts to obtain N glyph images; wherein, The disassembly granularity is a single character, and the disassembly granularity corresponds to a language type of the visual text.

14. The device according to any one of claims 11 to 13, wherein: The loss function calculation module includes: A single loss function determination module is used to determine multiple single loss functions, wherein the determining of multiple single loss functions includes: determining an attention loss function using the attention map of the mask image and the noisy image; and determining a glyph map included in the predicted image using the mask image and the predicted image; determining an optical character recognition loss function using the glyph map included in the predicted image and the glyph map corresponding to the visual text included in the instruction; A total loss function determination module is used to calculate the total loss function using the multiple individual loss functions.

15. The device according to claim 14, wherein: The single loss function determination module is also used for: Calculating a mean square error of noise prediction using the predicted noise and the noise added to the sample image; A local mean square error loss function is calculated using the mean square error and the mask image.

16. The device according to claim 14 or 15, wherein: The single loss function determination module is also used for: The diffusion loss function is calculated using the predicted noise and the noise added to the sample image.

17. The device according to any one of claims 14 to 16, wherein: The total loss function determination module is used to calculate the weighted sum of the multiple individual loss functions to obtain the total loss function; Among them, the weight of each of the single loss functions is set according to preset rules.

18. A visual text image generation device, the device comprising: The second input module is used to input instructions into a picture-text generation model, and the picture-text generation model generates a visual text image based on the instructions; wherein the picture-text generation model is trained using the training device described in any one of claims 10-17.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

21. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images