Figure graph model training method, character generation method and device

Abstract

The invention discloses a text graph model training method and a text generation method and device with semantic image information. According to the method, the character pattern of the sample character is learned by using the text graph model, the trained text graph model has the performance of generating the target character, the generated target character has the same character pattern as the character to be processed, and the visual effect presented by the character pattern carries semantic image information corresponding to the character to be processed; according to the method for generating the text with the semantic image information, the target text is generated by using the text generation graph model, the text with the semantic image information can be automatically and quickly generated, the manual participation degree in the text generation process is reduced, and the text generation efficiency is improved. Therefore, efficiency can be improved, cost can be reduced, and styles of different characters can be easily unified when multiple characters are generated. The method is widely applied to the technical field of text graphics.

Description

Technical Field

[0001] The present invention relates to the technical field of text graphs, and in particular to a text graph model training method, a text generation method and a device with semantic image information. Background Art

[0002] In the fields of printing, advertising design, art design, web page production, software development, and packaging decoration, it is often necessary to design special fonts. When a text is displayed in this special font, the visual effect presented by the text is like the semantic image of the text, thereby producing a good aesthetic and publicity effect. Such text is text with semantic image information. At present, when designing text with semantic image information, manual painting and other methods are usually used, which has the disadvantages of low efficiency, high cost, and difficulty in unifying the style. Summary of the invention

[0003] In view of the current technical problems faced in designing text with semantic image information, such as low efficiency, high cost, and difficulty in unifying style, the purpose of the present invention is to provide a text-generated graph model training method, a text generation method and device with semantic image information.

[0004] In one aspect, an embodiment of the present invention includes a method for training a text graph model, the method comprising the following steps:

[0005] Get sample text;

[0006] Obtaining a first sample image according to the sample text;

[0007] Performing noise addition processing on the first sample image to obtain a first noise addition feature map; wherein the noise addition processing is used to add actual noise information;

[0008] Performing noise prediction processing on the first noisy feature map to obtain predicted noise information;

[0009] The Vincent graph model is trained according to the predicted noise information and the actual noise information to obtain a trained Vincent graph model.

[0010] Furthermore, the Wensheng graph model includes: a rasterization module;

[0011] The step of obtaining a first sample image according to the sample text includes:

[0012] Rasterizing the sample text to obtain a first sample image;

[0013] The text image model includes: a rasterization module; the step of rasterizing the sample text to obtain a first sample image includes:

[0014] Extracting outline information of the sample text using a font library;

[0015] converting the contour information into a curve;

[0016] determining a set of control points according to the curve;

[0017] The control points are input into the rasterization module for rasterization processing to obtain the first sample image.

[0018] Furthermore, the performing noise processing on the first sample image to obtain a first noise feature map includes:

[0019] Encoding the first sample image to obtain a first feature map;

[0020] The actual noise information is randomly generated using a random function;

[0021] The actual noise information is added to the first feature map to obtain the first noisy feature map.

[0022] Furthermore, the text graph model includes: a text encoding module and an image segmentation module;

[0023] The performing noise prediction processing on the first noise-added feature map to obtain predicted noise information includes:

[0024] Get the sample description text corresponding to the sample text;

[0025] Inputting the sample description text into the text encoding module for processing to obtain a first text vector;

[0026] The first text vector is input into the image segmentation module as a control condition for noise prediction processing, and the first noisy feature map is input into the image segmentation module for noise prediction processing to obtain predicted noise information.

[0027] Furthermore, the image segmentation module includes a plurality of cross-attention downsampling blocks, a downsampling block, a cross-attention intermediate block, an upsampling block and a plurality of cross-attention upsampling blocks that are fully connected in sequence; wherein each of the cross-attention downsampling blocks and each of the cross-attention intermediate blocks have their own corresponding dimensional sizes.

[0028] Furthermore, the step of inputting the first text vector into the image segmentation module as a control condition for noise prediction processing, inputting the first noise feature map into the image segmentation module for noise prediction processing, and obtaining predicted noise information includes:

[0029] The first text vector is received through a plurality of the cross-attention downsampling blocks, and features are integrated into the first noisy feature map to obtain first low-dimensional feature information;

[0030] Processing the first low-dimensional feature information through the downsampling block to obtain second low-dimensional feature information;

[0031] Processing the second low-dimensional feature information through the cross-attention intermediate block to obtain third low-dimensional feature information;

[0032] Processing the third low-dimensional feature information by the upsampling block to obtain first high-dimensional feature information;

[0033] Receiving the first text vector and the processing results of the cross-attention down-sampling blocks of the same dimension through the multiple cross-attention up-sampling blocks, integrating the first high-dimensional feature information according to the first text vector, and adding the features according to the processing results of the cross-attention down-sampling blocks to obtain the second high-dimensional feature information;

[0034] Determine the difference between the second high-dimensional feature information and the first noisy feature map to obtain the predicted noise information.

[0035] Furthermore, the training of the Vincent graph model according to the predicted noise information and the actual noise information to obtain the trained Vincent graph model includes:

[0036] Perform at least one round of training process; each round of the training process obtains corresponding predicted noise information and actual noise information;

[0037] For any round of the training process, determining a loss function value according to the predicted noise information and the actual noise information;

[0038] Determine the cumulative loss function value according to the loss function value of each round of the training process;

[0039] Parameters of the Wensheng graph model are updated according to the cumulative loss function value to obtain a trained Wensheng graph model.

[0040] Further, determining the loss function value according to the predicted noise information and the actual noise information includes:

[0041] Calculate the loss function value, wherein the calculation formula of the loss function value is:

[0042] loss i =(b i -a i ) i

[0043] Among them, loss i is the loss function value corresponding to the i-th round of training process, b i is the predicted noise information obtained in the i-th round of the training process, a i is the actual noise information obtained in the i-th round of training process, z i The first noisy feature map obtained in the i-th round of the training process.

[0044] Furthermore, the determining of the cumulative loss function value according to the loss function value of each round of the training process includes:

[0045] Calculate the cumulative loss function value, where the calculation formula of the cumulative loss function value is:

[0046]

[0047] Among them, loss i is the loss function value corresponding to the i-th round of the training process, LOSS is the cumulative loss function value, and loss i is the loss function value corresponding to the i-th round of the training process, and N is the total number of rounds of the training process that have been executed.

[0048] Furthermore, the updating of parameters of the Wensheng graph model according to the cumulative loss function value includes:

[0049] According to the accumulated loss function value, back-propagation update is performed on the parameters of at least one module among the rasterization module, the image segmentation module and the text encoding module.

[0050] Furthermore, the back-propagation updating of the parameters of at least one of the rasterization module, the image segmentation module and the text encoding module according to the accumulated loss function value comprises:

[0051] Freezing parameters of the image segmentation module and the text encoding module;

[0052] According to the accumulated loss function value, back-propagation is performed on the parameters of the rasterization module to update them.

[0053] On the other hand, an embodiment of the present invention includes a method for generating text with semantic image information, the method for generating text with semantic image information comprising:

[0054] Obtaining a Wensheng graph model; the Wensheng graph model is obtained by training a Wensheng graph model training method;

[0055] Get the text to be processed and the corresponding description text to be processed;

[0056] The text to be processed and the description text to be processed are processed using the text graph model to generate target text.

[0057] Furthermore, the using the text graph model to process the text to be processed and the description text to be processed to generate the target text includes:

[0058] The processing result of the text-generated graph model is mapped to the pixel space to obtain the target text.

[0059] On the other hand, an embodiment of the present invention includes a device for training a text graph model, the device comprising:

[0060] A text acquisition module is used to acquire sample text;

[0061] An image acquisition module, used for acquiring a first sample image according to the sample text;

[0062] A noise adding module, used for performing noise adding processing on the first sample image to obtain a first noise adding feature map; wherein the noise adding processing is used for adding actual noise information;

[0063] A prediction module, used for performing noise prediction processing on the first noise-added feature map to obtain predicted noise information;

[0064] The training module is used to train the Vincent graph model according to the predicted noise information and the actual noise information to obtain a trained Vincent graph model.

[0065] Further, obtaining a first sample image according to the sample text includes:

[0066] Rasterizing the sample text to obtain a first sample image;

[0067] The text image model includes: a rasterization module; the step of rasterizing the sample text to obtain a first sample image includes:

[0068] Extracting outline information of the sample text using a font library;

[0069] converting the contour information into a curve;

[0070] determining a set of control points according to the curve;

[0071] The control points are input into the rasterization module for rasterization processing to obtain the first sample image.

[0072] Furthermore, the performing noise processing on the first sample image to obtain a first noise feature map includes:

[0073] Encoding the first sample image to obtain a first feature map;

[0074] The actual noise information is randomly generated using a random function;

[0075] The actual noise information is added to the first feature map to obtain the first noisy feature map.

[0076] Further, performing noise prediction processing on the first noisy feature map to obtain predicted noise information includes:

[0077] Get the sample description text corresponding to the sample text;

[0078] Inputting the sample description text into the text encoding module for processing to obtain a first text vector;

[0079] The first text vector is input into the image segmentation module as a control condition for noise prediction processing, and the first noisy feature map is input into the image segmentation module for noise prediction processing to obtain predicted noise information.

[0080] Furthermore, the image segmentation module includes a plurality of cross-attention downsampling blocks, a downsampling block, a cross-attention intermediate block, an upsampling block and a plurality of cross-attention upsampling blocks that are fully connected in sequence; wherein each of the cross-attention downsampling blocks and each of the cross-attention intermediate blocks have their own corresponding dimensional sizes.

[0081] Furthermore, the step of inputting the first text vector into the image segmentation module as a control condition for noise prediction processing, inputting the first noise feature map into the image segmentation module for noise prediction processing, and obtaining predicted noise information includes:

[0082] The first text vector is received through a plurality of the cross-attention downsampling blocks, and features are integrated into the first noisy feature map to obtain first low-dimensional feature information;

[0083] Processing the first low-dimensional feature information through the downsampling block to obtain second low-dimensional feature information;

[0084] Processing the second low-dimensional feature information through the cross-attention intermediate block to obtain third low-dimensional feature information;

[0085] Processing the third low-dimensional feature information by the upsampling block to obtain first high-dimensional feature information;

[0086] Receiving the first text vector and the processing results of the cross-attention down-sampling blocks of the same dimension through the multiple cross-attention up-sampling blocks, integrating the first high-dimensional feature information according to the first text vector, and adding the features according to the processing results of the cross-attention down-sampling blocks to obtain the second high-dimensional feature information;

[0087] Determine the difference between the second high-dimensional feature information and the first noisy feature map to obtain the predicted noise information.

[0088] Furthermore, the training of the Vincent graph model according to the predicted noise information and the actual noise information to obtain the trained Vincent graph model includes:

[0089] Perform at least one round of training process; each round of the training process obtains corresponding predicted noise information and actual noise information;

[0090] For any round of the training process, determining a loss function value according to the predicted noise information and the actual noise information;

[0091] Determine the cumulative loss function value according to the loss function value of each round of the training process;

[0092] Parameters of the Wensheng graph model are updated according to the cumulative loss function value to obtain a trained Wensheng graph model.

[0093] Further, determining the loss function value according to the predicted noise information and the actual noise information includes:

[0094] Calculate the loss function value, wherein the calculation formula of the loss function value is:

[0095] loss i =(b i -a i ) i

[0096] Among them, loss i is the loss function value corresponding to the i-th round of training process, b i is the predicted noise information obtained in the i-th round of the training process, a i is the actual noise information obtained in the i-th round of training process, z i The first noisy feature map obtained in the i-th round of the training process.

[0097] Furthermore, the determining of the cumulative loss function value according to the loss function value of each round of the training process includes:

[0098] Calculate the cumulative loss function value, where the calculation formula of the cumulative loss function value is:

[0099]

[0100] Among them, loss i is the loss function value corresponding to the i-th round of the training process, LOSS is the cumulative loss function value, and loss i is the loss function value corresponding to the i-th round of the training process, and N is the total number of rounds of the training process that have been executed.

[0101] Furthermore, the updating of parameters of the Wensheng graph model according to the cumulative loss function value includes:

[0102] According to the accumulated loss function value, back-propagation update is performed on the parameters of at least one module among the rasterization module, the image segmentation module and the text encoding module.

[0103] Furthermore, the back-propagation updating of the parameters of at least one of the rasterization module, the image segmentation module and the text encoding module according to the accumulated loss function value comprises:

[0104] Freezing parameters of the image segmentation module and the text encoding module;

[0105] According to the accumulated loss function value, back-propagation is performed on the parameters of the rasterization module to update them.

[0106] On the other hand, an embodiment of the present invention includes a text generation device with semantic image information, the text generation device with semantic image information includes:

[0107] A Wensheng graph module, used to obtain a Wensheng graph model; the Wensheng graph model is trained by a Wensheng graph model training method;

[0108] An acquisition module, used to acquire the text to be processed and the corresponding description text to be processed;

[0109] A processing module is used to use the text-generated graph model to process the text to be processed and the description text to be processed to generate target text.

[0110] The beneficial effects of the present invention are as follows: the text graph model training method in the embodiment learns the glyphs of sample texts by using the text graph model, and the trained text graph model has the performance of generating target texts, and the generated target texts have the same glyphs as the texts to be processed, and the visual effect presented by the font carries the semantic image information corresponding to the texts to be processed; the text generation method with semantic image information in the embodiment generates target texts by using the text graph model, and can automatically and quickly generate texts with semantic image information, and since the degree of manual participation in the text generation process is reduced, efficiency can be improved, costs can be reduced, and the styles of different texts can be easily unified when generating multiple texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0111] Figure 1 Schematic diagram of the steps of implementing the Chinese raw graph model training method;

[0112] Figure 2 and Figure 3 It is a schematic diagram of the Chinese raw graph model in the embodiment;

[0113] Figure 4 Schematic diagram of the structure of the image segmentation module in the embodiment;

[0114] Figure 5 Schematic diagram of the principle of any round of training process performed on the Wensheng graph model in the embodiment;

[0115] Figure 6 Schematic diagram of the steps of the method for training a text graph model when only one round of training process is performed in an embodiment;

[0116] Figure 7 Schematic diagram of the working principle of the image segmentation module in the embodiment;

[0117] Figure 8 Schematic diagram of the structure of the CrossAttnDownBlock2D unit in the embodiment;

[0118] Fig. 9 It is a structural diagram of the ResnetBlock2D unit in the embodiment;

[0119] Fig.10 Schematic diagram of the structure of the Transformer2DModel unit in the embodiment;

[0120] Fig.11 It is a schematic diagram of the structure of the MidBlock2DCrossAttn unit in the embodiment;

[0121] Fig.12 Schematic diagram of the structure of the CrossAttnUpBlock2D unit in the embodiment;

[0122] Fig.13 A schematic diagram of the steps of a method for generating text with semantic image information in an embodiment;

[0123] Fig.14 Schematic diagram of the principle of the method for generating text with semantic image information in the embodiment;

[0124] Fig.15 Schematic diagram of the display effect of the text to be processed in the embodiment;

[0125] Fig.16 A schematic diagram of the display effect of the target text output by the Chinese raw image model in the embodiment;

[0126] Fig.17 Schematic diagram of the structure of the Chinese raw image model training device in the embodiment;

[0127] Fig.18 It is a structural schematic diagram of a Chinese image generating device in an embodiment;

[0128] Fig.19 is a schematic diagram of the structure of a computer device in an embodiment;

[0129] Fig. 20 Schematic diagram of the working principle of the computer-readable storage medium in the embodiment. DETAILED DESCRIPTION

[0130] Terminology explanation:

[0131] Unet: UNet (also known as U-Net) is a convolutional neural network (CNN) architecture for image segmentation. The structure of UNet consists of two parts: an encoder and a decoder, which is figuratively U-shaped, hence the name. The encoder in Unet consists of multiple convolutional blocks, each of which includes a convolutional layer (usually a 3x3 convolution kernel), batch normalization, and an activation function (usually ReLU). These convolutional blocks help the network learn the low-level features of the image. After each convolutional block, a pooling layer is usually added, such as maximum pooling or average pooling, to reduce the size of the feature map while retaining important information. The decoder in Unet consists of multiple deconvolutional blocks, each of which contains a deconvolutional layer (also known as a transposed convolution), batch normalization, and an activation function. These blocks help learn the high-level semantic information of the image. After each deconvolutional block, an upsampling layer can be added. Layer) to increase the size of the feature map, corresponding to the pooling layer in the encoder; each layer in the decoder is connected to the corresponding layer in the encoder (Concatenation), through skip connections, which helps to combine the semantic information of the bottom and high layers together;

[0132] CLIP: The full name of CLIP is Contrastive Language-Image Pre-training, which is a pre-training method or model based on contrastive text-image pairs. CLIP is a multimodal model based on contrastive learning. The training data of CLIP is text-image pairs: an image and its corresponding text description. It is hoped that through contrastive learning, the model can learn the matching relationship between text-image pairs. CLIP includes an image encoder and a text encoder. The image encoder can map the input image to a shared vector space, and the text encoder can map the input text to the same vector space. In addition, only the image encoder or the text encoder in CLIP can be used as needed. CLIP can be used in combination with Unet. For example, the text features extracted by the text encoder in CLIP are embedded in the UNet of the diffusion model through cross attention. Specifically, the text features are used as the key and value of attention, and the features of UNet are used as queries. In other words, CLIP can serve as a bridge between text and pictures.

[0133] Diffvg: The full name of Diffvg is Differentiable Vector Graphics, which means differentiable vector graphics rasterization; Diffvg is a rasterization tool that can convert vector graphics or text layers into bitmaps (raster images).

[0134] In this embodiment, the task to be performed is to generate corresponding text with semantic image information based on a text to be processed. The text to be processed can be an existing font such as Songti, Kaiti, etc., or a text encoding obtained through encoding standards such as GB2312, GB18030 or Unicode (not corresponding to any specific font).

[0135] In order to perform the above task, in this embodiment, a trained text-graph model is used, and the text to be processed is input into the text-graph model for processing. The text-graph model can extract the semantics of the text to be processed and generate a new text. The meaning of this new text is the same as that of the text to be processed. That is, when the reader reads the text to be processed and the new text generated by the text-graph model, the reader can feel that the new text is the same as the text to be processed, and the configuration of the new text carries the semantic image information of the text to be processed.

[0136] In this embodiment, a text-based graph model training method may be executed to obtain a text-based graph model having a function of generating text with semantic image information.

[0137] In this embodiment, refer to Figure 1 ,The Wensheng graph model training method includes the following steps:

[0138] P1. Establish a Wensheng graph model;

[0139] P2. Perform at least one round of training process;

[0140] P3. Determine the cumulative loss function value based on the loss function value of each round of training process;

[0141] P4. Update the parameters of the Wensheng graph model according to the cumulative loss function value.

[0142] In step P1, the structure of the established text graph model is as follows Figure 2 As shown, it includes a rasterization module, an image segmentation module and a text encoding module. The rasterization module, the image segmentation module and the text encoding module can be hardware modules or software modules with corresponding functions. For example, the rasterization module is a software program with a rasterization function, the image segmentation module is a software program running a Unet network, and the text encoding module is a software program with a text encoding function.

[0143] Specifically, when executing step P1, that is, the step of establishing a cultural graph model, the following steps may be performed:

[0144] P101. Establish a differentiable vector graphics rasterization module as a rasterization module;

[0145] P102. Establish CLIP module as text encoding module;

[0146] P103. Establish multiple cross-attention downsampling blocks, a downsampling block, a cross-attention intermediate block, an upsampling block, and multiple cross-attention upsampling blocks and fully connect them to obtain the Unet network, and use the Unet network as the image segmentation module.

[0147] like Figure 3 As shown, step P101 is executed to establish a differentiable vector graphics rasterization module diffvg as a rasterization module.

[0148] like Figure 3 As shown, step P102 is executed to establish a CLIP network as a text encoding module. Specifically, the text encoder in CLIP is used to perform the text encoding function of the text encoding module.

[0149] Execute step P103 to establish Figure 4 The Unet network shown is Figure 3 The image segmentation module in . Figure 4, the Unet network includes multiple cross-attention downsampling blocks, a downsampling block, a cross-attention intermediate block, an upsampling block, and multiple cross-attention upsampling blocks. Figure 4 , the cross-attention downsampling block is DownBlock2D unit, the cross-attention middle block is MidBlock2DCrossAttn unit, the upsampling block is UpBlock2D unit, and the cross-attention upsampling block is CrossAttnUpBlock2D unit, which are fully connected in sequence. Each CrossAttnDownBlock2D unit and each CrossAttnUpBlock2D unit have their own corresponding dimension sizes. For example, Figure 4 The CrossAttnDownBlock2D units on the left side have dimensions of 32×32×640, 16×16×1280, and 8×8×1280 from top to bottom, respectively, while the CrossAttnUpBlock2D units on the right side have the same dimensions as the CrossAttnDownBlock2D units in the same row, i.e., the CrossAttnUpBlock2D units on the right side have dimensions of 32×32×640, 16×16×1280, and 8×8×1280 from bottom to top, respectively.

[0150] Reference Figure 4 , the Unet network also includes an input convolution layer Conv_in and an output convolution layer Conv_out.

[0151] After executing step P1 to establish the text graph model, execute step P2 to perform at least one round of training process on the text graph model. The process of each round of training process is as follows: Figure 5 See Figure 1 and Figure 5 , when executing step P2, the following steps are performed during the i-th round of training:

[0152] P201. Get sample text sample-character i and the corresponding sample description text sample-text i ;

[0153] P202. Use the rasterization module to sample text sample-character i Rasterize to obtain the first sample image sample-image i ;

[0154] P203. For the first sample image sample-image iPerform encoding and noise processing to obtain the first noise feature map noised-feature i ; Among them, the noise processing is used to add real noise information real-noise i ;

[0155] P204. Sample description text sample-text i Input the text encoding module for processing to obtain the first text vector text-embedding i ;

[0156] P205. Text-embedding the first text vector i The input image segmentation module is used as the control condition for noise prediction processing, and the first noise feature map noised-feature i Input the image segmentation module to perform noise prediction processing to obtain the predicted noise information forecast-noise i ;

[0157] P206. Based on forecast-noise information i and real-noise information i Determine the loss function value loss i .

[0158] Specifically, the Wensheng graph model can be trained in one round or in multiple rounds, and the principle of each round of training is the same. For example, if the Wensheng graph model has been established and only one round of training is performed, steps P1 and P3 can be omitted. If the image segmentation module is not used for noise prediction, P204 in step P2 can also be omitted. The specific process of steps P1-P4 is as follows: Figure 6 As shown, including:

[0159] P201. Get sample text sample-character i ;

[0160] P202. Based on sample text sample-character i , get the first sample image sample-image i ;

[0161] P203. For the first sample image sample-image i Perform noise processing to obtain the first noise feature map noised-feature i ; Among them, the noise processing is used to add real noise information real-noisei ;

[0162] P205. Perform noise prediction processing on the first noise-added feature map noised-featurei i to obtain predicted noise information forecast-noise i ;

[0163] P206. Determine the loss function value loss i based on the predicted noise information forecast-noise i and the actual noise information real-noise i ;

[0164] P4. Update the parameters of the text-to-image generation model according to the loss function value loss i to realize the training of the text-to-image generation model based on the predicted noise information forecast-noise i and the actual noise information real-noise i to obtain the trained text-to-image generation model.

[0165] It should be noted that the above steps P206 and P4 can be to train the text-to-image generation model according to the predicted noise information and the actual noise information to obtain the trained text-to-image generation model.

[0166] In steps P201 - P206 and P4, by using the text-to-image generation model to learn the glyphs of the sample text and the semantic images of the sample description text, the trained text-to-image generation model has the performance of generating target text, and the generated target text has the same glyph as the text to be processed, and the visual effect presented by its font carries the semantic image information corresponding to the text to be processed.

[0167] In step P201, the sample text sample-character i can be obtained from the font library. i In this embodiment, the sample text sample-character i is a single character and can be represented in the encoding form of standards such as GB2312, GB18030, or Unicode. For example, if the sample text to be obtained during the i-th round of training is the Chinese character "天", then sample-character

[0168] In step P201, the corresponding sample description text sample-text i to the sample text sample-character i, is a text describing the sample text sample-character i The text of the semantics. For example, if the sample text sample-character i is the Unicode encoding "22825", that is, the Chinese character "天", then the sample description text sample-text i The content can be "a sky with clouds".

[0169] In this embodiment, when performing step P202, that is, rasterizing the sample text using the rasterization module to obtain the first sample image, the following steps can be specifically performed:

[0170] P20201. Use the font library to extract the contour information of the sample text sample-character i ;

[0171] P20202. Convert the contour information into a curve;

[0172] P20203. Determine a set of control points according to the curve;

[0173] P20204. Input the control points into the rasterization module diffvg for rasterization processing to obtain the first sample image samle-image i .

[0174] In step P20201, the contour information of the sample text sample-character can be extracted using a font library such as FreeType i , and then step P20202 is executed. Among them, the curve can be a Bezier curve. For example, the contour information is converted into a set of cubic Bezier curves. Of course, the curve can also be other types of planar curves.

[0175] In step P20203, multiple points can be taken on the curve obtained in step P20202 to obtain a set of control points.

[0176] In step P20204, the control points obtained in step P20203 are input into the rasterization module diffvg. The cubic Bezier curve represented by the control points is a vector graph, and the differentiable vector graphics rasterization module diffvg can rasterize the vector graph to convert it into a bitmap-form first sample image sample-image i , so as to meet the format requirements of the input data of the image segmentation module. In this embodiment, the size of the first sample image sample-image i can be 512×512.

[0177] After executing step P202, step P203 is then executed to convert the first sample image sample-image i Encoded into the form of a feature map, and adding noise. Specifically, in step P203, that is, the first sample image sample-image i Perform encoding and noise processing to obtain the first noise feature map noised-feature i At this step, you can perform the following steps:

[0178] P20301. For the first sample image samle-image i Encode and obtain the first feature map feature i ;

[0179] P20302. Using random functions to randomly generate real-noise information i ;

[0180] P20303.Real-noise information i Add to the first feature map feature i , get the first noise feature map noised-feature i .

[0181] When executing step P20301, the VAE encoder can be used to encode the first sample image sample-image i Encode the first sample image sample-image with a size of 512×512 i After being encoded by the VAE encoder, the first feature map feature with a size of 64×64×4 is obtained i .

[0182] In step P20302, a random function can be run to randomly generate actual noise information real-noise i Specifically, real-noise information in the form of salt and pepper noise can be generated. i . Real-noise information i The size of the first feature map feature i same.

[0183] In step P20303, the actual noise information real-noise i Add to the first feature map feature i Specifically, the actual noise information real-noise i Each pixel value in the first feature map featurei The corresponding pixel values ​​in are added to obtain the first noise feature map noised-feature i .

[0184] After executing step P203, step P204 is executed to convert the sample description text sample-text i The input text encoding module is processed. Specifically, the sample description text sample-text i Input CLIP module, the text encoder in CLIP module converts sample description text sample-text i Encoded into the first text vector text-embedding i .

[0185] By executing steps P201-P204, the first noise-feature map obtained is i and the first text vector text-embedding i If the input data format requirements of the image segmentation module are met, then step P205 is executed.

[0186] In step P205, the first text vector text-embedding i And the first noise feature map noised-feature i All are input into the image segmentation module. Among them, the first text vector text-embedding i As a control condition for the image segmentation module to perform noise prediction processing, the image segmentation module performs noised-feature i Perform noise prediction processing to obtain forecast noise information forecast-noise i .

[0187] The actual noise information real-noise obtained by executing step P20302 i , is actually added to the first noised feature map noised-feature i Noise information in; prediction noise information forecast-noise obtained by executing step P205 i , is the image segmentation module for the first noise feature map noised-feature i The prediction result of the noise information added in the ideal case is forecast-noise i Approaching the actual noise information real-noise added in step P20303 i .

[0188] By using the actual noise information and the predicted noise information, the noise recognition performance of the feature map when the image segmentation module is processed can be trained, so that the image segmentation module can perform noise prediction processing to obtain predicted noise information, and the predicted noise information can be integrated into the features of the first text vector, thereby guiding the image segmentation module to generate an image with the features of the first text vector. The actual noise information and the predicted noise information in this embodiment are the basis of the loss function used when executing step P206, that is, through the actual noise information and the predicted noise information, the image segmentation module can be trained by back propagation based on the loss function.

[0189] In this embodiment, when executing step P205, that is, inputting the first text vector into the image segmentation module as a control condition, inputting the first noisy feature map into the image segmentation module for noise prediction processing, and obtaining the predicted noise information, the following steps can be specifically performed:

[0190] P20501. Receive the first text vector text-embedding through multiple cross-attention downsampling blocks (CrossAttnDownBlock2D units) i , for the first noise feature map noised-feature i Perform feature integration to obtain the first low-dimensional feature information;

[0191] P20502. Processing the first low-dimensional feature information through a downsampling block (DownBlock2D unit) to obtain second low-dimensional feature information;

[0192] P20503. Process the second low-dimensional feature information through the cross-attention middle block (MidBlock2DCrossAttn unit) to obtain the third low-dimensional feature information;

[0193] P20504. Processing the third low-dimensional feature information through an upsampling block (UpBlock2D unit) to obtain the first high-dimensional feature information;

[0194] P20505. Receive the first text vector text-embedding through multiple cross-attention upsampling blocks (CrossAttnUpBlock2D units) i And the processing results of the cross-attention downsampling block (CrossAttnDownBlock2D unit) of the same dimension size, according to the first text vector text-embedding iThe first high-dimensional feature information is integrated with features, and features are added according to the processing results of the cross-attention downsampling block (CrossAttnUpBlock2D unit) to obtain the second high-dimensional feature information;

[0195] P20506. Determine the second high-dimensional feature information and the first noised feature map noised-feature i The difference between them is used to obtain the forecast noise information forecast-noise i .

[0196] In step P20501, refer to Figure 7 , the first noise feature map noised-feature i After being input into the image segmentation module, the first noised feature map with a size of 64×64×4 is i First, it is processed by an input convolution layer Conv_in in the image segmentation module to obtain a feature with a dimension of 64×64×320, and then it is sequentially processed by three layers of cross-attention downsampling blocks (CrossAttnDownBlock2D units) to obtain the first low-dimensional feature information of the deep layer. Figure 7 , the dimension of the first low-dimensional feature information is 8×8×1280.

[0197] Specifically, in this embodiment, the structure of the cross-attention downsampling block (CrossAttnDownBlock2D unit) used is as follows: Figure 8 As shown, refer to Figure 8 , a CrossAttnDownBlock2D unit consists of two 2D residual blocks, two 2D transformer models, and a 2D downsampling layer.

[0198] Among them, the 2D residual block is ResnetBlock2D, ResnetBlock2D can introduce residual connections, so that the gradient can be directly transferred through the residual connection, thereby effectively alleviating the gradient disappearance problem; the 2D transformer model is Transformer2DModel, Transformer2DModel can implement the self-attention mechanism, so as to deeply understand the context information of the text; the 2D downsampling layer is Downsample2D, Downsample2D can reduce the dimension of the data, thereby reducing the computational burden of the model and accelerating the reasoning process.

[0199] The structure of the 2D residual block (ResnetBlock2D) is as follows Fig. 9 As shown, refer to Fig. 9, a 2D residual block (ResnetBlock2D) consists of two 3×3 grouped convolutions; the structure of the 2D transformer model (Transformer2DModel) is as follows Fig.10 As shown, refer to Fig.10 , a 2D transformer model (Transformer2DModel) consists of 1 grouped convolution, 2 transformer blocks, and 2 fully connected, and in the transformer block, the first text vector text-embedding is performed by cross attention. i The 2D downsampling layer (Dowmsample2D) is a convolution layer with a convolution kernel of 3×3 and a stride of 2.

[0200] Since the transformer block is set in the CrossAttnDownBlock2D unit, the CrossAttnDownBlock2D unit can be integrated into the first text vector text-embedding in the feature extraction process by cross-attention. i The first low-dimensional feature information obtained by executing step P20501 contains the first text vector text-embedding i feature information.

[0201] Reference Figure 7 After executing step P20501 to obtain the first low-dimensional feature information, step P20502 is then executed. In step P20502, the first low-dimensional feature information is further extracted through a downsampling block (DownBlock2D unit).

[0202] In this embodiment, Figure 7 The downsampling block (DownBlock2D) in Fig. 1 consists of two convolutional layers with a convolution kernel size of 3×3, where the convolution kernel of the second convolutional layer is 3×3 and the stride is 2.

[0203] In step P20502, the downsampling block (DownBlock2D unit) downsamples the first low-dimensional feature information to obtain second low-dimensional feature information. Figure 7 , the dimension of the second low-dimensional feature information is 8×8×1280.

[0204] Reference Figure 7After executing step P20502 to obtain the second low-dimensional feature information, step P20503 is then executed. In step P20503, the cross-attention intermediate block (MidBlock2DCrossAttn unit) is used to further extract features. Among them, MidBlock2DCrossAttn is as follows Fig.11 As shown, a MidBlock2DCrossAttn consists of 2 ResnetBlock2D and 1 Transformer2DModel.

[0205] In step P20503, the cross-attention middle block (MidBlock2DCrossAttn unit) outputs the third low-dimensional feature information.

[0206] In step P20504, refer to Figure 7 , the third low-dimensional feature information extracted by the cross-attention middle block (MidBlock2DCrossAttn unit) is input into the upsampling block (UpBlock2D unit) for feature extraction. Among them, UpBlock2D consists of a 3×3 convolution layer and a deconvolution layer with a stride of 2.

[0207] In step P20504, the upsampling block (UpBlock2D unit) upsamples the third low-dimensional feature information to obtain the first high-dimensional feature information.

[0208] In step P20505, refer to Figure 7 , the first high-dimensional feature information extracted by the upsampling block (UpBlock2D unit) is input into the 3-layer cross-attention upsampling block (CrossAttnUpBlock2D unit) for feature extraction.

[0209] In this embodiment, the structure of the cross-attention upsampling block (CrossAttnUpBlock2D unit) is as follows Fig.12 See Fig.12 , since the cross-attention up-sampling block (CrossAttnUpBlock2D unit) is the same as the cross-attention down-sampling block (CrossAttnDownBlock2D unit), it also has Fig.10 The 2D transformer model (Transformer2DModel) shown in Figure 1 is configured so that the CrossAttnUpBlock2D unit can be incorporated into the first text vector text-embedding in the feature extraction process by cross-attention. i Moreover, refer to Figure 7When performing feature extraction, each cross-attention upsampling block (CrossAttnUpBlock2D unit) obtains the processing results of the cross-attention downsampling block (CrossAttnDownBlock2D unit) of the same dimension (same row), adds it to the extracted features and outputs it. After being processed by multiple cross-attention upsampling blocks (CrossAttnUpBlock2D units), the scale of the feature map is improved. During this period, the deep features and shallow features of the same dimension are added, and features of different scales can be fused. The final second-highest dimensional feature information can be processed by an output convolution layer Conv_out and become the same as the first noisy feature map noised-feature. i The second highest dimensional feature information obtained is the noised-feature map obtained by Unet based on the first noised-feature map. i Therefore, in step P20506, by calculating the high-dimensional feature information and the first noise feature map noised-feature i The difference between them can be used to obtain the predicted noise information feature-noise i .

[0210] In this embodiment, by executing steps P20501-P20506, the first text vector text-embedding i And the first noise feature map noised-feature i The image segmentation module is input to perform noise prediction processing, so that the image segmentation module performs noise prediction processing to obtain the predicted noise information forecast-noise i is incorporated into the first text vector text-embedding i The features of the image segmentation module guide the image segmentation module to generate the first text vector text-embedding i The features of the image.

[0211] In step P206, the forecast noise information obtained in step P205 is converted into forecast-noise i The actual noise information real-noise added in step P20303 i Substitute it into the loss function for calculation and obtain the loss function value loss corresponding to the i-th round of training process i .

[0212] In this embodiment, when executing step P206, the loss function used is

[0213] loss i =(b i -ai )z i

[0214] where loss i is the loss function value corresponding to the i-th round of training process, and b i is the predicted noise information forecast-noise obtained in the i-th round of training process i , and a i is the actual noise information real-noise obtained in the i-th round of training process i , and z i is the first noise-added feature map noised-feature obtained in the i-th round of training process i .

[0215] Therefore, the calculation formula for the loss function value loss i obtained in the i-th round of training process can also be expressed as:

[0216] loss i =(forecast-noise i -real-noise i )×noise-feature i

[0217] When executing step P2, each round of training process will execute a round of steps P201 - P206, setting the corresponding sample character sample-character i , sample description text sample-text i and actual noise information real-noise i , and finally obtaining the loss function value loss i of this round of training process. Among them, the sample characters sample-characteri i and sample description text sample-text i in different rounds of training processes are different. For example, in one round of training process, the sample character used is the Chinese character "sky", and the sample description text is "sky with clouds", then in the next round of training process, the sample character used can be changed to "tree", and the corresponding sample description text can be changed to "woody plant with many leaves"; and since the actual noise information real-noise i is randomly generated, the actual noise information real-noise i in different rounds of training processes is generally different.

[0218] If a total of N rounds of training processes are executed in step P2, then in step P3, according to the loss function values lossi (i=1, 2, 3...N) calculate and obtain the cumulative loss function value LOSS.

[0219] After executing step P3 to obtain the cumulative loss function value LOSS, the parameters of the text graph model established in step P1 are updated according to the cumulative loss function value LOSS, so that the text graph model learns the sample text sample-character i With sample description text sample-text i Described sample text sample-character i The feature correspondence between the semantic image information.

[0220] Specifically, the calculation formula of the cumulative loss function value LOSS in this embodiment is:

[0221]

[0222] Among them, loss i is the loss function value corresponding to the i-th round of training process, LOSS is the cumulative loss function value, and loss i is the loss function value corresponding to the i-th round of training process, and N is the total number of training processes that have been executed. That is, the sum of the loss function values ​​of all the executed training processes can be used as the cumulative loss function value LOSS.

[0223] After executing steps P1-P3 to obtain the cumulative loss function value LOSS, step P4 is then executed to update the parameters of the Wensheng graph model according to the cumulative loss function value LOSS.

[0224] In this embodiment, when executing step P4, that is, the step of updating the parameters of the Wensheng graph model according to the cumulative loss function value, the following steps may be specifically performed:

[0225] P401. Based on the accumulated loss function value LOSS, back-propagate and update the parameters of at least one module among the rasterization module, the image segmentation module and the text encoding module.

[0226] Since the Wensheng graph model used in this embodiment includes components such as a rasterization module, an image segmentation module, and a text encoding module, at least one of the modules can be selected to back propagate and update its parameters. For example, the parameters of the rasterization module, the image segmentation module, and the text encoding module can all be back propagated and updated according to the cumulative loss function value LOSS; the parameters of the rasterization module can also be frozen, that is, the parameters of the rasterization module are kept unchanged, and then the parameters of the image segmentation module and the text encoding module are back propagated (BP) and updated according to the cumulative loss function value LOSS.

[0227] Specifically, when executing step P401, if multiple modules are selected for back propagation update, the back propagation update process for different modules can be independent of each other. Taking the back propagation update of the image segmentation module as an example, the gradient of the cumulative loss function value LOSS for each parameter of each layer in the image segmentation module can be calculated, and the gradient is propagated layer by layer from the output layer to the input layer of the image segmentation module.

[0228] In this embodiment, when executing step P401, that is, the step of back-propagating and updating the parameters of at least one of the rasterization module, the image segmentation module, and the text encoding module based on the cumulative loss function value, the following steps may be specifically performed:

[0229] P40101. Freeze the parameters of the image segmentation module and the text encoding module;

[0230] P40102. Based on the accumulated loss function value, back-propagate and update the parameters of the rasterization module.

[0231] By executing steps P40101-P40102, the parameters of the image segmentation module (Unet network) and the text encoding module (ie, CLIP module) can be frozen, and only the parameters of the rasterization module (ie, diffvg) are back-propagated and updated.

[0232] The principle of executing steps P40101-P40102 is that the task of the Vincent graph model training method in this embodiment is to train a Vincent graph model that can generate text with semantic image information, wherein the image segmentation module and the text encoding module (i.e., CLIP module) mainly play the role of the general "Vencent graph" function, and the rasterization module (i.e., diffvg) can make the "Vencent graph" have semantic image information. Therefore, the parameters of the image segmentation module and the text encoding module do not need to be updated, and only the parameters of the rasterization module (i.e., diffvg) are updated, thereby improving the training efficiency of the Vincent graph model.

[0233] By executing steps P1-P4, a trained text-image model can be obtained, and the trained text-image model can automatically and quickly generate text with semantic image information.

[0234] In this embodiment, the text-generating method with semantic image information can be executed by using the text-generating image model trained through steps P1-P4. Fig.13 ,The text generation method with semantic image information includes the following steps:

[0235] S1. Obtaining the Wensheng graph model;

[0236] S2. Obtain the pending-character to be processed and the corresponding pending-text to be processed;

[0237] S3. Use the text-to-image model to process the pending-character and the pending-text to be processed, and generate the target character.

[0238] In step S1, obtain the text-to-image model trained through steps P1 - P4, which includes components such as a rasterization module (diffvg), an image segmentation module (Unet network), and a text encoding module (CLIP module).

[0239] In step S2, the pending-character to be processed represents the meaning of the character to be generated. For example, if it is desired to generate the Chinese character "bird" with semantic image information, then the pending-character to be processed is set to the Unicode encoding 40479 corresponding to the Chinese character "bird".

[0240] In step S2, the pending-text to be processed is a text that describes the semantic content of the pending-character to be processed. For example, the content of the pending-text to be processed can be "a little bird singing".

[0241] Step S3 specifically includes the following steps:

[0242] S301. Rasterize the pending-character to be processed to obtain a rasterized image image;

[0243] S302. Perform multiple rounds of iterative processing on the rasterized image image and the pending-text to be processed, and generate the target character generated-character according to the processing result of the last round of iterative processing.

[0244] In step S301, refer to Fig.14 , use the rasterization module (diffvg) to rasterize the pending-character to be processed to obtain a rasterized image image. Specifically, the contour information of the pending-character to be processed can be extracted from a font library such as FreeType, then the contour information is converted into a set of cubic Bezier curves, and then the control points are collected, and finally the control points are input into diffvg for rasterization to obtain the rasterized image image.

[0245] In step S302, refer to Fig.14Specifically, only one round of iterative processing may be performed, or multiple rounds of iterative processing may be performed. The principle of each round of iterative processing is the same. In this embodiment, the principle of one round of iterative processing is used to illustrate.

[0246] Reference Fig.14 When executing step S302, the following steps are performed during the j-th round of iterative processing:

[0247] S30201. Encode and denoise the information to be processed to obtain the second noisy feature map noised-feature′ in this round of iterative processing j ;

[0248] S30202. Input the pending-text description text into the text encoding module for processing to obtain a second text vector text-embedding′;

[0249] S30203. The second text vector text-embedding′ j The input image segmentation module is used as the control condition for the noise reduction process, and the second noise feature map noised-feature′ j The input image segmentation module performs noise reduction processing to obtain the processing result of this round of iterative processing.

[0250] Reference Fig.14 If j=1, that is, in the first round of iterative processing, the information to be processed refers to the rasterized image image, and in each round of iterative processing after the first round of iterative processing, the information to be processed refers to the processing result of the previous round of iterative processing.

[0251] In step S30201, a VAE encoder can be used to encode the information to be processed, and actual noise information real-noise′ can be added to the obtained feature map. j , thus obtaining the second noise feature map noised-feature′ in this round of iterative processing j .

[0252] In step S30202, the pending-text description text is input into the text encoder in the CLIP module for processing to obtain a second text vector text-embedding′.

[0253] In step S30203, the second text vector text-embedding′ j The input image segmentation module is used as the control condition for the noise reduction process, and the second noise feature map noised-feature′j The input image segmentation module performs noise reduction processing to obtain the processing result of this round of iterative processing.

[0254] After the j-th round of iterative processing is completed, it is judged whether j reaches the threshold of the number of rounds to be executed. If the current j is less than the threshold of the number of rounds, then j = j + 1, and the next round of iterative processing is continued; if the current j is equal to the threshold of the number of rounds, then the just-executed j-th round of iterative processing is the last round of iterative processing, and the processing result of the last round of iterative processing is obtained to generate the target text generated-character.

[0255] Generally, this threshold of the number of rounds can be set to 20 or more, for example, set to 50, that is, a total of more than 20 rounds of iterative processing are performed, and it can be selected to perform a total of 50 rounds of iterative processing.

[0256] In this embodiment, the processing result of the last round of iterative processing is feature information, which can be mapped to the pixel space to obtain the target text generated-character.

[0257] In this embodiment, if the text to be processed pending-character is the Chinese character "bird" (specifically, the Unicode code 40479 corresponding to the character "bird"), then the display effect of the text to be processed pending-character is as Fig.15 shown. By executing steps S1 - S2 and S301 - S302, the result of the target text generated-character output by the text-to-image model is as Fig.16 shown. It can be seen from Fig.16 that the generated target text generated-character is also a "bird" character in terms of visual effect, and there is a little bird pattern in the font configuration of the target text generated-character, that is, it carries the semantic image information of the character "bird", thus realizing the generation of text with semantic image information.

[0258] In this embodiment, a text-to-image model training device is provided. Referring to Fig.17 , the text-to-image model training device includes:

[0259] A text acquisition module, configured to acquire sample text and corresponding sample description text;

[0260] An image acquisition module, configured to obtain a first sample image according to the sample text;

[0261] A noise adding module, used for performing noise adding processing on the first sample image to obtain a first noise adding feature map; wherein the noise adding processing is used for adding actual noise information;

[0262] A prediction module, used for performing noise prediction processing on the first noise-added feature map to obtain predicted noise information;

[0263] The training module is used to train the Vincent graph model according to the predicted noise information and the actual noise information to obtain a trained Vincent graph model.

[0264] In an optional implementation, the method for obtaining a first sample image according to the sample text includes:

[0265] Rasterizing the sample text to obtain a first sample image;

[0266] The text image model includes: a rasterization module; the step of rasterizing the sample text to obtain a first sample image includes:

[0267] Extracting outline information of the sample text using a font library;

[0268] converting the contour information into a curve;

[0269] determining a set of control points according to the curve;

[0270] The control points are input into the rasterization module for rasterization processing to obtain the first sample image.

[0271] In an optional implementation, performing noise processing on the first sample image to obtain a first noise feature map includes:

[0272] Encoding the first sample image to obtain a first feature map;

[0273] The actual noise information is randomly generated using a random function;

[0274] The actual noise information is added to the first feature map to obtain the first noisy feature map.

[0275] In an optional implementation, performing noise prediction processing on the first noisy feature map based on the sample description text to obtain predicted noise information includes:

[0276] Get the sample description text corresponding to the sample text;

[0277] Inputting the sample description text into the text encoding module for processing to obtain a first text vector;

[0278] The first text vector is input into the image segmentation module as a control condition for noise prediction processing, and the first noisy feature map is input into the image segmentation module for noise prediction processing to obtain predicted noise information.

[0279] In an optional embodiment, the image segmentation module includes a plurality of cross-attention downsampling blocks, a downsampling block, a cross-attention intermediate block, an upsampling block and a plurality of cross-attention upsampling blocks that are fully connected in sequence; wherein each of the cross-attention downsampling blocks and each of the cross-attention intermediate blocks have their own corresponding dimensional sizes.

[0280] In an optional implementation, the step of inputting the first text vector into the image segmentation module as a control condition for noise prediction processing, inputting the first noisy feature map into the image segmentation module for noise prediction processing, and obtaining predicted noise information includes:

[0281] The first text vector is received through a plurality of the cross-attention downsampling blocks, and features are integrated into the first noisy feature map to obtain first low-dimensional feature information;

[0282] Processing the first low-dimensional feature information through the downsampling block to obtain second low-dimensional feature information;

[0283] Processing the second low-dimensional feature information through the cross-attention intermediate block to obtain third low-dimensional feature information;

[0284] Processing the third low-dimensional feature information by the upsampling block to obtain first high-dimensional feature information;

[0285] Receiving the first text vector and the processing results of the cross-attention down-sampling blocks of the same dimension through a plurality of the cross-attention up-sampling blocks, integrating the first high-dimensional feature information according to the first text vector, and adding the features according to the processing results of the cross-attention down-sampling blocks to obtain second high-dimensional feature information;

[0286] Determine the difference between the second high-dimensional feature information and the first noisy feature map to obtain the predicted noise information.

[0287] In an optional implementation, the training of the Vincent graph model according to the predicted noise information and the actual noise information to obtain the trained Vincent graph model includes:

[0288] Perform at least one round of training process; each round of the training process obtains corresponding predicted noise information and actual noise information;

[0289] For any round of the training process, determining a loss function value according to the predicted noise information and the actual noise information;

[0290] Determine the cumulative loss function value according to the loss function value of each round of the training process;

[0291] Parameters of the Wensheng graph model are updated according to the cumulative loss function value to obtain a trained Wensheng graph model.

[0292] In an optional implementation, determining the loss function value according to the predicted noise information and the actual noise information includes:

[0293] Calculate the loss function value, wherein the calculation formula of the loss function value is:

[0294] loss i =(b i -a i ) i

[0295] Among them, loss i is the loss function value corresponding to the i-th round of training process, b i is the predicted noise information obtained in the i-th round of the training process, a i is the actual noise information obtained in the i-th round of training process, z i The first noisy feature map obtained in the i-th round of the training process.

[0296] In an optional implementation, determining the cumulative loss function value according to the loss function value of each round of the training process includes:

[0297] Calculate the cumulative loss function value, where the calculation formula of the cumulative loss function value is:

[0298]

[0299] Among them, loss i is the loss function value corresponding to the i-th round of the training process, LOSS is the cumulative loss function value, and loss i is the loss function value corresponding to the i-th round of the training process, and N is the total number of rounds of the training process that have been executed.

[0300] In an optional implementation, updating the parameters of the Wensheng graph model according to the cumulative loss function value includes:

[0301] According to the accumulated loss function value, back-propagation update is performed on the parameters of at least one module among the rasterization module, the image segmentation module and the text encoding module.

[0302] In an optional implementation, the back-propagation updating of the parameters of at least one of the rasterization module, the image segmentation module and the text encoding module according to the accumulated loss function value comprises:

[0303] Freezing parameters of the image segmentation module and the text encoding module;

[0304] According to the accumulated loss function value, back-propagation is performed on the parameters of the rasterization module to update them.

[0305] By running the text graph model training device, the same technical effect as the text graph model training method in the embodiment can be achieved.

[0306] This embodiment provides a text generation device with semantic image information. Fig.18 , the text generation device with semantic image information includes:

[0307] A Wensheng graph module, used to obtain a Wensheng graph model; the Wensheng graph model is trained by a Wensheng graph model training method;

[0308] An acquisition module, used to acquire the text to be processed and the corresponding description text to be processed;

[0309] A processing module is used to use the text-generated graph model to process the text to be processed and the description text to be processed to generate target text.

[0310] By running the text generation device with semantic image information, the same technical effect as the text generation method with semantic image information in the embodiment can be achieved.

[0311] In this embodiment, the computer is programmed to execute Figure 1 and storing such a computer program in a Fig.19 The memory of the computer device shown, or Fig. 20 The computer program stored in the memory or computer-readable storage medium may be Fig.19 The processor in Fig. 20 The processor in the processor reads it out and obtains the execution Figure 1 The processor executes the instructions of each step in the text-generated graph model training method or the text generation method with semantic image information.

[0312] because Fig.19 The computer device and Fig. 20The computer-readable storage media shown can enable the processor to execute the text-image model training method or the text generation method with semantic image information, so Fig.19 The computer device and Fig. 20 The computer-readable storage media shown can achieve the same technical effects as the text-graph model training method or the text generation method with semantic image information in the embodiment.

[0313] It should be noted that, unless otherwise specified, the singular forms of "a" and "the" used in this disclosure are also intended to include plural forms, unless the context clearly indicates other meanings. In addition, unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as those generally understood by those skilled in the art. The terms used in the present embodiment specification are only for describing specific embodiments, not for limiting the present invention. The term "and / or" used in this embodiment includes any combination of one or more related listed items.

[0314] It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided by the present embodiment is only intended to better illustrate embodiments of the present invention, and unless otherwise required, the scope of the present invention will not be limited.

[0315] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed dedicated integrated circuit for this purpose.

[0316] In addition, the operations of the process described in this embodiment may be performed in any suitable order, unless otherwise indicated in this embodiment or otherwise clearly contradicted by the context. The process described in this embodiment (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed on one or more processors in common, by hardware or a combination thereof. A computer program includes a plurality of instructions that may be executed by one or more processors.

[0317] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or part thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the above steps in conjunction with a microprocessor or other data processor, the invention of this embodiment includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0318] The above are only preferred embodiments of the present invention. The present invention is not limited to the above embodiments. As long as the technical effects of the present invention are achieved by the same means, any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical solutions and / or implementation methods may have various modifications and changes.

Claims

1. A method for training a text graph model, characterized in that: The method for training the culture graph model comprises: Get sample text; Obtaining a first sample image according to the sample text; Performing noise addition processing on the first sample image to obtain a first noise addition feature map; wherein the noise addition processing is used to add actual noise information; Performing noise prediction processing on the first noisy feature map to obtain predicted noise information; The Vincent graph model is trained according to the predicted noise information and the actual noise information to obtain a trained Vincent graph model.

2. The method for training a text graph model according to claim 1, characterized in that: The step of obtaining a first sample image according to the sample text includes: Rasterizing the sample text to obtain a first sample image; The text image model includes: a rasterization module; the step of rasterizing the sample text to obtain a first sample image includes: Extracting outline information of the sample text using a font library; converting the contour information into a curve; determining a set of control points according to the curve; The control points are input into the rasterization module for rasterization processing to obtain the first sample image.

3. The method for training a text graph model according to claim 2, characterized in that: The step of performing noise processing on the first sample image to obtain a first noise feature map includes: Encoding the first sample image to obtain a first feature map; The actual noise information is randomly generated using a random function; The actual noise information is added to the first feature map to obtain the first noisy feature map.

4. The method for training a text graph model according to claim 1, characterized in that: The text graph model includes: a text encoding module and an image segmentation module; The performing noise prediction processing on the first noise-added feature map to obtain predicted noise information includes: Get the sample description text corresponding to the sample text; Inputting the sample description text into the text encoding module for processing to obtain a first text vector; The first text vector is input into the image segmentation module as a control condition for noise prediction processing, and the first noisy feature map is input into the image segmentation module for noise prediction processing to obtain predicted noise information.

5. The method for training a text graph model according to claim 4, characterized in that: The image segmentation module includes multiple cross-attention downsampling blocks, a downsampling block, a cross-attention intermediate block, an upsampling block and multiple cross-attention upsampling blocks that are fully connected in sequence; wherein each of the cross-attention downsampling blocks and each of the cross-attention intermediate blocks have their own corresponding dimensional sizes.

6. The method for training a text graph model according to claim 5, characterized in that: The step of inputting the first text vector into the image segmentation module as a control condition for noise prediction processing, inputting the first noise feature map into the image segmentation module for noise prediction processing, and obtaining predicted noise information includes: The first text vector is received through a plurality of the cross-attention downsampling blocks, and features are integrated into the first noisy feature map to obtain first low-dimensional feature information; Processing the first low-dimensional feature information through the downsampling block to obtain second low-dimensional feature information; Processing the second low-dimensional feature information through the cross-attention intermediate block to obtain third low-dimensional feature information; Processing the third low-dimensional feature information by the upsampling block to obtain first high-dimensional feature information; Receiving the first text vector and the processing results of the cross-attention down-sampling blocks of the same dimension through a plurality of the cross-attention up-sampling blocks, integrating the first high-dimensional feature information according to the first text vector, and adding the features according to the processing results of the cross-attention down-sampling blocks to obtain second high-dimensional feature information; Determine the difference between the second high-dimensional feature information and the first noisy feature map to obtain the predicted noise information.

7. The method for training a text graph model according to any one of claims 2 to 6, characterized in that: The step of training the Vincent graph model according to the predicted noise information and the actual noise information to obtain the trained Vincent graph model includes: Perform at least one round of training process; each round of the training process obtains corresponding predicted noise information and actual noise information; For any round of the training process, determining a loss function value according to the predicted noise information and the actual noise information; Determine the cumulative loss function value according to the loss function value of each round of the training process; Parameters of the Wensheng graph model are updated according to the cumulative loss function value to obtain a trained Wensheng graph model.

8. The method for training a text graph model according to claim 7, characterized in that: The determining of the loss function value according to the predicted noise information and the actual noise information comprises: Calculate the loss function value, wherein the calculation formula of the loss function value is: loss i =(b i -a i )z i Among them, loss i is the loss function value corresponding to the i-th round of training process, b i is the predicted noise information obtained in the i-th round of the training process, a i is the actual noise information obtained in the i-th round of training process, z i The first noisy feature map obtained in the i-th round of the training process.

9. The method for training a text graph model according to claim 7, characterized in that: Determining the cumulative loss function value according to the loss function value of each round of the training process includes: Calculate the cumulative loss function value, where the calculation formula of the cumulative loss function value is: Among them, loss i is the loss function value corresponding to the i-th round of the training process, LOSS is the cumulative loss function value, and loss i is the loss function value corresponding to the i-th round of the training process, and N is the total number of rounds of the training process that have been executed.

10. The method for training a text graph model according to claim 7, characterized in that: The updating of parameters of the Wensheng graph model according to the cumulative loss function value includes: According to the accumulated loss function value, back-propagation update is performed on the parameters of at least one module among the rasterization module, the image segmentation module and the text encoding module.

11. The method for training a text graph model according to claim 8, characterized in that: The back-propagation updating of the parameters of at least one of the rasterization module, the image segmentation module and the text encoding module according to the accumulated loss function value comprises: Freezing parameters of the image segmentation module and the text encoding module; According to the accumulated loss function value, back-propagation is performed on the parameters of the rasterization module to update them.

12. A method for generating text with semantic image information, characterized in that: The method for generating text with semantic image information comprises: Acquire a Wensheng graph model; the Wensheng graph model is trained by the Wensheng graph model training method according to any one of claims 1 to 11; Get the text to be processed and the corresponding description text to be processed; The text to be processed and the description text to be processed are processed using the text graph model to generate target text.

13. The method for generating text with semantic image information according to claim 12, characterized in that: The step of using the text graph model to process the text to be processed and the description text to be processed to generate target text includes: The processing result of the text-generated graph model is mapped to the pixel space to obtain the target text.

14. A device for training a text graph model, characterized in that: The Wensheng graph model training device comprises: A text acquisition module is used to acquire sample text; An image acquisition module, used for acquiring a first sample image according to the sample text; A noise adding module, used for performing noise adding processing on the first sample image to obtain a first noise adding feature map; wherein the noise adding processing is used for adding actual noise information; A prediction module, used for performing noise prediction processing on the first noise-added feature map to obtain predicted noise information; The training module is used to train the Vincent graph model according to the predicted noise information and the actual noise information to obtain a trained Vincent graph model.

15. A text generation device with semantic image information, characterized in that: The text generation device with semantic image information comprises: A Wensheng graph module, used to obtain a Wensheng graph model; the Wensheng graph model is trained by the Wensheng graph model training method according to any one of claims 1 to 11; An acquisition module, used to acquire the text to be processed and the corresponding description text to be processed; A processing module is used to use the text-generated graph model to process the text to be processed and the description text to be processed to generate target text.

Images