Character generation method, character generation model training method and related device

Abstract

The character generation method comprises the following steps: acquiring a content template image, a style reference image, a content template edge image and a noise image; and generating a target image of the target character based on the noise image under the guidance of the content template image, the style reference image and the content template edge image. Wherein the content template image is used for indicating the content of the target character, the style reference image is used for indicating the style of the target character, and the content template edge image is used for indicating the geometric structure information of the target character. In the embodiment of the invention, the guidance of the edge image of the content template is introduced in the process of generating the target image, so that the geometric structure information of handwritten characters such as font contours and the like is explicitly concerned and utilized, and the font structure of the generated target image is complete and the style control is accurate when the diffusion model is applied to a handwritten font generation task.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to text generation methods, text generation model training methods, and related devices. Background Technology

[0002] In the field of artificial intelligence technology, handwritten font generation technology is a type of Artificial Intelligence Generated Content (AIGC) technology, suitable for applications such as personalized font design and augmenting training samples for handwritten character recognition models. However, due to the vast variety of handwritten characters and writing styles, generating high-quality handwritten fonts with complete character structures and accurate style control remains an unsolved problem. Summary of the Invention

[0003] To address the aforementioned issues, embodiments of this application provide a text generation method, a text generation model training method, and related apparatus, which to a certain extent achieve controllable generation of high-quality, controllable handwritten fonts.

[0004] Therefore, the following technical solutions are adopted in the embodiments of this application:

[0005] In a first aspect, embodiments of this application provide a text generation method, mainly including the following steps: acquiring a content template image, a style reference image, a content template edge image, and a noise image; and generating a target image of the target text based on the noise image, guided by the content template image, the style reference image, and the content template edge image. The content template image indicates the content of the target text, the style reference image indicates the style of the target text, and the content template edge image indicates the geometric structure information of the target text.

[0006] In this embodiment, guided by the content template image, a target image is generated based on a noise image, ensuring that the text content of the target image matches the text content of the content template image, thereby controlling the text content of the target image. Guided by the style reference image, a target image is also generated based on a noise image, ensuring that the text style of the target image matches the text style of the style reference image, thereby controlling the text style of the target image. Furthermore, guided by the content template edge image, a target image is generated based on a noise image, ensuring that the text outlines and other geometric structural information of the target image match the content template edge image, thereby controlling the text geometric structural information of the target image. Because the content template edge image is introduced during the generation of the target image, the geometric structural information of handwritten characters, such as the character outlines, is explicitly considered and utilized. This ensures that when the diffusion model is applied to the handwritten font generation task, the generated target image has a complete character structure and accurate style control. In other words, it solves the problem of incomplete character structure in the target image and the problem of inaccurate style control caused by incomplete character structure, such as the loss or error of a certain part of the character structure, leading to incorrect style control of that part of the character. Furthermore, the text generation method proposed in this application can be applied to the generation of handwritten fonts, and can also be used in other text generation tasks, such as artistic font generation, printed font generation, etc., and this application does not impose any limitations on this. Additionally, the text generation method proposed in this application can be applied to the generation of any language text; for example, the target text language can be, but is not limited to, Chinese, English, Japanese, etc.

[0007] In one possible implementation, the above-mentioned generation of a target image of the target text based on a noise image, guided by a content template image, a style reference image, and a content template edge image, includes: in the process of generating the target image based on the noise image, controlling the content of the target text based on the content template image; controlling the geometric structure information of the target text based on the content template edge image; and controlling the style of the target text based on the style reference image.

[0008] In this implementation, for example, the guidance process can be achieved by introducing a specially designed neural network layer into the image encoder and image decoder of the neural network-based diffusion model. The input of this neural network layer is the output of the previous neural network layer; the output of this neural network layer is the input of the next neural network layer. That is, the embodiments of this application do not limit the position of the neural network layer. For example, the neural network layer corresponding to the guidance process of the content template edge image can be upstream or downstream of the neural network layers corresponding to the guidance processes of the style reference image and the content template edge image. In other words, the content template image, style reference image, and content template edge image can serve as control conditions for the diffusion model-based text generation model to obtain a target image that meets the control condition requirements.

[0009] In one possible implementation, the above-mentioned method of controlling the geometric structure information of the target text based on the content template edge image includes: using the content template edge image as input to a contour encoder and outputting contour features; obtaining a first output feature map based on the contour features and the first input feature map through a cross-attention mechanism; and controlling the geometric structure information of the target text based on the first output feature map.

[0010] In this implementation, based on contour features and the first input feature map, a first output feature map is obtained through a cross-attention mechanism. By controlling the contour features, the integrity of the geometric structure information of the characters in the target image, such as the outline, edges, and glyphs, is ensured. This solves the problem of incomplete glyph structure in the generated target image and the inaccurate style control caused by incomplete glyph structure. In other words, to address the problem of unstable font structure in the generated target image, this application proposes to introduce a specially designed neural network layer (such as a content geometry control layer) into the image encoder and image decoder of the diffusion model, using the content template edge image to control the stability of the font structure in the target image.

[0011] In one possible implementation, the first input feature map based on the contour features and the input is used to obtain the first output feature map through a cross-attention mechanism, which includes: using the first input feature map as a query vector and the contour features as a key vector and a value vector, and calculating the first output feature map through cross-attention.

[0012] In this implementation, cross-attention is a method that updates the value vector based on the query vector and the key vector. In other words, the first output feature map is obtained by weighted summation of the value vectors, with the weights representing the similarity between the query vector and the key vector. Through the cross-attention mechanism, the first input feature map can be updated using contour features to obtain the first output feature map, thus achieving control of the edge contours of text in the target image using contour features.

[0013] In one possible implementation, the above-described method of controlling the content of target text based on a content template image includes: using the content template image as input to a content encoder to output a content template feature map; obtaining pixel offsets and attention coefficients based on the content template feature map and the input second feature map; processing the content template feature map based on the pixel offsets and attention coefficients to obtain an aggregated feature map; obtaining a second output feature map based on the aggregated feature map and the input second input feature map; and controlling the content of the target text based on the second output feature map.

[0014] In this implementation, pixel offsets and attention coefficients are obtained based on the content template feature map and the input feature map. The content template feature map is then processed using these offsets and attention coefficients to obtain an aggregated feature map. Finally, the output feature map is obtained based on the aggregated feature map and the input feature map. Therefore, by using pixel offsets and attention coefficients, the content template feature map can be aligned with the target text in the target image, reducing the risk of incomplete character shapes in the target text. In other words, to address the issue of imprecise control caused by the difference in character shapes between the content template image and the target image's handwritten font, unlike existing methods that directly concatenate the input feature map with the content template feature map, this application proposes introducing a specially designed neural network layer (such as a deformable content aggregation layer) into the diffusion model's image encoder. This layer adaptively aggregates the character shape information from the content template by deforming the content template feature map.

[0015] In one possible implementation, the pixel offsets include m horizontal pixel offsets and m vertical pixel offsets, and the number of attention coefficients is m, where m is an integer greater than or equal to 1. The above-described processing of the content template feature map based on the pixel offsets and attention coefficients to obtain the aggregated feature map includes: performing bilinear sampling on the content template feature map based on the m horizontal and m vertical pixel offsets to obtain m deformed content template feature maps; and performing a weighted summation on the m deformed content template feature maps based on the m attention coefficients to obtain the aggregated feature map.

[0016] In this implementation, the content template feature map is bilinearly sampled based on m horizontal and m vertical pixel offsets to obtain m deformed content template feature maps. Then, based on m attention coefficients, the m deformed content template feature maps are weighted and summed to obtain an aggregated feature map. The aggregated feature map, through the m horizontal and m pixel offsets, addresses the issue of horizontal and vertical offsets between the text shapes of the content template image and the target text image to a certain extent. The m attention coefficients represent the degree of importance given to the m deformed content template feature maps sampled m times, ensuring that the final aggregated feature map is a balanced result of the m possible offsets.

[0017] In one possible implementation, the above-mentioned process of obtaining pixel offsets and attention coefficients based on the content template feature map and the input second input feature map includes: concatenating the content template feature map and the input second input feature map to obtain a concatenated feature map; performing convolution calculation on the concatenated feature map to output pixel offsets and attention coefficients.

[0018] In this implementation, the content template feature map and the input feature map are concatenated to obtain a concatenated feature map. This concatenated feature map is then used as input to a convolutional network module, which outputs pixel offsets and attention coefficients. In other words, the convolutional network module processes the concatenated feature map to obtain multiple pixel offsets and the corresponding attention coefficients for each pixel offset, thus solving the problem of calculating the text glyph offsets in the content template image.

[0019] In one possible implementation, obtaining the second output feature map based on the aggregated feature map and the input second input feature map includes: performing a residual connection on the aggregated feature map and the input second input feature map to obtain the second output feature map.

[0020] In this implementation, a residual connection is performed between the aggregated feature map and the input feature map to obtain the output feature map. The output feature map after the residual connection includes both the aggregated feature map and the input feature map, which facilitates further processing of the aggregated feature map and the input feature map by downstream neural network layers.

[0021] In one possible implementation, the above-mentioned control of the style of target text based on style reference image includes: using the style reference image as input to a style encoder and outputting style reference features; processing the input third input feature map and style reference features based on a cross-attention mechanism to obtain the output third output feature map; and controlling the style of target text based on the third output feature map.

[0022] In this implementation, a third input feature map and a style reference feature map are processed based on a cross-attention mechanism to obtain a third output feature map. By processing the third input feature map and the style reference feature map using a cross-attention mechanism, control over the style of the target text is achieved. Simultaneously, the guidance of the content template image and the content template edge image ensures the integrity of the geometric structure information of the target text, such as its outline, edges, and glyphs. Therefore, the guidance of the style reference image enables accurate control of the text style when the glyph structure is complete, solving the problem of inaccurate style control caused by incomplete glyph structure in the target image.

[0023] In one possible implementation, the content template edge image is obtained by edge extraction from the content template image.

[0024] In this implementation, extracting the outline edges of the text in the content template image yields a content template edge image that represents the outline edges of the text content. Therefore, the content template edge image can better represent the geometric structure of the text content, facilitating control over the geometric structure of the target text, such as its outline edges, in the target image during the text generation process.

[0025] In one possible implementation, the generation process of the target image is a reverse denoising process of the diffusion model, implemented through T iterations, where T is an integer greater than or equal to 1, and the loop control variable t is the current time step, changing in descending order from T to 1, which is the opposite of the time step change in the forward denoising process of the corresponding diffusion model; the value range of t is decreasing by 1 from T to 1, and the processing of the t-th (t = T, T-1, ..., 1) iteration includes:

[0026] Acquire noisy image X t The noisy image is the denoised image output from the previous iteration during the iterative process. When the iterative process begins (t=T), the noisy image is the noisy image X. T ;

[0027] The current time step t and the noisy image X are used. t The content template image, the style reference image, and the content template edge image serve as inputs to the text generation model. Guided by these images, the model outputs the predicted noise e at time step t. t The prediction noise e at the t-th time step t This represents the noisy image X at time step t during the forward noise addition process of the corresponding diffusion model. t Adding noise The predicted value;

[0028] According to the noisy image X t and the prediction noise e at the t-th time step t The denoised image X is obtained. t-1 Denoising image X t-1 The noisy image used as input for the next iteration in the iterative process;

[0029] When the loop iteration terminates (t=1), the denoised image is the generated target image X0.

[0030] This implementation provides an illustrative explanation of the target image generation process. In the inverse denoising process of the diffusion model, T iterations are required to obtain the desired target image. In step t, the main steps include: acquiring the noisy image output from the previous iteration; guided by the content template image, style reference image, and content template edge image, the text generation model outputs the predicted noise for step t based on the input noisy image and the current time step; and obtaining the denoised image based on the noisy image and the predicted noise for step t, which is used as the input for the next time step. After T iterations, the final target image is output.

[0031] Secondly, embodiments of this application provide a training method for a text generation model, mainly including the following steps: obtaining training samples, which include ground truth samples of the target image, content template image samples, style reference image samples, and content template edge image samples. The content template image samples are used to indicate the content of the text in the ground truth samples of the target image, the style reference image samples are used to indicate the style of the text in the ground truth samples of the target image, and the content template edge image samples are used to indicate the geometric structure information of the text in the ground truth samples of the target image; the training process includes T iterations, where T is an integer greater than or equal to 1. For each iteration in the training process, a time step t (1≤t≤T) is randomly sampled, and corresponding noise is added. (This adds noise) (Following a standard normal distribution), noise is added based on the ground truth sample X0 of the target image, time step t, and the corresponding time step t. The noisy image sample X at time step t is obtained by using the forward noise addition process of the diffusion model. t When t = T, X T The forward noise addition process of the diffusion model ultimately yields a noisy image that follows a standard normal distribution; the text generation model is trained through N iterations, and the training process of the nth (1≤n≤N) iteration includes: setting the time step t and the noisy image sample X... t The text generation model takes content template image samples, style reference image samples, and content template edge image samples as inputs, and outputs predicted noise e. t Predicted noise et This indicates that, guided by the content template image sample, style reference image sample, and content template edge image sample, the noisy image sample X... t Adding noise The predicted value; based on the predicted noise e t and the added noise A first loss value is obtained; the ground truth sample of the target image is processed to obtain a corner image and an edge image, wherein the corner image is used to indicate the corner position information of the text in the ground truth sample of the target image, and the edge image is used to indicate the edge position information of the text in the ground truth sample of the target image; the first loss value is weighted according to at least one of the corner image and the edge image to obtain a second loss value; the parameters of the text generation model are optimized by minimizing the second loss value as the training objective.

[0032] In this embodiment, during the training phase of the text generation model, a method is introduced to weight the first loss value using edge and corner images of ground truth samples of the target image. This allows the text generation model to pay more attention to the geometric structure of characters in the input noisy image during the generation of the target image, ensuring the accuracy of the character geometry in the target image and improving the integrity of the character structure and the accuracy of style control. In other words, during the training phase, by explicitly focusing on and utilizing the geometric structure information of handwritten characters, such as character outlines and corner points, the diffusion model generates target images with more complete character structures and more accurate style control when applied to handwritten font generation tasks.

[0033] In one possible implementation, the text generation model includes a noise prediction model, a content encoder, a contour encoder, and a style encoder. The above-mentioned time step t and noisy image sample X... t The text generation model takes content template image samples, style reference image samples, and content template edge image samples as inputs, and outputs predicted noise e. t The process includes: using content template image samples as input to a content encoder and outputting a sample content template feature map; using content template edge image samples as input to a contour encoder and outputting sample contour features; using style reference image samples as input to a style encoder and outputting sample style reference features; and, guided by the sample content template feature map, sample contour features, and sample style reference features, a noise prediction model processes the input noisy image samples according to time step t and outputs predicted noise. The training objective is to minimize the second loss value to optimize the parameters of the text generation model, including: using the minimization of the second loss value as the training objective to optimize the parameters of the noise prediction model, content encoder, contour encoder, and style encoder.

[0034] In this implementation, the time step t and the noisy image sample X are... t The text generation model takes content template image samples, style reference image samples, and content template edge image samples as inputs. Guided by the sample content template feature map, sample contour features, and sample style reference features, it outputs predicted noise e. t The predicted noise output by the text generation model is equivalent to the predicted value of the added noise in the noisy image, so the predicted noise e t and adding noise The loss value can characterize the ability of a text generation model to predict noise from a noisy image.

[0035] In one possible implementation, the noise prediction model includes an image encoder and an image decoder, as well as an embedding encoding module that converts time step t into a vector. The above-mentioned conversion of time step t and noisy image sample X... t The text generation model takes content template image samples, style reference image samples, and content template edge image samples as inputs, and outputs predicted noise e. t This includes: an embedding coding module that converts time step t into a time step embedding vector; and a module that converts the time step embedding vector and the noisy image sample X into a time step embedding vector. t The image encoder takes sample content template feature maps, sample contour features, and sample style reference features as inputs and outputs image encoded features. The image decoder takes time-step embedding vectors, image encoded features, sample contour features, and sample style reference features as inputs and outputs prediction noise e. t .

[0036] In this implementation, a noise prediction model is used to predict noise during the inverse denoising process of the diffusion model. The noise prediction model includes an image encoder and an image decoder, as well as an embedding encoding module that converts time step t into a time step embedding vector. Multiple neural network layers are added to the image encoder to receive sample content template feature maps, sample contour features, and sample style reference features as inputs, thereby controlling the encoding process. Similarly, multiple neural network layers are added to the image decoder to receive sample contour features and sample style reference features as inputs, thereby controlling the decoding process. Thus, during the image encoding and decoding processes of the diffusion model, the sample content template feature maps, sample contour features, and sample style reference features control the generation process of predicted noise. Furthermore, the time step embedding vector is used to control the computation process of each iteration.

[0037] In one possible implementation, the above-mentioned weighting of the first loss value based on at least one of the corner image and the edge image to obtain the second loss value includes: weighting and summing the corner image and the edge image using a first weight and a second weight respectively to obtain image geometric structure weight coefficients, wherein the first weight and the second weight are values ​​greater than or equal to 0 and less than or equal to 1; and weighting the first loss value based on the image geometric structure weight coefficients to obtain the second loss value.

[0038] This implementation provides a method for weighting a first loss value using corner images and edge images. First, the corner images and edge images are weighted and summed separately using a first weight and a second weight to obtain image geometric structure weight coefficients. Then, based on these image geometric structure weight coefficients, the first loss value is weighted to obtain a second loss value. Different weights can be assigned to the first and second weights to control the respective influence of the corner images and edge images during the weighting process of the first loss value.

[0039] Thirdly, embodiments of this application provide a text generation apparatus, mainly comprising: an acquisition module, used to acquire a content template image, a style reference image, a content template edge image, and a noise image, wherein the content template image is used to indicate the content of the target text, the style reference image is used to indicate the style of the target text, and the content template edge image is used to indicate the geometric structure information of the target text; and a processing module, used to generate a target image of the target text based on the noise image, guided by the content template image, the style reference image, and the content template edge image.

[0040] In one possible implementation, the processing module is specifically used to: control the content of the target text based on the content template image; control the geometric structure information of the target text based on the content template edge image; and control the style of the target text based on the style reference image.

[0041] In one possible implementation, the processing module is specifically used to: take the content template edge image as input to the contour encoder and output contour features; based on the contour features and the first input feature map, obtain the first output feature map through a cross-attention mechanism; and control the geometric structure information of the target text based on the first output feature map.

[0042] In one possible implementation, the processing module is specifically used to: take the first input feature map as the query vector, take the contour features as the key vector and value vector, and calculate the first output feature map through cross-attention.

[0043] In one possible implementation, the processing module is specifically used for: taking the content template image as input to the content encoder and outputting a content template feature map; obtaining pixel offsets and attention coefficients based on the content template feature map and the input second input feature map; processing the content template feature map based on the pixel offsets and attention coefficients to obtain an aggregated feature map; obtaining an output second output feature map based on the aggregated feature map and the input second output feature map; and controlling the content of the target text based on the second output feature map.

[0044] In one possible implementation, the pixel offset includes m horizontal pixel offsets and m vertical pixel offsets, and the number of attention coefficients is m, where m is an integer greater than or equal to 1. The processing module is specifically used to: perform bilinear sampling on the content template feature map based on the m horizontal and m vertical pixel offsets to obtain m deformed content template feature maps; and perform weighted summation on the m deformed content template feature maps based on the m attention coefficients to obtain an aggregated feature map.

[0045] In one possible implementation, the processing module is specifically used to: concatenate the content template feature map and the second input feature map to obtain a concatenated feature map; perform convolution calculation on the concatenated feature map, and output pixel offset and attention coefficient.

[0046] In one possible implementation, the processing module is specifically used to: perform a residual connection between the aggregated feature map and the input second input feature map to obtain a second output feature map.

[0047] In one possible implementation, the processing module is specifically used to: take the style reference image as input to the style encoder and output style reference features; and process the third input feature map and style reference features based on a cross-attention mechanism to obtain a third output feature map.

[0048] In one possible implementation, the content template edge image is obtained by edge extraction from the content template image.

[0049] In one possible implementation, the generation of the target image is a reverse denoising process of the diffusion model, implemented through T iterations, where T is an integer greater than or equal to 1. The loop control variable t is the current time step, changing in descending order from T to 1, which is the opposite of the time step change in the forward denoising process of the corresponding diffusion model. The processing of the t-th (t = T, T-1, ..., 1) iteration includes: acquiring the noisy image X. t The noisy image is the denoised image output from the previous iteration during the iterative process. When the iterative process begins (t=T), the noisy image is the noisy image X. T The current time step t and the noisy image X are used to...t The content template image, the style reference image, and the content template edge image serve as inputs to the text generation model. Guided by these images, the model outputs the predicted noise e at time step t. t The prediction noise e at the t-th time step t This represents the noisy image X at time step t during the forward noise addition process of the corresponding diffusion model. t Adding noise The predicted value; based on the noisy image X t and the prediction noise e at the t-th time step t The denoised image X is obtained. t-1 Denoising image X t-1 The noisy image is used as the input for the next iteration in the iterative process; when the iterative process terminates (t=1), the denoised image is the generated target image X0.

[0050] Fourthly, embodiments of this application provide a training apparatus for a text generation model, comprising: a sample acquisition module for acquiring training samples, the training samples including ground truth samples of a target image, content template image samples, style reference image samples, and content template edge image samples, the content template image samples indicating the content of the text in the ground truth samples of the target image, the style reference image samples indicating the style of the text in the ground truth samples of the target image, and the content template edge image samples indicating the geometric structure information of the text in the ground truth samples of the target image; and a preprocessing module, the training process including T iterations, where T is an integer greater than or equal to 1, for each iteration in the training process, a time step t (1≤t≤T) is randomly sampled, and corresponding noise is added. (This adds noise) Following a standard normal distribution, based on the ground truth sample X0 of the target image, time step t, and the noise added at time step t, the noisy image sample X at time step t is obtained using the forward noise addition process of the diffusion model. t When t = T, X T The noise image that ultimately follows a standard normal distribution is obtained from the forward noise addition process of the diffusion model.

[0051] The training module is used to train the text generation model. The text generation model is trained through N iterations. The training process of the nth (1≤n≤N) iteration includes: processing the time step t and the noisy image sample X output by the preprocessing module. t Add noise The content template image sample, style reference image sample, and content template edge image sample output by the sample acquisition module are used as inputs to the text generation model, and the output is the predicted noise e.t Predicted noise e t This indicates that, guided by the content template image sample, style reference image sample, and content template edge image sample, the noisy image sample X... t Adding noise The predicted value; based on the predicted noise e t and the added noise A first loss value is obtained; and the ground truth sample of the target image is processed to obtain a corner image and an edge image, wherein the corner image is used to indicate the corner position information of the text in the ground truth sample of the target image, and the edge image is used to indicate the edge position information of the text in the ground truth sample of the target image; and the first loss value is weighted according to at least one of the corner image and the edge image to obtain a second loss value; and the parameters of the text generation model are optimized by minimizing the second loss value as the training objective.

[0052] In one possible implementation, the text generation model described above includes a noise prediction model, a content encoder, a contour encoder, and a style encoder. The training module is specifically used for: taking content template image samples as input to the content encoder and outputting a sample content template feature map; taking content template edge image samples as input to the contour encoder and outputting sample contour features; taking style reference image samples as input to the style encoder and outputting sample style reference features; and taking time step t and noisy image samples X... t The sample content template feature map, sample contour features, and sample style reference features are used as inputs to the noise prediction model, and the output is the predicted noise e. t The text generation model is optimized by minimizing the second loss value as the training objective. This includes optimizing the parameters of the noise prediction model, content encoder, contour encoder, and style encoder by minimizing the second loss value as the training objective.

[0053] In one possible implementation, the noise prediction model includes an image encoder and an image decoder, as well as an embedding encoding module that converts time step t into a vector. The training module is specifically used for: the embedding encoding module converting time step t into a time step embedding vector; and the embedding vector and the noisy image sample X... t The image encoder takes sample content template feature maps, sample contour features, and sample style reference features as inputs and outputs image encoded features. The image decoder takes time-step embedding vectors, image encoded features, sample contour features, and sample style reference features as inputs and outputs prediction noise e. t .

[0054] In one possible implementation, the training module is specifically used to: perform a weighted summation of the corner image and the edge image using a first weight and a second weight, respectively, to obtain image geometric structure weight coefficients, wherein the first weight and the second weight are values ​​greater than or equal to 0 and less than or equal to 1; and weight the first loss value according to the image geometric structure weight coefficients to obtain a second loss value.

[0055] Fifthly, this application provides a chip system including a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, and the processor being used to execute the method described in any possible implementation of the first aspect and / or the algorithmic functions embodied in the apparatus described in any possible implementation of the second aspect.

[0056] Sixthly, this application provides a computing device including a processor and a memory. The processor is configured to execute instructions stored in the memory to cause the computing device to perform the methods described in any possible implementation of the first aspect and / or the algorithmic functions embodied by the means described in any possible implementation of the second aspect.

[0057] In a seventh aspect, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the method described in any possible implementation of the first aspect and / or the algorithmic function embodied by the apparatus described in any possible implementation of the second aspect.

[0058] Eighthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in any possible implementation of the first aspect and / or the algorithmic function embodied by the apparatus described in any possible implementation of the second aspect, wherein the cluster of computing devices includes at least one computing device.

[0059] Ninthly, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in any possible implementation of the first aspect and / or the algorithmic function embodied by the apparatus described in any possible implementation of the second aspect, wherein the cluster of computing devices includes at least one computing device.

[0060] It is understood that the beneficial effects of the third to ninth aspects mentioned above can be found in the relevant descriptions in the first and / or second aspects mentioned above, and will not be repeated here. Attached Figure Description

[0061] The accompanying drawings used in the embodiments or technical description are briefly introduced below.

[0062] Figure 1 A schematic diagram illustrating the composition of a text generation architecture provided in an embodiment of this application;

[0063] Figure 2 A schematic diagram illustrating the inference stage of a text generation model provided in this application embodiment;

[0064] Figure 3 A flowchart illustrating a text generation method provided in an embodiment of this application;

[0065] Figure 4a A flowchart illustrating a training method for a text generation model provided in an embodiment of this application;

[0066] Figure 4b A flowchart illustrating the training process with a sampling time step of t, as provided in the embodiments of this application;

[0067] Figure 5 A schematic diagram illustrating the training phase of a text generation model provided in an embodiment of this application;

[0068] Figure 6 A schematic diagram illustrating an example of a text generation model provided in an embodiment of this application;

[0069] Figure 7 A schematic diagram of a deformable content aggregation layer provided in an embodiment of this application;

[0070] Figure 8 This is a schematic diagram illustrating an example of a corner image and an edge image provided in an embodiment of this application.

[0071] Figure 9 A schematic diagram illustrating an example of handwritten text generation results provided in an embodiment of this application;

[0072] Figure 10 This is a schematic diagram illustrating the composition of a text generation system provided in an embodiment of this application;

[0073] Figure 11 This is a schematic diagram illustrating the composition of a text generation device provided in an embodiment of this application;

[0074] Figure 12 A schematic diagram illustrating the composition of a training device for a text generation model provided in an embodiment of this application;

[0075] Figure 13 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0076] Figure 14This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0077] Figure 15 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. Detailed Implementation

[0078] In this article, the term "and / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. The symbol " / " in this article indicates that the related objects have an "or" relationship; for example, A / B means A or B.

[0079] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0080] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0081] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0082] To facilitate understanding of the solutions provided in the embodiments of this application, a brief introduction to some of the terms involved in this solution will be given first.

[0083] Content: refers to the specific meaning or significance expressed by the text.

[0084] Style: A concept relative to content definition. It refers to the information remaining in the text besides the content itself. For example, style can refer to the font of the text, such as Song, Kai, or Hei. Style can also refer to the way the text is written, such as artistic fonts or handwriting. When the text is handwritten, style can also indicate the writing style and habits of the person who wrote it.

[0085] Geometric structure information: refers to the information about the geometric structure of characters, such as the outline of the characters and the corner points of the characters.

[0086] Diffusion Model: Originating from non-equilibrium thermodynamics, the diffusion model is a type of generative model based on probability likelihood. In other words, it's a deep learning-based generative model with a basic architecture including an image encoder and an image decoder. It features a forward noise addition process and a reverse denoising process. The forward noise addition process, or diffusion, progressively adds noise to a real image to obtain a noisy image that follows a standard normal distribution. The reverse denoising process can be used for text generation tasks. It initializes the noisy image with a standard normal distribution as the initial value. For each iteration of the noisy image, the model predicts the noise added during the forward noise addition process based on the current time step, removes the noise from the noisy image, and finally generates a denoised handwritten text image.

[0087] A time step represents a sequential sequence of steps, where the output of the previous step becomes the input of the next. Taking the diffusion model as an example, the forward noise addition process is divided into a fixed number of T steps, where T is an integer greater than or equal to 1. The forward noise addition process is completed iteratively by iterations t = 1, ..., T. The reverse denoising process of the diffusion model also requires a corresponding number of T steps. Each time step in the reverse denoising process is used to predict the noise added at the corresponding time step in the forward noise addition process, thus achieving the reverse denoising process. This reverse denoising process is completed iteratively by iterations t = T, ..., 1. For example, if a time step t (1 ≤ t ≤ T) is randomly sampled, the forward noise addition process is as follows: based on the noisy image X... t-1 Add noise Obtain the noisy image X t The reverse denoising process is as follows: based on the noisy image X... t This application proposes to predict noise e under the guidance of a content template image, a style reference image, and a content template edge image. t According to the predicted noise e t and noisy image X t This yields the denoised image output at time step t, which corresponds to the noisy image X input at time step t-1. t-1 During the training phase, the prediction noise e is minimized. t and adding noise The loss value is used to optimize the model parameters; during the inference phase, the noise e is predicted iteratively t = T, ..., 1. t For noisy image X t Denoising is performed to obtain the denoised image X. t-1 .

[0088] Handwritten font generation is an Artificial Intelligence Generated Content (AIGC) technology suitable for applications such as personalized font design and augmenting training samples for handwritten character recognition models. Compared to printed font generation technology, the generation of handwritten fonts with controllable content and style remains an unsolved problem due to the diverse styles of handwritten characters.

[0089] The first solution, based on Generative Adversarial Networks (GANs), improves text generation by adversarially training the generator and discriminator. It introduces a content encoder and a style encoder to conditionally control the generation process. This approach includes a pre-adjustment phase and a downstream font generation phase, with the content encoder and style encoder shared between the two phases. However, GAN-based methods use adversarial loss to guide the learning of the generative model, leading to unstable training, difficulty in convergence, and a susceptibility to problems such as mode collapse. For example, mode collapse refers to a significant deviation between the generated image and the ground truth glyphs of the target image, resulting in incomplete or incorrect glyph structures and causing training failure.

[0090] The second solution utilizes a diffusion model for printed font generation, employing a content encoder and style encoder similar to those in generative adversarial networks (GANs) to conditionally control the image generation process. In one example, the conditional diffusion model in this solution is a U-Net-based network consisting of a content encoder and a style encoder. The reference image is processed by both the style encoder and content encoder to obtain style embeddings and structure maps. A style contrast refinement module is used to separate different styles from the image and provide guidance for the diffusion model. However, this solution is primarily geared towards printed fonts. When applied to handwritten font generation tasks, the generated handwritten text images tend to suffer from incomplete glyph structures and inaccurate style control.

[0091] In view of this, this application proposes a text generation method. In the process of generating handwritten text images from noisy images using a diffusion model, a content encoder, a contour encoder, and a style encoder are used for conditional control. The contour encoder considers the geometric structure information of handwritten text, such as the character outlines, and explicitly focuses on and utilizes this information. This results in a more complete character structure and more accurate style control in the handwritten text image, solving problems such as incomplete character structure and inaccurate style control in handwritten text images.

[0092] In some possible implementations of this application, a deformable content aggregation layer is introduced into the image encoder to control the content of handwritten text generation using content template images. A content geometry control layer is introduced into both the image encoder and image decoder to further control the glyph structure of the handwritten text using edge images of the content templates. During the training phase, a geometrically weighted image reconstruction loss is used to optimize the objective function of the model, with weighting coefficients including geometric structure information such as corner points and edges extracted from the ground truth of the handwritten text target image.

[0093] To better understand the text generation method of this application embodiment, the application scenarios of the text generation method of this application embodiment are described below, but the description of the application scenarios does not constitute a limitation on the embodiments of this application.

[0094] See Figure 1 , Figure 1 This is a schematic diagram illustrating the composition of a text generation architecture provided in an embodiment of this application. In this embodiment, a text generation architecture 100 deployed with this embodiment may include a terminal 110 and a server 120. The server 120 may include one or more servers (…). Figure 1 (The example includes a server) The server 120 can provide the methods or apparatus provided in the embodiments of this application to one or more terminals.

[0095] Optionally, a relevant application may be installed on terminal 110. This application or webpage can provide an interface, allowing terminal 110 to receive content, style, and other related information input by the user on the interface, and send this information to server 120. Server 120 deploys the text generation model of this embodiment, which can generate handwritten text images according to the aforementioned text generation method. Optionally, the handwritten text image can also be returned to terminal 110. Optionally, the handwritten text image can also be used as training data for downstream tasks, which can be any task that learns from the handwritten text image.

[0096] It should be understood that in some optional implementations, terminal 110 may deploy the text generation model of this application embodiment and complete the text generation work on its own, without the need for the cooperation of server 120. This application embodiment is not limited in this respect. In some optional implementations, server 120 may also use content, style and other related information received from other devices outside the terminal or stored locally, without the need for the cooperation of terminal 110. This application embodiment is not limited in this respect.

[0097] The following description Figure 1The product form of the terminal 110. In this application embodiment, the terminal 110 can be a mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0098] The following description Figure 1 The product form of server 120. It can be further understood that server 120 can be various types of servers, such as x89 architecture servers, specifically rack servers, blade servers, high-density servers, platform servers, or high-performance servers, etc. In other words, this application embodiment does not specifically limit the specific type of server. Furthermore, it can be understood that... Figure 1 The server structure shown does not constitute a limitation on the server structure. A server may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0099] Furthermore, server 120 can be configured as an independent physical server, or as a server cluster or distributed system consisting of multiple physical servers. It can also be configured as a cloud server or cloud server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms. The cloud server cluster is deployed in several cloud data centers. The software can be an application that implements object control methods, but it is not limited to the above forms.

[0100] Next, the communication connection method between terminal 110 and server 120 is described. For example, terminal 110 and server 120 are connected via a network, enabling terminal 110 to access the cloud management platform deployed on the cloud server cluster. The network can be a wired network or a wireless network. For example, a wired network can be a cable network, fiber optic network, Digital Data Network (DDN), etc., while a wireless network can be a telecommunications network, intranet, Internet, Local Area Network (LAN), Wide Area Network (WAN), Wireless Local Area Network (WLAN), Metropolitan Area Network (MAN), Public Service Telephone Network (PSTN), Bluetooth network, ZigBee network, Global System for Mobile Communications (GSM), CDMA (Code Division Multiple Access) network, CPRS (General Packet Radio Service) network, etc., or any combination thereof.

[0101] Understandably, a network can use any known network communication protocol to enable communication between different terminal layers and gateways. These network communication protocols can be various wired or wireless communication protocols, such as Ethernet, Universal Serial Bus (USB), FireWire, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), New Radio (NR), Bluetooth, Wireless Fidelity (Wi-Fi), and other communication protocols.

[0102] In one possible scenario, server 120 can function as a cloud platform (a software platform employing application virtualization technology, integrating multiple functions such as software search, download, use, management, and backup). In practical use, server 120 can deploy a cloud management platform and a data center, with terminal 110 interacting with the cloud through the cloud management platform. Additionally, the data center can deploy nodes, which can be virtual machine instances, container instances, physical servers, etc.

[0103] In another possible scenario, the method provided in this application embodiment can be implemented by software. The software has a terminal and a server; the terminal 110 runs the software, and the server 120 runs the software's server. While the terminal 110 is running the software, it can call the server running on the server 120 to implement the method provided in this application embodiment.

[0104] In other words, the method provided in this application embodiment can be applied to the terminal 110 or the server 120. In specific implementation, it can run as software on the terminal 110 or the server 120; for example, the software can be a service or an application. This application embodiment can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application embodiment can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0105] Furthermore, the systems that can implement the technical solutions of this application embodiment include cloud servers, web pages, application program interfaces (APIs), mobile application software, etc. The text generation model of this application embodiment can be deployed on the server side, website, or mobile application software to meet the text generation needs of enterprise users and individual users, and can also be used by developers for secondary development through open API interfaces. Furthermore, the implementation of this application embodiment mainly involves deploying the relevant platforms and applications of the text generation model of this application embodiment, such as cloud platforms, websites, and applications (APPs).

[0106] For example, the application scenarios involved in the embodiments of this application include: handwritten text generation applications in fields such as personalized font library construction, handwriting simulation, and personalized design; obtaining training data for tasks such as handwritten text recognition through handwritten text generation methods; and generating images containing various fonts such as artistic fonts and printed fonts based on text generation models.

[0107] Based on such Figure 1 The application scenario of the text generation architecture shown in this application embodiment provides a text generation model, which mainly includes a training stage and an inference stage. The training process and inference process of the text generation model will be described below.

[0108] See Figure 2 , Figure 2 This is a schematic diagram illustrating the inference stage of a text generation model provided in an embodiment of this application. For example... Figure 2 As shown, the text generation model includes: a noise prediction model, a content encoder, a contour encoder, and a style encoder. During the inference phase of the text generation model, input data is acquired, including noise images, content template images, style reference images, and content template edge images.

[0109] Optionally, the outline edges of the text in the content template image can be extracted to obtain the content template edge image.

[0110] Furthermore, the iterative control variable time step t, the noise image, the content template image, the style reference image, and the content template edge image are input into the text generation model. Under the constraints of the text content indicated by the content template image, the text style indicated by the style reference image, and the text outline edges indicated by the content template edge image, the text generation model outputs predicted noise, which is the predicted value of the noise in the noise image. The noise image and the predicted noise are then used as input to the image denoising module, which outputs a denoised image. After multiple iterations, the generated target image is output.

[0111] For example, such as Figure 2 As shown, the generation process of the target image is achieved through T iterations, resulting in a noisy image X that follows a standard normal distribution. T The input for the first iteration is the target image, and the output for the last iteration is the target image. T is an integer greater than or equal to 1, and the loop control variable t is an integer greater than or equal to 1 and less than or equal to T. The iterations are performed in descending order of the time steps corresponding to the loop control variables t = T, ..., 1. Figure 2 As shown, the processing steps at time step t include: acquiring the noisy image X. t Noisy image X tThe image output from the previous iteration is shown. At the start of the iteration (t = T), the noisy image is X. T Guided by the content template image, style reference image, and content template edge image, the noisy image X is... t As input to the text generation model, the output is the predicted noise e at time step t. t The prediction noise e at time step t t This represents the predicted noise value in the noisy image input at time step t; based on the noisy image X... t and the prediction noise e at time step t t The denoised image X is obtained. t-1 Denoising image X t-1 Used as input for the next loop iteration; when the loop iteration terminates (t=1), the denoised image is the generated target image X0.

[0112] In this embodiment, the image denoising module refers to a method, software, or apparatus for removing specified predicted noise from a noisy image by following the inverse process of the corresponding denoising step in the forward denoising process of the diffusion model. This will not be elaborated further in this embodiment.

[0113] Optionally, the content template image is used as input to the content encoder, and the output is a content template feature map; the content template edge image is used as input to the contour encoder, and the output is contour features; the style reference image is used as input to the style encoder, and the output is style reference features. The time step t and the noisy image sample X are then used as input. t The sample content template feature map, sample contour features, and sample style reference features are used as inputs to the noise prediction model, and the output is the predicted noise e. t The predicted noise represents the predicted value of the noise in the noisy image; the target image is obtained based on the noisy image and the predicted noise.

[0114] For example, such as Figure 2As shown, the content template image indicates the content of the target text, such as the text content being "滞"; the content template edge image indicates the geometric structure information of the target text, such as the glyph outline, stroke inflection points, etc. of the character "滞"; the style reference image is generally an image of text other than the target text but with the style of the pre-generated target text, such as the style of the handwritten character "沼" in the style reference image, used to indicate the style of the target text. The target image is an image containing the target text. In the target image, the content of the target text is "滞", which is the content indicated by the content template image; the style of the target text is the style of the handwritten character "沼", which is the style indicated by the style reference image; the geometric structure information of the target text is the geometric structure information of "滞", which is the geometric structure information indicated by the content template edge image. Thus, the target text in the target image meets the requirements in terms of content, style, and geometric structure.

[0115] Exemplarily, the text generation model can be understood as an application of the diffusion model based on a neural network. The limiting conditions for the reverse denoising process of the diffusion model proposed in the embodiments of the present application are the content template image, the style reference image, and the content template edge image.

[0116] Among them, since the content template contour image is also a condition restriction of the text generation model, the text contour of the generated target image is the contour of the text content indicated by the content template image, avoiding the loss of some text contours and the inconsistency between the styles of some text contours and the overall style, ensuring that the text structure in the generated image is more complete and the style control is more accurate, and solving problems such as incomplete glyph structures and inaccurate style control in the generated image.

[0117] See Figure 3 , Figure 3 is a schematic flowchart of a text generation method provided by an embodiment of the present application. As Figure 3 shown, an embodiment of the present application provides a text generation method, which can be understood as an inference method of a text generation model in the inference stage. As Figure 3 shown, the text generation method mainly includes the following steps:

[0118] Step S310, obtain a content template image, a style reference image, a content template edge image, and a noise image. Among them, the content template image is used to indicate the content of the target text, the style reference image is used to indicate the style of the target text, and the content template edge image is used to indicate the geometric structure information of the target text.

[0119] In a possible implementation, the noise image is obtained by initializing a noise image that follows a standard normal distribution.

[0120] In one possible implementation, edge extraction processing is performed on the content template image to obtain a content template edge image. For example, the edge extraction algorithm could be the Canny operator, etc.

[0121] In this implementation, extracting the outline edges of the text in the content template image yields a content template edge image that represents the outline edges of the text content. Therefore, the content template edge image can better represent the geometric structure of the text content, facilitating control over the outline edges and geometric structure of the target text in the target image during the text generation process.

[0122] Step S320: Guided by the content template image, style reference image and content template edge image, generate a target image of the target text based on the noise image.

[0123] In this embodiment, guided by the content template image, a target image is generated based on a noise image, ensuring that the text content of the target image is identical to that of the content template image, thereby controlling the text content of the target image. Guided by the style reference image, a target image is also generated based on a noise image, ensuring that the text style of the target image is identical to that of the style reference image, thereby controlling the text style of the target image. Furthermore, guided by the content template edge image, a target image is generated based on a noise image, ensuring that the text outline edges of the target image are identical to those of the content template edge image, thereby controlling the text outline edges of the target image. Because the content template edge image is introduced during the target image generation process, the geometric structure information of handwritten characters, such as glyph outlines, is explicitly considered and utilized. This ensures that when the diffusion model is applied to handwritten font generation tasks, the generated target image has a complete glyph structure and accurate style control. In other words, it solves the problem of incomplete glyph structure in the target image and the problem of inaccurate style control caused by incomplete glyph structure, such as the loss or error of a certain part of the glyph structure leading to incorrect style control of that part of the glyph. Furthermore, the text generation method proposed in this application can be applied to the generation of handwritten character images, and can also be used in other text generation tasks, such as artistic character generation, printed character generation, etc., and this application does not impose any limitations on this. Additionally, the text generation method proposed in this application can be applied to the generation of any language, and the target language can be, but is not limited to, Chinese, English, Japanese, etc.

[0124] In one possible implementation, the above-mentioned generation of a target image of the target text based on a noise image, guided by a content template image, a style reference image, and a content template edge image, includes: in the process of generating the target image based on the noise image, controlling the content of the target text based on the content template image; controlling the geometric structure information of the target text based on the content template edge image; and controlling the style of the target text based on the style reference image.

[0125] In one possible implementation, the above-described generation of a target image based on a noisy image under the guidance of a content template image includes: obtaining an input feature map; encoding the content template image to obtain a content template feature map; obtaining pixel offsets and attention coefficients based on the content template feature map and the input feature map; processing the content template feature map based on the pixel offsets and attention coefficients to obtain an aggregated feature map; and obtaining an output feature map based on the aggregated feature map and the input feature map.

[0126] In this implementation, for example, the guidance process of the content template image can be implemented through a specially designed neural network layer. The input of this neural network layer is the output of the previous neural network layer, i.e., the input feature map; another input of this neural network layer is the content template feature map obtained by encoding the content template image; the output of this neural network layer is the input of the next neural network layer, i.e., the output feature map. This embodiment does not limit the position of the neural network layer; it can be upstream or downstream of the neural network layer corresponding to the guidance process of the style reference image and the content template edge image. Specifically, the neural network layer can perform the following: obtain pixel offsets and attention coefficients based on the content template feature map and the input feature map; process the content template feature map based on the pixel offsets and attention coefficients to obtain an aggregated feature map; and obtain the output feature map based on the aggregated feature map and the input feature map. By using pixel offsets and attention coefficients, the content template feature map can be aligned with the target text in the target image, reducing the risk of incomplete character shapes in the target text. In other words, to address the problem of inaccurate control caused by the difference in the glyphs of handwritten characters between the content template image and the target image, unlike existing methods that directly concatenate the input feature map with the content template feature map, this application proposes to introduce a specially designed neural network layer (such as a deformable content aggregation layer) into the image encoder of the diffusion model. By deforming the content template feature map, the glyph information of the characters in the content template is adaptively aggregated.

[0127] During the training phase of the text generation model, the content template feature map of the content template image after deformation by pixel offset and attention coefficient can better align the corresponding pixels in the ground truth sample of the target image with the content template image sample. After deformation, the content template feature map can better align with the glyphs of the fonts in the ground truth sample of the target image, which improves the quality of the ground truth sample of the target image generated by the text generation model, improves the training effect of the text generation model, solves the problem of incomplete glyph structure in the target image, and solves the problem of inaccurate style control caused by incomplete glyph structure.

[0128] In the inference phase of the text generation model, the inverse denoising process based on the diffusion model achieves text generation through multiple time steps. As text gradually appears in the noisy image corresponding to the intermediate process during the multiple iterations in the inference phase, the text generation model can align the text in the content template image with the text in the noisy image through pixel offset and attention coefficient, thereby improving the accuracy of the diffusion model in predicting noise, improving the quality of the target image generated by the text generation model, solving the problem of incomplete glyph structure in the target image, and the problem of inaccurate style control caused by incomplete glyph structure.

[0129] In one possible implementation, the above-mentioned processing of the content template feature map based on pixel offset and attention coefficient to obtain an aggregated feature map includes: sampling the content template feature map based on pixel offset to obtain a deformed content template feature map; and weighting the deformed content template feature map based on attention coefficient to obtain an aggregated feature map.

[0130] In this implementation, the content template feature map is sampled based on the pixel offset to obtain a deformed content template feature map; then, the deformed content template feature map is weighted based on the attention coefficient to obtain an aggregated feature map. In other words, the aggregated feature map solves the problem of text glyph offset between the content template image and the target image using pixel offsets, and the attention coefficient represents the degree of importance given to each pixel offset.

[0131] In one possible implementation, the pixel offsets include m horizontal pixel offsets and m vertical pixel offsets, and the number of attention coefficients is m, where m is an integer greater than or equal to 1. The above-described processing of the content template feature map based on the pixel offsets and attention coefficients to obtain the aggregated feature map includes: performing bilinear sampling on the content template feature map based on the m horizontal and m vertical pixel offsets to obtain m deformed content template feature maps; and performing a weighted summation on the m deformed content template feature maps based on the m attention coefficients to obtain the aggregated feature map.

[0132] In this implementation, bilinear sampling is performed on the content template feature map based on m horizontal and m vertical pixel offsets to obtain m deformed content template feature maps. Then, based on m attention coefficients, the m deformed content template feature maps are weighted and summed to obtain an aggregated feature map. The m horizontal and m vertical pixel offsets are corresponding, equivalent to m pixel offset pairs, each pair including one horizontal and one vertical pixel offset. The content template feature map is bilinearly sampled m times using these m pixel offset pairs to obtain the m deformed content template feature maps. Each bilinear sampling operation shifts the pixels of the content template feature map in both the horizontal and vertical directions. In other words, by using m horizontal pixel offsets and m pixel offsets, the aggregated feature map solves the problem of the offset of the text shapes in the content template image and the target image in the horizontal and vertical directions; by using m attention coefficients to represent the degree of importance attached to the m deformed content template feature maps sampled m times, the final aggregated feature map can be the result of comprehensively considering the balance of m possible offsets.

[0133] It can be further understood that the pixel offset can be one or more. When there is only one pixel offset, an attention coefficient is not needed; the content template feature map is directly processed based on the pixel offset to obtain an aggregated feature map; and the output feature map is obtained based on the aggregated feature map and the input feature map. When there are multiple pixel offsets, an attention coefficient can be set; the content template feature map is sampled based on multiple pixel offsets to obtain multiple deformed content template feature maps; the multiple deformed content template feature maps are weighted according to the attention coefficient to obtain an aggregated feature map, which ensures that the final aggregated feature map is the result of comprehensively considering the balance of multiple offsets.

[0134] In one possible implementation, the above-mentioned process of obtaining pixel offsets and attention coefficients based on the content template feature map and the input feature map includes: concatenating the content template feature map and the input feature map to obtain a concatenated feature map; using the concatenated feature map as input to a convolutional network module, and outputting pixel offsets and attention coefficients.

[0135] In this implementation, the content template feature map and the input feature map are concatenated to obtain a concatenated feature map. This concatenated feature map is then used as input to a convolutional network module, which outputs pixel offsets and attention coefficients. In other words, the convolutional network module processes the concatenated feature map to obtain multiple pixel offsets and the corresponding attention coefficients for each pixel offset, thus solving the problem of calculating the text glyph offsets in the content template image.

[0136] In one possible implementation, the above-mentioned process of obtaining the output feature map based on the aggregated feature map and the input feature map includes: performing a residual connection on the aggregated feature map and the input feature map to obtain the output feature map.

[0137] In this implementation, a residual connection is performed between the aggregated feature map and the input feature map to obtain the output feature map. The output feature map after the residual connection includes both the aggregated feature map and the input feature map, which facilitates further processing of the aggregated feature map and the input feature map by downstream neural network layers.

[0138] In one possible implementation, guided by the content template edge image, a target image is generated based on the noisy image, including: obtaining the input feature map; encoding the content template edge image to obtain contour features; and obtaining the output feature map based on the contour features and the input feature map through a cross-attention mechanism.

[0139] In this implementation, for example, the guidance process of the content template edge image can be implemented through a neural network layer. The input of this neural network layer is the output of the previous neural network layer, i.e., the input feature map; another input of this neural network layer is the contour feature obtained by encoding the content template edge image; the output of this neural network layer is the input of the next neural network layer, i.e., the output feature map. This embodiment does not limit the position of the neural network layer; it can be upstream or downstream of the neural network layer corresponding to the guidance process of the style reference image and the content template image. Specifically, the neural network layer can implement: based on the contour feature and the first input feature map, a first output feature map is obtained through a cross-attention mechanism. By controlling the contour feature, the integrity of the geometric structure information of the characters in the target image, such as the outline, edges, and glyphs, is ensured, solving the problem of incomplete glyph structure in the target image and the problem of inaccurate style control caused by incomplete glyph structure. In other words, to address the problem of unstable font structure in the generated target image, this embodiment proposes introducing a specially designed neural network layer (such as a content geometry control layer) into the image encoder and image decoder of the diffusion model, using the content template edge image to control the stability of the font structure in the target image.

[0140] In one possible implementation, the first input feature map based on the contour features and the input is used to obtain the first output feature map through a cross-attention mechanism, which includes: using the input feature map as a query vector and the contour features as a key vector and a value vector, and calculating the first output feature map through cross-attention.

[0141] In this implementation, cross-attention is a method that updates the value vector based on the query vector and the key vector. In other words, the first output feature map is obtained by weighted summation of the value vectors, with the weights representing the similarity between the query vector and the key vector. Through the cross-attention mechanism, the input feature map and contour features can be mapped to the first output feature map, enabling the contour features to control the edge contours of text in the target image.

[0142] In one possible implementation, the above-mentioned generation of a target image based on a noisy image under the guidance of a style reference image includes: obtaining a third input feature map; encoding the style reference image to obtain style reference features; and processing the third input feature map and style reference features based on a cross-attention mechanism to obtain a third output feature map.

[0143] In this implementation, for example, the guidance process of the style reference image can be implemented through a neural network layer. The input of this neural network layer is the output of the previous neural network layer, i.e., the third input feature map; another input of this neural network layer is the style reference feature obtained by encoding the style reference image; the output of this neural network layer is the input of the next neural network layer, i.e., the third output feature map. This embodiment does not limit the position of the neural network layer; it can be upstream or downstream of the neural network layer corresponding to the guidance process of the content template image and the content template edge image. Specifically, the neural network layer can implement: processing the third input feature map and the style reference feature based on a cross-attention mechanism to obtain the third output feature map. By processing the third input feature map and the style reference feature using a cross-attention mechanism, control over the text style of the target image is achieved. Simultaneously, due to the guidance of the content template image and the content template edge image, the integrity of the geometric structure information such as the outline, edges, and glyphs of the text in the target image is ensured. Therefore, the guidance of the style reference image achieves accurate control of the text style when the glyph structure is complete, solving the problem of inaccurate style control caused by incomplete glyph structure in the target image.

[0144] In one possible implementation, the above-mentioned cross-attention mechanism-based processing of the third input feature map and style reference features to obtain the third output feature map includes: using the third input feature map as a query vector, using the style reference features as key vector and value vector, and calculating the third output feature map through cross-attention.

[0145] In this implementation, cross-attention is a method that updates the value vector based on the query vector and the key vector. In other words, the third output feature map is obtained by weighted summation of the value vectors, with the weights representing the similarity between the query vector and the key vector. Through the cross-attention mechanism, the third input feature map and style reference features can be mapped to the third output feature map, enabling the style reference features to control the text style in the target image.

[0146] In one possible implementation, obtaining the content template edge image from the content template image includes: performing edge extraction processing on the content template image to obtain the content template edge image.

[0147] This implementation provides a method for obtaining a content template edge image. Edge extraction processing is performed on the content template image to obtain the content template edge image. Edge extraction algorithms include the Canny operator, etc.

[0148] In one possible implementation, the above-mentioned generation of a target image based on a noisy image, guided by a content template image, a style reference image, and a content template edge image, includes: using the content template image as input to a content encoder and outputting a content template feature map; using the content template edge image as input to a contour encoder and outputting contour features; using the style reference image as input to a style encoder and outputting style reference features; obtaining predicted noise based on the noisy image, guided by the content template feature map, contour features, and style reference features, where the predicted noise represents the predicted value of noise in the noisy image; and obtaining the target image based on the noisy image and the predicted noise.

[0149] In this implementation, a content encoder, a contour encoder, and a style encoder encode the content template image, the content template edge image, and the style reference image, respectively, to obtain content template feature maps, contour features, and style reference features. Guided by these features, predicted noise is generated based on a noise image. Finally, the target image is obtained based on the noise image and the predicted noise. In other words, for example, the text generation model includes a content encoder, a contour encoder, and a style encoder to encode the content template image, the content template edge image, and the style reference image, respectively. The encoded features are then combined to generate the target image. This achieves control over the text content, text style, and text contour edges of the target image by the content template image, the content template edge image, and the style reference image. It ensures the integrity of the geometric structure information of the text in the target image, such as its outline, edges, and glyphs, solving the problem of incomplete glyph structure and inaccurate style control caused by incomplete glyph structure.

[0150] The training phase of the text generation model will be further illustrated below.

[0151] See Figure 4a , Figure 4a This is a flowchart illustrating a training method for a text generation model provided in an embodiment of this application. This training method can be understood as the training method for the text generation model during the training phase. Figure 4a As shown in the embodiment of this application, a method for training a text generation model mainly includes the following steps:

[0152] Step S410: Obtain training samples, which include ground truth samples of the target image X0, content template image samples, style reference image samples, and content template edge image samples.

[0153] Step S420: The training process includes T iterations, where T is an integer greater than or equal to 1. For each iteration, a time step t (1 ≤ t ≤ T) is randomly sampled, and corresponding noise is added. (This adds noise) Following a standard normal distribution, based on the ground truth sample X0 of the target image, time step t, and the noise added at time step t, the noisy image sample X at time step t is obtained using the forward noise addition process of the diffusion model. t .

[0154] Step S430: After N iterations, train the text generation model.

[0155] See Figure 4b , Figure 4b This is a flowchart illustrating the training process with a sampling time step of t, as provided in an embodiment of this application. Figure 4b As shown, the training process at sampling time step t includes:

[0156] Step S431, set time step t and noisy image sample X. t The text generation model takes content template image samples, style reference image samples, and content template edge image samples as inputs and outputs predicted noise e. t .

[0157] Step S432, based on the predicted noise e t and adding noise The first loss value is obtained.

[0158] Step S433: Process the ground truth sample X0 of the target image to obtain the corner image and edge image.

[0159] Step S434: Based on at least one of the corner image and the edge image, the first loss value is weighted to obtain the second loss value.

[0160] Step S435: With minimizing the second loss value as the training objective, optimize the parameters of the text generation model.

[0161] In this embodiment, during the training phase of the text generation model, a method is introduced to weight the first loss value using edge and corner images of ground truth samples of the target image. This allows the text generation model to pay more attention to the position of the text geometric structure in the input noisy image during the target image generation process, ensuring the positional accuracy of the text geometric structure in the target image and improving the integrity of the character structure and the accuracy of style control. In other words, during the training phase, by explicitly focusing on and utilizing the geometric structure information of handwritten characters, such as character outlines and corner points, the diffusion model generates target images with more complete character structures and more accurate style control when applied to handwritten font generation tasks.

[0162] In one possible implementation, the text generation model described above includes a noise prediction model, a content encoder, a contour encoder, and a style encoder. The above-mentioned time step t and noisy image sample X... t The text generation model takes content template image samples, style reference image samples, and content template edge image samples as inputs and outputs predicted noise e. t This includes: using content template image samples as input to a content encoder and outputting sample content template feature maps; using content template edge image samples as input to a contour encoder and outputting sample contour features; using style reference image samples as input to a style encoder and outputting sample style reference features; and using time step t and noisy image samples X... t The sample content template feature map, sample contour features, and sample style reference features are used as inputs to the noise prediction model, and the output is the predicted noise e. t The text generation model is optimized by minimizing the second loss value as the training objective. This includes optimizing the parameters of the noise prediction model, content encoder, contour encoder, and style encoder by minimizing the second loss value as the training objective.

[0163] In this implementation, a noise prediction model is used for noise prediction in the inverse denoising process of the diffusion model. The noise prediction model includes an image encoder and an image decoder, as well as an embedding encoding module that converts time step t into a vector. For example, the image encoder includes one or more encoding units, each containing at least one encoding block, which can be implemented using, but is not limited to, convolutional layers. For example, the image decoder includes one or more decoding units, each containing at least one decoding block, which can be implemented using, but is not limited to, convolutional layers. For example, a first neural network layer (e.g., a deformable content aggregation layer), a second neural network layer (e.g., a content geometry control layer), and a third neural network layer (e.g., a style control layer) are added to the image encoder to take into account sample content template feature maps, sample contour features, and sample style reference features, respectively, to control the encoding process; a fourth neural network layer (e.g., a content geometry control layer) and a fifth neural network layer (e.g., a style control layer) are added to the image decoder to take into account sample contour features and sample style reference features, respectively, to control the decoding process. Thus, in the image encoding and decoding process of the diffusion model, the sample content template feature map, sample contour features, and sample style reference features are used to control the prediction noise generation process. For example, the embedding encoding module processes the input time step t using a positional encoding method similar to that used in the Transformer model in deep learning, obtaining a time step embedding vector. This time step embedding vector is then mapped to a vector with a corresponding dimension in the encoding blocks (excluding controllable encoding blocks) of each encoding unit in the image encoder and the decoding blocks (excluding controllable decoding blocks) of each decoding unit in the image decoder. This vector is then superimposed on the feature maps of the outputs of each encoding and decoding block to regulate the iterative calculation process.

[0164] In one possible implementation, the above-mentioned weighting of the first loss value based on at least one of the corner image and the edge image to obtain the second loss value includes: weighting and summing the corner image and the edge image using a first weight and a second weight respectively to obtain image geometric structure weight coefficients, wherein the first weight and the second weight are values ​​greater than or equal to 0 and less than or equal to 1; and weighting the first loss value based on the image geometric structure weight coefficients to obtain the second loss value.

[0165] In this implementation, a method for weighting a first loss value by a corner image and an edge image is provided. First, the corner image and the edge image are weighted and summed respectively by a first weight and a second weight to obtain an image geometric structure weight coefficient; then, according to the image geometric structure weight coefficient, the first loss value is weighted to obtain a second loss value. Exemplarily, when the first weight is 0, it means that only the edge image is used to weight the first loss value; when the second weight is 0, it means that only the corner image is used to weight the first loss value; when the first weight is 0.9 and the second weight is 0.1, it means that when weighting the first loss value, more attention is paid to the influence of the corner image. Therefore, different weight values can be assigned to the first weight and the second weight to control the respective influence degrees of the corner image and the edge image in the process of weighting the first loss value.

[0166] See Figure 5 , Figure 5 is a schematic diagram of the training stage of a text generation model provided by an embodiment of this application. As Figure 5 shown, training samples are obtained. Among them, the training samples include target image truth samples, such as Figure 5 the image of the character "滞" in handwritten font shown. A time step t (1≤t≤T) is randomly sampled, and the corresponding added noise (subject to the standard normal distribution). According to the target image truth sample, the time step t, and the added noise corresponding to the time step t, through the processing of the noise addition module, the noisy image sample corresponding to the time step t is obtained.

[0167] Exemplarily, as Figure 5 shown, taking the training process at the sampled time step t as an example, the noisy image sample X t corresponding to the time step t, the content template image sample, the style reference image sample, and the content template edge image sample are used as the inputs of the text generation model, and the predicted noise corresponding to the time step t is output. The predicted noise corresponding to the time step t represents the predicted value of the noise in the noisy image sample corresponding to the time step t under the guidance of the content template image sample, the style reference image sample, and the content template edge image sample. According to the predicted noise corresponding to the time step t and the added noise corresponding to the time step t, through the processing of the loss calculation module, a first loss value is obtained. The target image truth sample is processed to obtain a corner image and an edge image. According to at least one of the corner image and the edge image, the first loss value is weighted to obtain a second loss value. Taking minimizing the second loss value as the training objective, the parameters of the text generation model are optimized and solved.

[0168] Among them, the training samples include content template image samples, such as Figure 5An image of the text content corresponding to the handwritten character "滞". That is, the text content of the content template image sample is the same as that of the target image true value sample, but the text style of the content template image sample is different from that of the target image true value sample, so that the trained text generation model has the learning ability to learn the text content from the content template image sample.

[0169] Among them, the training samples include style reference image samples, such as Figure 5 An image of the handwritten character "沼" corresponding to the text style of the handwritten character "滞" as shown. That is, the text style of the style reference image sample is the same as that of the target image true value sample, but the text content of the style reference image sample is different from that of the target image true value sample, so that the trained text generation model has the learning ability to learn the text style from the style reference image sample.

[0170] Among them, the training samples include content template edge image samples, such as Figure 5 An image of the geometric structure information of the text corresponding to the handwritten character "滞" as shown. That is, the text content of the content template edge image sample is the same as that of the target image true value sample; but the text style of the content template edge image sample can be the same as or different from that of the target image true value sample; that is, it only needs the geometric structure information of the text in the content template edge image sample to be the same as that of the text in the target image true value sample. Furthermore, the trained text generation model has the learning ability to learn the geometric structure information of the text from the content template edge image sample.

[0171] To further illustrate the text generation method as shown in Figure 2 、 Figure 3 and the training method of the text generation model as shown in Figure 4a 、 Figure 4b 、 Figure 5 Below, a specific example of the text generation model will be used to further exemplarily illustrate the text generation model, its text generation method, and training method in the embodiments of the present application. It should be further noted that this specific example does not limit the embodiments of the present application.

[0172] [[ID=二十六]]Refer to Figure 6 , Figure 6 which is a schematic diagram of an example of a text generation model provided by an embodiment of the present application. As shown in Figure 6As shown, in one example, a text generation model includes a noise prediction model, a content encoder, a contour encoder, and a style encoder. The noise prediction model includes an image encoder and an image decoder, as well as an embedding encoding module that converts time step t into a time step embedding vector. This application proposes a handwritten font generation method based on a text generation model. It takes a noisy image following a standard normal distribution as input, performs control based on glyph geometry guidance according to a given content template image and style reference image, and controls the iterative process using the time step embedding vector. Through iterative processing by the image encoder and image decoder, the corresponding handwritten text target image is generated.

[0173] like Figure 6 As shown in the example, this section provides an exemplary illustration of the data acquisition and preprocessing process. The acquired data includes a content template image and a style reference image. The content template image and style reference image are preprocessed. For example, an edge extraction operator (such as the Canny operator) is used to process the content template image to obtain a content template edge image. An initialized standard normal distribution noise image X is then acquired. T .

[0174] like Figure 6 As shown in the example, the glyph feature extraction process is illustrated. The content template image, style reference image, content template edge image, and noise image X are used as examples. T As input to the text generation model, the output is a target image containing handwritten text. The text generation model includes a content encoder, a contour encoder, a style encoder, an image encoder, and an image decoder. The content encoder processes the content template image to obtain multi-scale content template feature maps; the contour encoder processes the content template edge image to obtain contour features; and the style encoder processes the style reference image to obtain style reference features.

[0175] like Figure 6 As shown in the example, this section provides an exemplary description of controlling the handwritten text generation process using glyph features. During the inference phase, a pre-trained text generation model is used to process a standard normally distributed noisy image X. T Perform a reverse denoising process; iterate at time steps t = T, ..., 1. For example, when denoising a noisy image X... t The signal is fed into the image encoder, and the output of the image encoder is fed into the image decoder to obtain the predicted noise e. t Using time step t and prediction noise e t Based on the inverse process of the corresponding noise addition stage in the forward noise addition process of the diffusion model, for the noisy image X... t Image denoising is performed to obtain an image X with reduced noise. t-1 Among them, X TThe first image represents a noisy image that follows a standard normal distribution; X0 represents the target image of the handwritten text to be generated; the middle X... t A noisy image is an image with added noise.

[0176] like Figure 6 As shown, in this example, the image encoder includes multiple encoding units, each encoding unit containing an encoding block and an optional controllable encoding block. The controllable encoding block includes a deformable content aggregation layer, a content geometry control layer, and a style control layer. The image decoder includes multiple decoding units, each decoding unit containing a decoding block and an optional controllable decoding block. The controllable decoding block includes a content geometry control layer and a style control layer.

[0177] like Figure 6 As shown, in this example, the embedding encoding module processes the input time step t using a positional encoding method similar to that used in the Transformer model in deep learning, to obtain a time step embedding vector. This time step embedding vector is then mapped to a vector of a single dimension within the encoding blocks (excluding controllable encoding blocks) of each encoding unit in the image encoder and the decoding blocks (excluding controllable decoding blocks) of each decoding unit in the image decoder. This mapping is performed on the feature maps of the outputs of each encoding and decoding block to control the iterative computation process.

[0178] For example, an image encoder includes four coding units; each of the four coding units contains one coding block; the first coding unit does not contain a controllable coding block, while the second, third, and fourth coding units contain corresponding controllable coding blocks. Figure 5 As shown, the coding unit is obtained by cascading M steps, where M≥2 indicates that the coding unit includes at least the first and second coding units. In this example, M=4, meaning the image encoder contains 4 coding units. Figure 5 As shown, the controllable coding block in the image encoder is an optional part.

[0179] For example, the first coding unit does not contain a controllable coding block, while the second, third, and fourth coding units contain the corresponding controllable coding blocks.

[0180] For example, an image decoder includes four decoding units; each of the four decoding units contains one decoding block; the first and fourth decoding units do not contain a controllable decoding block, while the second and third decoding units contain the corresponding controllable decoding block. Figure 5 As shown, the decoding units are obtained through M cascaded steps, where M≥2 indicates that the decoding unit includes at least the first and second decoding units. In this example, M=4, meaning the image decoder contains 4 decoding units. Figure 5As shown, the controllable decoding block in the image decoder is an optional part. For example, the first and fourth decoding units do not contain controllable decoding blocks, while the second and third decoding units contain the corresponding controllable decoding blocks.

[0181] For example, the coding blocks in the first, second, and third coding units each contain 4 convolutional layers; the coding block in the fourth coding unit contains 6 convolutional layers; the decoding blocks in the first, second, and third decoding units each contain 6 convolutional layers; and the decoding block in the fourth decoding unit contains 7 convolutional layers. The convolutional kernel size of the convolutional layers is 3×3, the stride is 1×1, and SiLU is used as the non-linear activation function for the convolutional layers.

[0182] For example, the noisy image X at the current time step t The first to fourth encoded feature maps and the first to fourth decoded feature maps are obtained sequentially through each encoding and decoding unit. The noisy image X at the current time step... t The dimension size is H×W×3, and the dimensions of the 1st to 4th encoded feature maps are respectively... The dimensions of the first to fourth decoded feature maps are respectively H×W×128、H×W×3.

[0183] Optionally, the controllable encoding block includes a deformable content aggregation layer, a content geometry control layer, and a style control layer; the controllable decoding block includes a content geometry control layer and a style control layer.

[0184] The following uses a controllable coding block as an example to illustrate the working mechanism of the deformable content aggregation layer, content geometry control layer, and style control layer. The content geometry control layer and style control layer in the controllable decoding block are similar to those in the controllable coding block. Therefore, the relevant descriptions of the content geometry control layer and style control layer in the controllable decoding block will not be repeated in this embodiment; please refer to the descriptions of the content geometry control layer and style control layer in the controllable coding block.

[0185] See Figure 7 , Figure 7 This is a schematic diagram of a deformable content aggregation layer provided in an embodiment of this application. Figure 7 As shown, the input to the deformable content aggregation layer is the encoded feature map output by the encoded block. Multiple pixel offsets and weighting coefficients are predicted by the encoded feature map and the content template feature map of the corresponding scale. The content template feature map of the corresponding scale is processed by the multiple pixel offsets and weighting coefficients to obtain the aggregated feature map. The encoded feature map is added to the aggregated feature map through residual connection to obtain the output feature map of the deformable content aggregation layer.

[0186] Furthermore, such as Figure 7 As shown, the deformable content aggregation layer concatenates the encoded feature map output from the encoding block and the content template feature map of the corresponding scale along the feature dimension. The concatenated feature map is then fed into a regular convolutional block. For all pixel positions on the feature map, m horizontal offsets, m vertical offsets, and m attention coefficients are obtained. Based on the horizontal and vertical offsets, the content template feature map of the corresponding scale is bilinearly sampled to obtain m deformed content template feature maps. The m deformed content template feature maps are then weighted and summed using the m attention coefficients to obtain an aggregated feature map. The encoded feature map is then added to the aggregated feature map through a residual connection to obtain the output feature map of the deformable content aggregation layer.

[0187] Optionally, the deformable content aggregation layer includes two input convolutional blocks (CNNs). I1 CNN I2 ), 2 intermediate convolutional blocks (CNN) M1 CNN M2 ) and 1 output convolutional block (CNN) O Specifically, the input content template feature map f ct The size of the feature map is h×w×c1, and the size of the encoding feature map f is h×w×c2. The content template feature map and the encoding feature map are respectively input into two input convolutional blocks to transform the feature dimensions so that they are the same. The size of the transformed feature map is h×w×c. The calculation formulas are formula (1) and formula (2).

[0188] f = CNN I1 (f) Formula (1)

[0189] f ct =CNN I2 (f ct ) Formula (2)

[0190] Optionally, each input convolutional block contains one convolutional layer and its corresponding group normalization layer and SiLU activation layer. The kernel size and stride of the convolutional layer are both 1×1. The transformed content template feature map and the encoded feature map are concatenated along the feature dimension. The concatenation formula is shown in formula (3).

[0191] f a =Concat(f,f ct ) Formula (3)

[0192] Optionally, the stitched feature map f aThey are respectively input into two intermediate convolutional blocks to obtain the two-dimensional offset Δp=(Δx, Δy) of the pixel positions of the deformable sampling points and the attention coefficient α, with the dimension sizes of h×w×m×2 and h×w×m respectively. That is, each pixel on the feature map corresponds to m deformable sampling points. For example, m = 16. The calculation formula of the two-dimensional offset Δp of the pixel position is shown in formula (4), and the calculation formula of the attention coefficient α is shown in formula (5).

[0193] Δp = CNN M1 (f a ) Formula (4)

[0194] α = softmax(CNN M2 (f a )) Formula (5)

[0195] Optionally, each intermediate convolutional block contains 2 convolutional layers, and the convolutional kernel size of the convolutional layer is 3×3, and the stride is 1×1. Among them, softmax(·) represents the softmax activation function.

[0196] Optionally, for any pixel point (each integer coordinate position p=(i, j), 0≤i<w, 0≤j<h) on the h×w feature map, according to the pixel position offsets {Δp (k) , k = 1, …, m} of the corresponding m deformable sampling points, bilinear sampling is performed on the content template feature map to obtain m deformed content template feature maps {f ct (p + Δp (k) ), k = 1, …, m}, and the deformed content template feature maps are weighted and summed using the attention coefficients {α (k) , k = 1, …, m} of the corresponding m deformable sampling points. Then, a residual connection is made with the transformed encoded feature map to obtain the output feature map f′ of the deformable content aggregation layer. The calculation formula of the output feature map f′ is shown in formula (6).

[0197]

[0198] As Figure 6 shown, the input of the content geometric structure control layer in the controllable encoding block is the output feature map of the corresponding deformable content aggregation layer, and the input of the content geometric structure control layer in the controllable decoding block is the decoded feature map output by the corresponding decoding block. By using the contour features, the input feature map is updated using the cross-attention mechanism, and the updated feature map is used as the output feature map of the content geometric structure control layer. The content geometric structure control layer in the controllable encoding block is described exemplarily below. The content geometric structure control layer in the controllable decoding block can refer to this part of the description, and the content geometric structure control layer in the controllable decoding block will not be described repeatedly.

[0199] Optionally, the content geometry control layer uses the output feature map of the deformable content aggregation layer as a query vector and the contour features as key and value vectors, and performs cross-attention calculation to obtain the output feature map of the content geometry control layer.

[0200] For example, the content geometry control layer includes a cross-attention layer. In this embodiment, the cross-attention is calculated using the conventional cross-attention method of Transformers in deep learning. The input dimension of the cross-attention layer is the same as the feature dimension of the input content aggregation encoding feature map (controllable encoding block) or decoding feature map (controllable decoding block), employing a single-head attention mechanism (i.e., one head). The activation function used is GEGLU, which is a variation of the GELU activation function. Specifically, the output feature map f′ of the deformable content aggregation layer is used as the query vector, and the contour feature f′... ce As key vectors and value vectors, they are input to the cross-attention layer to calculate the output feature map f″ of the content geometry control layer. The calculation formula is shown in formula (7).

[0201] f″=CA(q=f′,k=f ce v = f ce ) Formula (7)

[0202] In formula (7), CA(·) represents the Cross Attention (CA) layer, and q, k, and v represent the query vector, key vector, and value vector, respectively. Attention is a method that maps a query and a key-value pair to the output. q, k, and v are all vectors, and the output is obtained by weighted summation of v. The weights are the similarity between q and k.

[0203] like Figure 6 As shown, the input to the style control layer in the controllable coding block or controllable decoding block is the output feature map of the corresponding content geometry control layer. By utilizing style reference features, the input feature map is updated using a cross-attention mechanism, and the updated feature map is used as the output feature map of the style control layer.

[0204] Optionally, the style control layer uses the output feature map of the content geometry control layer as a query vector and the style reference features as key and value vectors, and performs cross-attention calculation to obtain the output feature map of the style control layer, which is the encoding feature map output by the current encoding unit.

[0205] For example, the style control layer includes a cross-attention layer, whose structure and parameters are the same as the cross-attention layer in the content geometry control layer, and will not be repeated here. Specifically, the output feature map f″ of the content geometry control layer is used as the query vector, and the style reference feature f... st As key vectors and value vectors, they are input into the cross-attention layer to calculate the output feature map f″′ of the style control layer. The calculation formula is shown in formula (8).

[0206] f″′=CA(q=f″,k=f st v = f st ) Formula (8)

[0207] Optionally, the content encoder comprises three basic convolutional blocks. Each convolutional block contains two convolutional layers and one average pooling layer, using ReLU as the non-linear activation function for the convolutional layers. The kernel size of the convolutional layers is 3×3, and the stride is 1×1. The number of output channels of the convolutional layers in the three basic convolutional blocks are 64, 128, and 256, respectively.

[0208] For example, the content template image is input into the content encoder for feature extraction to obtain a multi-scale content template feature map. Specifically, the input is a color RGB three-channel content template image x. ct The size is H×W×3, where 3 represents the number of image channels, and the image height H and width W are set to 96. The content template image is sequentially passed through the three basic convolutional blocks to obtain three content template feature maps. The dimensions are respectively The content template feature maps of the corresponding scales are utilized in the deformable content aggregation layer contained in the corresponding controllable coding block in the image encoder.

[0209] Optionally, the contour encoder comprises five basic convolutional blocks. Each convolutional block contains two convolutional layers and one average pooling layer, using ReLU as the non-linear activation function for the convolutional layers. The kernel size of the convolutional layers is 3×3, and the stride is 1×1. The number of output channels of the convolutional layers in the five basic convolutional blocks are 64, 128, 256, 512, and 1024, respectively.

[0210] For example, the content template edge image is input into a contour encoder for feature extraction to obtain contour features. Specifically, the input is a color RGB three-channel content template edge image x. ce The dimensions are H×W×3. The content template edge image is processed through 5 basic convolutional blocks to obtain the contour features f. ce The dimension size is

[0211] Optionally, the style encoder and the contour encoder have the same structure and share parameters.

[0212] For example, the style reference image is input into the style encoder for feature extraction to obtain style reference features. Specifically, the input is a color RGB three-channel style reference image x. st The dimensions are H×W×3. The style reference image is processed through 5 basic convolutional blocks to obtain the style reference feature f. st The dimension size is

[0213] Optionally, during the testing phase, for the initial noisy image X that follows a standard normal distribution... T By utilizing multi-scale content template feature maps, contour features, and style reference features, an image encoder and decoder perform T iterations at time steps t = T, ..., 1 (e.g., T = 1000). Based on the input time step t, the time step is mapped to a vector. The computational results are then adjusted by superimposing this vector onto the feature map in the encoder and decoder, ultimately yielding the generated handwritten text target image X0. Figure 5 As shown, the input for each iteration is the noisy image X at the current time step t of the corresponding forward noise addition process. t The output is the noisy image X at time step t-1 of the forward noise addition process. t-1 Continue until the generated handwritten text target image X0 is output.

[0214] The technical solution of this application embodiment in the training stage of the text generation model is described below.

[0215] In this embodiment, during the training phase, the model (the text generation model before training) undergoes parameter initialization. A time step t (1≤t≤T) is randomly sampled, and noise following a standard normal distribution is added to the ground value samples of the target image according to time step t through a forward noise addition process. Obtain the noisy image corresponding to time step t; feed time step t and the corresponding noisy image into the model, map the time step to a vector based on the input time step t, and adjust the calculation results by superimposing them on the feature map in the encoder and decoder to obtain the noise prediction e added at time step t. t The noise prediction e at step t t The noise actually added in step t The mean squared error is used as the first loss value. For example, the ground truth sample of the target image is processed using an edge extraction operator (such as the Canny operator) and a corner detection operator (such as the Harris corner detection operator) to obtain the edge image and corner image corresponding to the ground truth sample of the target image, respectively; the first loss value is weighted using the edge image and corner image of the ground truth sample of the target image to obtain the weighted second loss value L. w Using the second loss value L wMinimize and iteratively optimize and solve the parameters of the model.

[0216] See Figure 8 , Figure 8 is a schematic diagram of an example of the corner image and edge image provided by the embodiment of the present application. As Figure 8 shown, in the visualization schematic diagram of the weighting coefficient of the first loss value, the positions with larger weighting coefficients in the corner image are the text corners of the handwritten font "gua" character, the inflection points of handwritten strokes, etc. in the target image true value sample; the positions with larger weighting coefficients in the edge image are the glyph outlines of the handwritten font "gua" character in the target image true value sample; mixing the corner image and the edge image to obtain a mixed image, and the positions with larger weighting coefficients in the mixed image are the text corners of the handwritten font "gua" character, the inflection points of handwritten strokes, the glyph outlines, etc. in the target image true value sample. Therefore, using the corresponding weighting coefficients of the corner image and the edge image to weight the first loss value, the weighted second loss value L<00001�2>is obtained, so that the model pays more attention to the explicit attention and utilization of the handwritten text geometric structure information such as text corners, handwritten stroke inflection points, and glyph outlines during the training process. After the model training is completed, the obtained text generation model can analyze the handwritten text geometric structure information such as glyph outlines and text corners in the input noisy image, so that the glyph outlines and text corners of the output image are complete, and the glyph style can be accurately controlled to meet the style requirements, avoiding style loss or incomplete style caused by the lack of some glyph structures.

[0217] Exemplarily, in the training stage, in order to make the model pay attention to the glyph geometric structure information during the handwritten text generation process, the geometric information weighted reconstruction loss function L w is adopted, and the calculation formula is formula (9).

[0218]

[0219] Among them, represents taking the expectation of a batch of data during the training process, ||·|| 2 represents the norm of a vector or matrix, e t is the predicted noise, is the noise actually added in the t-th time step of the forward noise addition process of the diffusion model, ⊙ represents element-wise multiplication, represents the weight function calculated using the target image true value sample , and the calculation formula is formula (10).

[0220]

[0221] Among them, The target image is the ground truth sample. Harris(·) and Canny(·) are corner detection and edge extraction algorithms, respectively. The return values ​​are the binarized corner image and edge image, λ. corner , λ edge and λ all These are hyperparameters, representing the weights for corner locations, edge locations, and a default weight, used to adjust the weights of different geometric features. These hyperparameters can be chosen empirically or obtained through a hyperparameter grid search.

[0222] For example, the experiments compared conventional image reconstruction loss, corner-weighted image reconstruction loss, edge-weighted image reconstruction loss, and image reconstruction loss based on a combination of corner and edge weights. In this example, training the model using corner-weighted image reconstruction loss yielded the best generation results. Specifically, when calculating the corner-weighted image reconstruction loss, the weight λ of the corner position... corner The weight λ at the edge position is 0.9. edge The default weight λ is 0. all The value is 0.1; when calculating the edge-weighted image reconstruction loss, the weight λ at the edge location is... edge The weight λ of the corner position is 0.9. corner The default weight λ is 0. all The weight λ of the corner location is 0.1; when calculating the image reconstruction loss based on a combination of corner and edge weights. corner The weight λ at the edge position is 0.9. edge The default weight λ is 0.9. all The value is 0.1. Alternatively, this example can be implemented using a deep learning framework on a computer with a graphics processing unit (GPU) deployed.

[0223] For example, 3755 characters each from 100 different writers were randomly sampled from the first handwritten Chinese dataset as the training set, and 5 characters each from 60 different writers in the second handwritten Chinese dataset were randomly sampled as the test set. In addition, 5 characters were randomly selected from each of the 100 writers in the training set to verify the model's font generation effect on the training set. During model training, the batch size (BatchSize) for GPU parallel computing in this embodiment was set to 16, the number of iterations (N) was set to 80000, and the learning rate was 10. -4 An optimizer with parameters β1 = 0.9 and β2 = 0.999.

[0224] Optionally, during the training phase, it is also necessary to preprocess the ground truth samples of the target image. For example, preprocessing includes image size normalization, pixel value normalization, etc.

[0225] To further illustrate the beneficial effects of the technical solutions in the embodiments of this application, the methods proposed in the embodiments of this application were also verified through relevant experiments. Analysis of the experimental results verified the beneficial effects achieved by the methods provided in the embodiments of this application.

[0226] In one example, the commonly used technical evaluation metrics for text generation employed in the experiments of this application embodiment include Fréchet distance (Fréchet Inception Distance, FID), Learned Perceptual Image Patch Similarity (LPIPS), Structural Similarity Index Measure (SSIM), and L1 error. Tables 1, 2, and 3 present the experimental results of this application embodiment.

[0227] In this example, Table 1 compares the experimental results on the training and test sets regarding whether the deformable content aggregation layer, content geometry control layer, and geometric information weighted loss are used in this embodiment. "X" indicates no use, and "√" indicates use. In the first row, the model with "X" for "deformable content aggregation layer," "content geometry control layer," and "geometric information weighted loss" is the baseline model. Optionally, the baseline model is implemented using a basic diffusion model structure. The baseline model's content aggregation layer directly concatenates the encoded feature map with the corresponding content template feature map, without using either the deformable content aggregation layer or the content geometry control layer. During training, the loss function uses the unweighted mean square error loss of the original image reconstruction. In this embodiment, the training phase uses corner-weighted image reconstruction loss. The content template font is all boldface. The experimental results show that the methods in this embodiment can improve the generation effect of handwritten text images, and the methods in this embodiment outperform the baseline model in all indicators.

[0228] Table 1. Experimental results of the method of this application embodiment on the training set and test set.

[0229]

[0230]

[0231] Table 2 shows the experimental results on the training and test sets when different content template fonts were used in the embodiments of this application. The experimental results show that the handwritten text generation effect is best when using sans-serif as the content template font.

[0232] Table 2. Experimental results of different content template fonts on the training and test sets.

[0233]

[0234] Table 3 shows the experimental results on the training and test sets when different geometric information-weighted image reconstruction losses (examples of the second loss value) are used in the embodiments of this application. The experimental results show that the model achieves the best font generation effect when using corner-weighted image reconstruction loss.

[0235] Table 3 shows the experimental results on the training and test sets with different geometric information weightings.

[0236]

[0237] See Figure 9 , Figure 9 This is a schematic diagram illustrating an example of handwritten text generation results provided in an embodiment of this application. For example... Figure 9 As shown in the handwritten text generation result example of this application embodiment, the handwritten font generation example of this application embodiment is compared with the baseline model on the test set, indicating that the style of the font generated by this application embodiment is closer to the style reference image than the baseline model, especially with better control over the connected font structure.

[0238] In this embodiment, the handwritten font generation method based on the text generation model, by adopting a main framework including a content encoder, a style encoder, a contour encoder, an image encoder, and an image decoder, can effectively generate handwritten fonts with specified content and style, providing a feasible solution for controlling the glyph structure and writing style in handwritten font generation.

[0239] Next, based on the methods in the above embodiments, a text generation system provided in this application will be further described.

[0240] See Figure 10 , Figure 10 This is a schematic diagram illustrating the composition of a text generation system provided in an embodiment of this application. For example... Figure 10 As shown, a text generation system 1000 mainly includes: a data acquisition module 1010 and a text generation model 1020. The text generation model 1020 further includes: a content encoder 1021, a contour encoder 1022, a style encoder 1023, an image encoder 1024, and an image decoder 1025.

[0241] The data acquisition module 1010 is used to acquire data, including content template images, style reference images, noise images, and content template edge images.

[0242] Optionally, the data acquisition module 1010 is further configured to: preprocess the content template image and style reference image, for example, preprocessing includes image size normalization and pixel value normalization; process the content template image using an edge extraction operator (such as the Canny operator) to obtain a content template edge image; and process noisy images, for example, randomly initializing a noisy image X that follows a standard normal distribution. T .

[0243] Text generation model 1020 is used to generate target images containing handwritten text based on the input content template image, content template edge image, and style reference text during the testing phase. Text generation model 1020 is trained during the training phase using training set sample data containing ground truth values ​​for the content template image, style reference image, and handwritten text target image.

[0244] The content encoder 1021 is used to extract features from the content template image to obtain a multi-scale content template feature map.

[0245] Contour encoder 1022 is used to extract contour features from the edge image of the content template.

[0246] The style encoder 1023 is used to extract style reference features from the style reference image.

[0247] Image encoder 1024 and image decoder 1025 control the text generation process using multi-scale content template features, contour features and style reference features, and perform an inverse denoising process on noisy images to obtain handwritten text target images.

[0248] In this embodiment, the text generation system based on a text generation model, employing a main framework including a content encoder, style encoder, contour encoder, image encoder, and image decoder, can effectively generate handwritten fonts with specified content and style, providing a feasible solution for controlling glyph structure and writing style in handwritten font generation. The handwritten font generation method and system based on a text generation model in this embodiment adopts a structure with a content encoder, contour encoder, style encoder, image encoder, and image decoder, which can guide the handwritten text generation process using glyph geometric information, resulting in handwritten text images with a relatively realistic writing style and stable glyph structure.

[0249] Next, based on the methods in the above embodiments, a text generation device and a text generation model training device provided in this application will be introduced.

[0250] See Figure 11 , Figure 11 This is a schematic diagram illustrating the composition of a text generation device provided in an embodiment of this application. Figure 11 As shown, this application embodiment provides a text generation device 1100, which mainly includes:

[0251] The acquisition module 1110 is used to acquire a content template image, a style reference image, a content template edge image, and a noise image. The content template image is used to indicate the content of the target text, the style reference image is used to indicate the style of the target text, and the content template edge image is used to indicate the geometric structure information of the target text.

[0252] The processing module 1120 is used to generate a target image of the target text based on the noise image, guided by the content template image, the style reference image and the content template edge image.

[0253] In one possible implementation, the processing module 1120 is specifically used to: control the content of the target text based on the content template image; control the geometric structure information of the target text based on the content template edge image; and control the style of the target text based on the style reference image.

[0254] In one possible implementation, the processing module 1120 is specifically used to: take the content template edge image as input to the contour encoder and output contour features; based on the contour features and the first input feature map, obtain the first output feature map through a cross-attention mechanism; and control the geometric structure information of the target text based on the first output feature map.

[0255] In one possible implementation, the processing module 1120 is specifically used to: take the first input feature map as the query vector, take the contour features as the key vector and value vector, and calculate the first output feature map through cross attention.

[0256] In one possible implementation, the processing module 1120 is specifically used for: taking the content template image as input to the content encoder and outputting a content template feature map; obtaining pixel offsets and attention coefficients based on the content template feature map and the input second input feature map; processing the content template feature map based on the pixel offsets and attention coefficients to obtain an aggregated feature map; obtaining an output second output feature map based on the aggregated feature map and the input second output feature map; and controlling the content of the target text based on the second output feature map.

[0257] In one possible implementation, the pixel offset includes m horizontal pixel offsets and m vertical pixel offsets, and the number of attention coefficients is m, where m is an integer greater than or equal to 1. The processing module 1120 is specifically used for: performing bilinear sampling on the content template feature map based on the m horizontal and m vertical pixel offsets to obtain m deformed content template feature maps; and performing weighted summation on the m deformed content template feature maps based on the m attention coefficients to obtain an aggregated feature map.

[0258] In one possible implementation, the processing module 1120 is specifically used to: concatenate the content template feature map and the input second input feature map to obtain a concatenated feature map; perform convolution calculation on the concatenated feature map, and output pixel offset and attention coefficient.

[0259] In one possible implementation, the processing module 1120 is specifically used to: perform a residual connection between the aggregated feature map and the input second input feature map to obtain a second output feature map.

[0260] In one possible implementation, the processing module 1120 is specifically used to: take the style reference image as input to the style encoder and output style reference features; and process the third input feature map and style reference features based on a cross-attention mechanism to obtain a third output feature map.

[0261] In one possible implementation, the content template edge image is obtained by edge extraction from the content template image.

[0262] In one possible implementation, the target image generation process is achieved through T iterations, where T is an integer greater than or equal to 1. The loop control variable t is the current time step, which changes in descending order from T to 1, the opposite of the time step change in the forward noise addition process of the corresponding diffusion model. The processing module 1120 is specifically used for: acquiring a noisy image, which is the image output from the previous loop iteration; taking the current time step t, the noisy image, the content template image, the style reference image, and the content template edge image as input to the text generation model; and outputting the predicted noise at the t-th time step, guided by the content template image, the style reference image, and the content template edge image, where the predicted noise at the current time step represents the predicted value of the noise in the noisy image input at the current time step; and obtaining a denoised image based on the noisy image and the predicted noise at the current time step, which is used as the input for the next time step.

[0263] See Figure 12 , Figure 12 This is a schematic diagram illustrating the composition of a training device for a text generation model provided in an embodiment of this application. For example... Figure 12 As shown, this application embodiment provides a text generation model training device 1200, including:

[0264] The sample acquisition module 1210 is used to acquire training samples, which include ground truth samples of the target image, content template image samples, style reference image samples, and content template edge image samples. The content template image samples are used to indicate the content of the text in the ground truth samples of the target image, the style reference image samples are used to indicate the style of the text in the ground truth samples of the target image, and the content template edge image samples are used to indicate the geometric structure information of the text in the ground truth samples of the target image.

[0265] Preprocessing module 1220, for each iteration in the training process, randomly samples a time step t (1≤t≤T) and adds corresponding noise. (This adds noise) Following a standard normal distribution, based on the ground truth sample X0 of the target image, time step t, and the noise added at time step t, the noisy image sample X at time step t is obtained using the forward noise addition process of the diffusion model. t When t = T, X T The noise image that ultimately follows a standard normal distribution is obtained from the forward noise addition process of the diffusion model.

[0266] Training module 1230 is used to train the text generation model. The text generation model is obtained through N iterations of training. The training process of the nth (1≤n≤N) iteration includes: processing the time step t and the noisy image sample X output by the preprocessing module. t Add noise The content template image sample, style reference image sample, and content template edge image sample output by the sample acquisition module are used as inputs to the text generation model, and the output is the predicted noise e. t Predicted noise e t This indicates that, guided by the content template image sample, style reference image sample, and content template edge image sample, the noisy image sample X... t Adding noise The predicted value; based on the predicted noise e t and the added noise A first loss value is obtained; and the ground truth sample of the target image is processed to obtain a corner image and an edge image, wherein the corner image is used to indicate the corner position information of the text in the ground truth sample of the target image, and the edge image is used to indicate the edge position information of the text in the ground truth sample of the target image; and the first loss value is weighted according to at least one of the corner image and the edge image to obtain a second loss value; and the parameters of the text generation model are optimized by minimizing the second loss value as the training objective.

[0267] In one possible implementation, the aforementioned text generation model includes a noise prediction model, a content encoder, a contour encoder, and a style encoder. The training module 1230 is specifically used for: taking content template image samples as input to the content encoder and outputting a sample content template feature map; taking content template edge image samples as input to the contour encoder and outputting sample contour features; taking style reference image samples as input to the style encoder and outputting sample style reference features; and, guided by the sample content template feature map, sample contour features, and sample style reference features, taking the noisy image sample corresponding to time step t of the nth iteration sampling as input to the noise prediction model and outputting the predicted noise corresponding to time step t of the nth iteration sampling. The training objective is to minimize the second loss value to optimize the parameters of the text generation model, including: optimizing the parameters of the noise prediction model, content encoder, contour encoder, and style encoder with the goal of minimizing the second loss value.

[0268] In one possible implementation, the noise prediction model includes an image encoder and an image decoder, as well as an embedding encoding module that converts time step t into a vector. The training module 1230 is specifically used for: converting time step t into a time step embedding vector; taking the time step embedding vector, noisy image samples, sample content template feature maps, sample contour features, and sample style reference features as input to the image encoder, and outputting image encoded features; taking the time step embedding vector, image encoded features, sample contour features, and sample style reference features as input to the image decoder, and outputting predicted noise.

[0269] In one possible implementation, the training module 1230 is specifically used to: perform weighted summation on the corner image and the edge image using a first weight and a second weight respectively to obtain the image geometric structure weight coefficient, wherein the first weight and the second weight are values ​​greater than or equal to 0 and less than or equal to 1; and weight the first loss value according to the image geometric structure weight coefficient to obtain the second loss value.

[0270] The following is about... Figure 11 A text generation device 1100 as shown and / or such as Figure 12 The software and hardware implementation of the text generation model training device 1200 (hereinafter referred to as: the device related to the text generation method) shown needs further explanation.

[0271] As an example of a software functional unit, a module can include code running on a computing instance. A computing instance can include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the aforementioned computing instance can be one or more. For example, a module can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region can include multiple AZs.

[0272] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0273] As an example of a hardware functional unit, a module may include at least one computing device, such as a server. Alternatively, a module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0274] The multiple computing devices included in the module can be distributed within the same region or in different regions. Similarly, the multiple computing devices included in the module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0275] It should be noted that, in other embodiments, the related apparatus of the text generation method is additionally provided with one or more modules for performing any of the steps included in the above implementation. The steps implemented by one or more modules in the related apparatus of the text generation method can be specified as needed, and more or fewer modules can be obtained than in the embodiments of this application to implement different steps in the above method, thereby realizing all the functions of the related apparatus of the text generation method.

[0276] This application also provides a computing device 1300. For example... Figure 13 As shown, the computing device 1300 includes a bus 1302, a processor 1304, a memory 1306, and a communication interface 1308. The processor 1304, memory 1306, and communication interface 1308 communicate with each other via the bus 1302. The computing device 1300 can be a server, such as a central server, an edge server, or a local server in a local data center, or it can be an electronic device such as a desktop computer, laptop computer, or smartphone. It should be understood that this application does not limit the number of processors and memories in the computing device 1300.

[0277] Bus 1302 can be a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The Unified Bus is also known as the Lingqu Bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 13The bus 1304 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1304 may include a path for transmitting information between various components of the computing device 1300 (e.g., memory 1306, processor 1304, communication interface 1308).

[0278] The processor 1304 may include any one or more of the following computing devices: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, SoC, offload card, accelerator card, etc.

[0279] Memory 1306 may include volatile memory, such as random access memory (RAM). Processor 1304 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 1306 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0280] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 1306, or two or more types of storage media can be configured to realize the function of memory 1306. This application does not limit this.

[0281] The memory 1306 stores executable program code, and the processor 1304 executes the executable program code to implement the functions of the relevant devices of the text generation method, thereby implementing the method described in the above embodiments. That is, the memory 1306 stores instructions for executing the method described in the above embodiments.

[0282] Alternatively, the memory 1306 may store executable code, which the processor 1304 executes to implement the functions of the relevant devices for the text generation method, thereby implementing the method described in the above embodiments. That is, the memory 1306 stores instructions for executing the method described in the above embodiments.

[0283] The communication interface 1308 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1300 and other devices or communication networks.

[0284] As one possible implementation, the computing device 1300 may also include a chip system, which includes a processor and a power supply circuit. The power supply circuit supplies power to the processor, and the processor executes the operation steps corresponding to the methods in the embodiments of this application. For simplicity, further details are omitted here. The processor can be implemented using a GPU, or it can be implemented using computing devices or AI chips such as a DPU, NPU, XPU, SoC, offloading card, or accelerator card.

[0285] As one possible implementation, the computing device 1300 may include various types of processors 1304, that is, the computing device 1300 is a heterogeneous device. For example, the computing device 1300 includes a CPU and a GPU, and at least one of the processors 1304 can execute the operation steps corresponding to the method of the embodiments of this application. For the sake of brevity, further details are omitted here.

[0286] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be an electronic device such as a desktop computer, a laptop computer, or a smartphone.

[0287] like Figure 14 As shown, the computing device cluster includes at least one computing device 1300. The memory 1306 of one or more computing devices 1300 in the computing device cluster may store the same instructions for performing the methods described in the above embodiments.

[0288] In some possible implementations, the memory 1306 of one or more computing devices 1300 in the computing device cluster may also store partial instructions for executing the methods described in the above embodiments. In other words, a combination of one or more computing devices 1300 can jointly execute instructions for performing the methods described in the above embodiments.

[0289] It should be noted that the memory 1306 in different computing devices 1300 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the relevant apparatus of the text generation method. That is, the instructions stored in the memory 1306 of different computing devices 1300 can implement the functions of one or more modules of the relevant apparatus of the text generation method.

[0290] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 15 One possible implementation is shown. For example... Figure 15 As shown, the two computing devices 1300A and 1300B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1306 in computing device 1300A stores instructions for the functions of one or more modules of the relevant apparatus for the text generation method. Simultaneously, the memory 1306 in computing device 1300B stores instructions for the functions of another one or more modules of the relevant apparatus for the text generation method.

[0291] It should be understood that Figure 15 The functions of computing device 1300A shown can also be performed by multiple computing devices 1300. Similarly, the functions of computing device 1300B can also be performed by multiple computing devices 1300.

[0292] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 14 and Figure 15 The connection method of the computing device cluster is different in that the memory 1306 of one or more computing devices 1300 in the computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0293] In some possible implementations, the memory 1306 of one or more computing devices 1300 in the computing device cluster may also store partial instructions for executing the aforementioned data processing method. In other words, a combination of one or more computing devices 1300 can jointly execute instructions for performing the aforementioned method.

[0294] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When executed by a cluster of computing devices including at least one computing device, the computer program instructions cause the cluster of computing devices to perform the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium capable of being stored in the cluster of computing devices or a data storage device such as a data center containing one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives).

[0295] Based on the methods in the above embodiments, this application provides a computer program product containing instructions. The computer program product may be software or program products containing instructions, capable of running on a computing device or stored on any available medium. When the instructions are executed by a cluster of computing devices containing at least one computing device, at least one computing device in the cluster of computing devices performs the methods in the above embodiments.

[0296] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0297] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0298] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0299] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0300] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A text generation method characterized by, The method comprises: obtaining a content template image, a style reference image, a content template edge image and a noise image, the content template image being used to indicate the content of a target text, the style reference image being used to indicate the style of the target text, and the content template edge image being used to indicate the geometric structure information of the target text; generating a target image of the target text based on the noise image under the guidance of the content template image, the style reference image and the content template edge image.

2. The method of claim 1, wherein, The generating a target image of the target text based on the noise image under the guidance of the content template image, the style reference image and the content template edge image comprises: controlling the content of the target text based on the content template image during the generation of the target image based on the noise image; controlling the style of the target text based on the style reference image; controlling the geometric structure information of the target text based on the content template edge image.

3. The method of claim 2, wherein, The controlling the geometric structure information of the target text based on the content template edge image comprises: inputting the content template edge image into a contour encoder to output contour features; introducing a content geometric structure control layer, obtaining output first output feature maps through cross-attention mechanisms based on the contour features and input first input feature maps; controlling the geometric structure information of the target text based on the first output feature maps.

4. The method of claim 3, wherein, The obtaining output first output feature maps through cross-attention mechanisms based on the contour features and input first input feature maps comprises: in the content geometric structure control layer, inputting the first input feature maps as query vectors, inputting the contour features as key vectors and value vectors, and obtaining output first output feature maps through cross-attention calculation.

5. The method according to any one of claims 2 to 4, characterized in that, The controlling the content of the target text based on the content template image comprises: inputting the content template image into a content encoder to output a content template feature map; introducing a deformable content aggregation layer, the input of the deformable content aggregation layer being the content template feature map and input second input feature maps, the output being second output feature maps, and the calculation process comprising: obtaining pixel offset amounts and attention coefficients based on the content template feature map and the input second input feature maps; processing the content template feature map based on the pixel offset amounts and the attention coefficients to obtain an aggregated feature map; obtaining output second output feature maps based on the aggregated feature map and the second input feature maps; controlling the content of the target text based on the second output feature maps.

6. The method of claim 5, wherein, The pixel offset amounts comprise m horizontal pixel offset amounts and m vertical pixel offset amounts, the number of the attention coefficients is m, m is an integer greater than or equal to 1, and the processing the content template feature map based on the pixel offset amounts and the attention coefficients to obtain an aggregated feature map comprises: According to the m horizontal pixel offsets and the m vertical pixel offsets, the content template feature map is bilinearly sampled to obtain m deformed content template feature maps; According to the m attention coefficients, the m deformed content template feature maps are respectively weighted and summed to obtain the aggregated feature map.

7. The method according to claim 5 or 6, characterized in that, The obtaining of the pixel offset and the attention coefficient according to the content template feature map and the input second input feature map comprises: The content template feature map and the input second input feature map are spliced to obtain a spliced feature map; The spliced feature map is subjected to convolution calculation to output the pixel offset and the attention coefficient.

8. The method according to any one of claims 5-7, characterized in that, The obtaining of the output second output feature map according to the aggregated feature map and the second input feature map comprises: The aggregated feature map and the second input feature map are subjected to residual connection to obtain the output second output feature map.

9. The method according to any one of claims 2-8, characterized in that, The control of the style of the target text based on the style reference image comprises: The style reference image is taken as the input of a style encoder to output a style reference feature; A style control layer is adopted to process the input third input feature map and the style reference feature based on a cross-attention mechanism to obtain an output third output feature map; The style of the target text is controlled based on the third output feature map.

10. The method according to any one of claims 1 to 9, characterized in that, The content template edge image is obtained by performing edge extraction on the content template image.

11. The method according to any one of claims 1 to 10, characterized in that, The generation process of the target image is a reverse denoising process of a diffusion model, and is implemented through T times of loop iteration, T being an integer greater than or equal to 1, and the value range of t being from T to 1 in turn and decreasing by 1, and the processing of the tth loop iteration comprising: According to the current time step t, a noisy image X is obtained t , the noisy image being a de-noised image output in the last iteration in the loop iteration process, when the loop iteration starts (t=T), the noisy image being the noise image X T ; the current time step t, the noisy image X t , the content template image, the style reference image and the content template edge image as inputs of a text generation model, output a predicted noise e t of the t-th time step under the guidance of the content template image, the style reference image and the content template edge image. t , the predicted value of the added noise in the noisy image X t corresponding to the t-th time step in the forward noise adding process of the corresponding diffusion model. According to the current time step t and the predicted noise e of the tth time step t , using the inverse process of the corresponding noise adding link in the forward noise adding process of the diffusion model, the noisy image X t is denoised to obtain a denoised image X t-1 , wherein the denoised image X t-1 is used as the noisy image input for the next iteration in the cyclic iteration process. When the loop iteration is terminated (t = 1), the denoised image is the generated target image X0. 12.A method for training a text generation model, comprising: Comprise: Obtaining training samples, the training samples comprising a target image true value sample X0, a content template image sample, a style reference image sample and a content template edge image sample, the content template image sample being used to indicate the content of the text in the target image true value sample, the style reference image sample being used to indicate the style of the text in the target image true value sample, and the content template edge image sample being used to indicate the geometric structure information of the text in the target image true value sample; The training process comprises T loop iterations, T being an integer greater than or equal to 1, for each iteration of the training process, randomly sampling a time step t (1≤t≤T), and adding noise subject to a standard normal distribution Using the target image ground truth sample X0, the time step t, and the added noise corresponding to the time step t Based on the forward noise adding process of the diffusion model, a noisy image sample X corresponding to the time step t is obtained t ; The text generation model is obtained through N times of iterative training, and the training process of the nth (1≤n≤N) iteration comprises: inputting a time step t, a noisy image sample X t , a content template image sample, a style reference image sample and a content template edge image sample as inputs of the text generation model, outputting a corresponding predicted noise e t , the predicted noise e t represents a predicted value of adding noise in the noisy image sample X t under the guidance of the content template image sample, the style reference image sample and the content template edge image sample. According to the predicted noise e t and the added noise a first loss value is obtained; Processing the target image true value sample to obtain an angle point image and an edge image, the angle point image being used to indicate the angle point position information of the text in the target image true value sample, and the edge image being used to indicate the edge position information of the text in the target image true value sample; According to at least one of the angle point image and the edge image, the first loss value is weighted to obtain a second loss value; Taking minimizing the second loss value as a training target, the parameters of the text generation model are optimized and solved.

13. The method of claim 12, wherein, The text generation model further comprises a noise prediction model, a content encoder, an outline encoder, and a style encoder, wherein the text generation model takes as input a time step t, a noisy image sample X t , a content template image sample, a style reference image sample, and a content template edge image sample, and outputs a corresponding predicted noise e t , comprising: The content template image sample is taken as the input of the content encoder to output a sample content template feature map; The content template edge image sample is taken as the input of the contour encoder to output a sample contour feature. The style reference image sample is taken as an input of the style encoder, and a sample style reference feature is output; inputting the time step t, the noisy image sample X t , the sample content template feature map, the sample outline feature, and the sample style reference feature as inputs of the noise prediction model, outputting the predicted noise e t ; The parameter of the text generation model is optimized by taking minimizing the second loss value as a training target, including: The parameters of the noise prediction model, the content encoder, the contour encoder, and the style encoder are optimized by taking minimizing the second loss value as a training target.

14. The method of claim 13, wherein, The noise prediction model comprises an image encoder and an image decoder, and an embedding coding module; the image encoder comprises one or more encoding units, each encoding unit comprising at least one encoding block, and at least one encoding unit comprising a controllable encoding block, the controllable encoding block comprising at least one of a deformable content aggregation layer, a content geometry control layer, and a style control layer; the image decoder comprises one or more decoding units, each decoding unit comprising at least one decoding block, and at least one decoding unit comprising a controllable decoding block, the controllable decoding block comprising at least one of a content geometry control layer and a style control layer; The time step t, the noisy image sample X t , the sample content template feature map, the sample contour feature, and the sample style reference feature as inputs of the noise prediction model, output the predicted noise e t , comprising: The embedding coding module converts the time step t into a time step embedding vector, which is used to superimpose on the feature maps corresponding to the output of the encoding block and the decoding block to regulate the iterative calculation process; embedding the time step vector, the noisy image sample X t , the sample content template feature map, the sample contour feature, and the sample style reference feature as inputs of the image encoder, outputting image encoding features; embedding the time step embedding vector, the image encoding feature, the sample profile feature, and the sample style reference feature as inputs to the image decoder, outputting the predicted noise e t .

15. The method according to any one of claims 12-14, characterized in that, The second loss value is obtained by weighting the first loss value according to at least one of the corner point image and the edge image, including: The first weight and the second weight are used for weighted summation on the corner point image and the edge image respectively to obtain an image geometry weight coefficient, the first weight and the second weight being a value greater than or equal to 0 and less than or equal to 1; The second loss value is obtained by weighting the first loss value according to the image geometry weight coefficient.

16. A character generating apparatus characterized by comprising: Including: An acquisition module is configured to acquire a content template image, a style reference image, a content template edge image, and a noise image, the content template image being used to indicate the content of a target text, the style reference image being used to indicate the style of the target text, the content template edge image being used to indicate the geometric structure information of the target text, and the noise image being subject to a standard normal distribution; A processing module is configured to generate a target image of the target text iteratively by using a reverse denoising process of a diffusion model based on the noise image under the guidance of the content template image, the style reference image, and the content template edge image. 17.A device for training a text generation model, comprising: Including: A sample acquisition module is configured to acquire a training sample, the training sample comprising a target image ground truth sample X0, a content template image sample, a style reference image sample, and a content template edge image sample, the content template image sample being used to indicate the content of the text in the target image ground truth sample, the style reference image sample being used to indicate the style of the text in the target image ground truth sample, and the content template edge image sample being used to indicate the geometric structure information of the text in the target image ground truth sample; The pre-processing module, the training process includes T iterations, T is an integer greater than or equal to 1, for each iteration in the training process, the pre-processing module is configured to randomly sample a time step t (1≤t≤T), and add noise subject to a standard normal distribution According to the target image true value sample X0, the time step t and the added noise corresponding to the time step t, a noisy image sample X corresponding to the time step t is obtained by using a forward noise adding process of the diffusion model t When t=T, X T is a noise image subject to a standard normal distribution finally obtained by the forward noise adding process of the diffusion model The training module is used to train the text generation model. The text generation model is trained through N iterations. The training process of the nth (1≤n≤N) iteration includes: processing the time step t and the noisy image sample X output by the preprocessing module. t The text generation model takes the content template image sample, style reference image sample, and content template edge image sample output by the sample acquisition module as input, and outputs the prediction noise e. t Predicted noise e t This indicates that the noisy image sample X is obtained under the guidance of the content template image sample, the style reference image sample, and the content template edge image sample. t Adding noise The predicted value; based on the predicted noise e t and the added noise A first loss value is obtained; and the ground truth sample of the target image is processed to obtain a corner image and an edge image, wherein the corner image is used to indicate the corner position information of the text in the ground truth sample of the target image, and the edge image is used to indicate the edge position information of the text in the ground truth sample of the target image. And, according to at least one of the corner point image and the edge image, the first loss value is weighted to obtain a second loss value; and a parameter of the character generation model is optimized by taking minimizing the second loss value as a training target.

18. A chip system, characterized by The chip system comprises a processor and a power supply circuit, the power supply circuit is used for powering the processor, and the processor is used for executing the method according to any one of claims 1-15.

19. A computing device, comprising: The computing device comprises a processor and a memory; The processor is used for executing instructions stored in the memory, so that the computing device executes the method according to any one of claims 1-15.

20. A cluster of computing devices, characterized in that, The computing device comprises a processor and a memory; The processor of the at least one computing device is used for executing instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1-15.

21. A computer-readable storage medium, characterized in that, The computer program instructions, when executed by a computing device cluster, cause the computing device cluster to execute the method according to any one of claims 1-15, wherein the computing device cluster comprises at least one computing device.

22. A computer program product comprising instructions, wherein: The computer program instructions, when executed by a computing device cluster, cause the computing device cluster to execute the method according to any one of claims 1-15, wherein the computing device cluster comprises at least one computing device.

Images