A small sample font generation method based on diffusion model

Abstract

The application discloses a small sample font generation method based on a diffusion model, which comprises the following steps: constructing a font image dataset, continuously adding noise to a target font image to pure noise, selecting content and style reference images, respectively using content and style encoders for encoding, extracting structure and semantic information, training a diffusion model to predict noise of a noise image, gradually removing hidden encoding noise, and finally aiming at an original image without noise. In the denoising process, the content and style reference images are used as denoising conditions, the style refinement module is used to strengthen the style features, the style-content fusion module is used to promote the interaction between the content and style information, the total loss function is calculated to update the model parameters, and based on the new font and Chinese character, a font image is generated. Compared with the prior art, the training process is more stable, the problem that the generated Chinese character is incomplete or incorrect is solved, the model can learn the style information such as thickness, brush stroke style, the method is simple, the effect is good, and the application prospect is good.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and specifically to a few-sample font generation method based on a diffusion model. Background Technology

[0002] Typography technology benefits many important applications, such as ancient book restoration, data augmentation for optical character recognition, and the creation of new fonts. Therefore, it has significant commercial and cultural value. However, traditional font design heavily relies on expert designers manually rendering glyph styles for each character, making font creation costly, especially for languages ​​with large scripts, such as Chinese, Japanese, and Korean. Recently, with the development of deep learning, many studies have proposed few-sample font generation methods, which automatically generate thousands of glyph images with the same style based on fewer than five reference style images.

[0003] Existing research aims to generate previously unseen fonts by decoupling style and content representations. To enable the generator to capture local style features, the LF-Font model leverages prior knowledge, such as strokes and components, employing a component-based learning strategy to enhance local style representation. Several prior-independent methods have also been proposed; the Fs-Font method aims to discover spatial correspondences between content and style images to learn local style details; CF-Font integrates various content features from different fonts and introduces an iterative style vector refinement strategy. However, these methods have the following problems:

[0004] These methods are all based on the Generative Adversarial Networks (GAN) framework. Due to the adversarial nature of their training, this can lead to training instability. LF-Font utilizes prior knowledge such as strokes and components, requiring component annotation for each content character. During training, it selects style glyphs containing these components as references, resulting in high annotation costs. Furthermore, if the model doesn't learn the components within the glyphs, the generated results may be poor, requiring new preparation for applying new glyphs, further increasing generation costs. For CF-Font, a method that learns the spatial relationship between content and style images, the selection strategy for reference images significantly impacts the quality of the results. In addition, these methods still face difficulties in generating complex characters and handling large-scale style transfer. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a few-sample font generation method based on a diffusion model. This method involves constructing a font image dataset, continuously adding noise to the target font image to obtain pure noise, selecting content images and style reference images, and encoding these images using content encoders and style encoders to extract structural and semantic information. Then, the noise in the noisy image is predicted, and denoising inference is performed. The diffusion model is trained to progressively remove intermediate implicitly encoded noise, with the training target being the original image without added noise. During the denoising process, the content image and style reference image serve as auxiliary conditions. A style refinement module enhances style features, and a content-style fusion module promotes better interaction between content and style information. The total loss function of the model is calculated to update the model parameters, and finally, a font image is generated based on the new font and Chinese characters.

[0006] The method is simple, effective, and has good application prospects.

[0007] The objective of this invention is achieved as follows:

[0008] A few-sample font generation method based on a diffusion model is characterized by the following steps: First, a font image dataset is constructed; noise is added multiple times to the target image to obtain pure noise; then, a content encoder encodes the content image and style image, and a style encoder encodes the style vector of the style reference image; next, a UNet network is used to predict the noise in the noisy image, with the content image and style reference image serving as auxiliary conditions for denoising. To better refine the style information, a style refinement module is used to enhance the style patterns in the style image; to strengthen the fusion of style and content information, a style-content fusion module is proposed; next, the mean square error of the predicted noise and the added noise, as well as the sum of the differences between the multi-layer feature maps of the generated image and the target image, are calculated to adjust the model parameters; finally, a new style reference image and the Chinese character to be generated are input into the trained model to generate a new style glyph image; specifically, the method includes the following steps:

[0009] A few-sample font generation method based on a diffusion model, characterized by the following steps:

[0010] 1) Construct a font image dataset;

[0011] 2) Add noise multiple times to the target image to obtain a noisy image that satisfies a Gaussian distribution;

[0012] 3) Use the regular script style of the text A in the target generated image as the content image x. c Images of other characters in style B of the target image are used as style image x. s We use content encoders and style encoders to extract structural and semantic features;

[0013] 4) Train the diffusion model to predict noise in noisy images, gradually removing intermediate hidden-coded noise. The training target is the original image without added noise; content image x c and style images x s As a conditional denoising mechanism for the diffusion model, a style refinement module and a style-content fusion module are constructed in the diffusion model, and finally an image of the content Chinese character A of style font B is generated.

[0014] 5) Calculate the mean square error of the predicted noise and the added noise, and adjust the model parameters by summing the differences between the multi-layer feature maps of the generated image and the target image;

[0015] 6) After training, test the model by inputting the reference image of the style to be generated and the Chinese characters to be generated into the trained model to generate new style glyph images.

[0016] Furthermore, step 1) specifically includes:

[0017] Given a font file with the .ttf extension, render the font image based on the Chinese characters to be generated.

[0018] Furthermore, step 2) specifically includes:

[0019] Adding noise multiple times to the images in the dataset yields purely noisy images, which are then used to obtain the training data. The noise-adding expression is: Where t is the time step, ∈ is Gaussian noise, and β t is the variance of the t-th noise addition, which is a constant that increases as t increases; x0 is the original image, x t This is the image after adding noise; α t It is also a constant, and the formula for calculation is: α t =1-β t ;

[0020] For α t Cumulative product from time 1 to time t;

[0021] Furthermore, step 3) specifically includes:

[0022] 3.1: Encoding the content image x using a content encoder c Obtain content feature map f c ;

[0023] 3.1.1: The content encoder consists of four convolutional blocks. Each convolutional block contains the Identity() normalization function, the ReLU activation function, the reflection padding function to maintain the consistency of the input and output dimensions, and a convolutional layer. The initial number of input channels of the convolutional layer is 1, and the number of output channels are 32, 64, 128, and 256, respectively.

[0024] 3.2: Style image x is encoded using a style encoder. s Obtain style feature map f s ;

[0025] 3.2.1: The style encoder consists of six convolutional blocks;

[0026] 3.2.2: For the first five convolutional blocks, each convolutional block contains the Identity() normalization function, the ReLU activation function, the reflection padding function to maintain the consistency of the input and output dimensions, and a convolutional layer; the initial input channel of the convolutional layer is 1, and the output channels are 64, 128, 256, 512 and 1024 respectively;

[0027] 3.2.3: The last convolutional block contains an instance normalization function, a ReLU activation function, and a convolutional layer. The input and output channels of the convolutional layer are both 1024, and the kernel size is 1×1.

[0028] Furthermore, step 4) specifically includes:

[0029] 4.1: Construct a style refinement module;

[0030] 4.1.1: Style feature map f s The inputs are fed into two convolutional layers respectively to obtain f. s1 and f s2 Simultaneously, the style feature map is input into another convolutional layer to obtain f. s3 ;

[0031] 4.1.2: f s1 f s2 f s3 Change the shape to in Let C be the number of channels, H and W be the length and width of the feature map, respectively, and N = H × W, which is the product of the length and width.

[0032] 4.1.3: Then, calculate the style space attention weights. The calculation expression is

[0033]

[0034] in, Representing the feature map f s2 transpose,

[0035] Then, the feature map f is adjusted by matrix multiplication and element-level addition. s The output of the style refinement module is obtained, and the calculation is performed.

[0036] The expression is

[0037]

[0038] in, Representative matrix A s Transpose of;

[0039] 4.2: Construct a style-content fusion module

[0040] 4.2.1: Content feature maps of each layer obtained by the content encoder and the feature map f output by the style refinement module ss Implement cross-attention mechanisms. First, input the noisy image x into the model. T , representing the noisy image at time T, undergoes channel attention mechanism at time t, and is then projected onto the query matrix Q. The feature map f output by the style refinement module is projected onto the key matrix K and the value matrix V, respectively.

[0041] 4.2.2: Construct a bias term for the attention mechanism This represents the content feature map of layer 1. It is a fine-grained appearance encoder consisting of M transformer layers with zero convolutions. A zero convolution is a standard 1×1 convolutional layer with weights and biases initialized to zero.

[0042] 4.2.3: Calculating the expression for the attention mechanism

[0043]

[0044] Among them, K T d represents the transpose of matrix K, d is the dimension of matrix Q, Q is the query matrix, K is the key matrix, and V is the value matrix;

[0045] 4.3: Perform downsampling on the output of the style-content fusion module;

[0046] 4.4: After passing through four style-content fusion modules and downsampling operations, the data is input into the intermediate module;

[0047] 4.5: Subsequently, the output of the intermediate module undergoes four upsampling operations to obtain the predicted noise. The input x is then divided by the predicted noise at this moment to obtain x. t-1 , will x t-1 Input the data into the model and repeat step 4.5. After continuous denoising, the generated font image is obtained.

[0048] Furthermore, step 5) specifically includes:

[0049] 5.1: Calculate the mean square error of the predicted noise and the added noise. The calculation expression is as follows:

[0050] L MSE =||∈-∈ θ (x t ,t,x c ,x s )||,

[0051] Where ∈ represents the actual noise added, ∈ θ The noise in the final prediction;

[0052] 5.2: Calculate the sum of the differences between the multi-layer feature maps of the generated image and the target image. The calculation expression is as follows:

[0053]

[0054] Where x0 is the generated image, x target For the target image, VGG is a VGG16 network pre-trained on ImageNet;

[0055] 5.3: Calculate the total loss function, the expression is as follows:

[0056] L total =λ1L MSE E+λ2L feature

[0057] Where λ1 and λ2 are the weight coefficients of the two loss functions, respectively, satisfying the condition λ1+λ2=1.

[0058] Furthermore, step 6) specifically includes:

[0059] Given a new style reference image and the Chinese characters to be generated, input them into the trained model to generate new style glyph images.

[0060] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:

[0061] 1) This invention proposes a style refinement module, where learning the style patterns (strokes, textures, thickness) of style images is crucial for font generation. Therefore, applying a channel attention mechanism to the style can better enhance style features;

[0062] 2) This invention proposes a style-content fusion module, which projects the refined style features onto K and V in the cross-attention mechanism, projects the content feature map of each layer onto Q, and inputs the content feature map into a fine-grained appearance encoder as a bias term of the attention mechanism, which enables the content features to be learned better. Attached Figure Description

[0063] Figure 1 This is a flowchart of the present invention;

[0064] Figure 2 This is a schematic diagram illustrating the specific operation of an embodiment. Detailed Implementation

[0065] The present invention will be further described below with reference to specific embodiments and accompanying drawings.

[0066] This invention constructs a font dataset. Noise is continuously added to target font images to obtain pure noise. Content images and style reference images are selected, and content encoders and style encoders are used to encode the content and style images to extract structural and semantic information. Then, the noise in the noisy images is predicted, and denoising inference is performed. A diffusion model is trained to progressively remove intermediate hidden-coded noise, with the training target being the original, un-noised image. During the denoising process, the content and style reference images serve as auxiliary conditions. A style refinement module enhances style features, and a content-style fusion module promotes better interaction between content and style information. The total loss function of the model is calculated to update the model parameters, and finally, font images are generated based on new fonts and Chinese characters.

[0067] See Figure 1 The present invention provides a few-sample font generation method based on a diffusion model, which specifically includes the following steps:

[0068] 1) Construct a font image dataset;

[0069] 1.1: Collect font files with the .ttf extension, write the Chinese characters into a txt file, and render it into a glyph image;

[0070] 2) Add noise to the images in the dataset multiple times to obtain noisy images that satisfy a Gaussian distribution;

[0071] 2.1: The expression for adding noise to an image is as follows: Where t is the time step, ∈ is Gaussian noise, and β is the variance of the t-th noise addition. x0 is the original image, x t This is the image after noise has been added. α t The formula for calculating α is: t =1-β t

[0072]

[0073] 3) Use the KaiTi style of text A in the target image as the content image x. c Images of other characters in style B of the target image are used as style image x. s We use content encoders and style encoders to extract structural and semantic features;

[0074] 4) Predict the noise in the noisy image, perform denoising inference, and train the diffusion model to gradually remove the noise in the intermediate hidden code. The training target is the original image without added noise. Content image x c and style images x s As a conditional denoising method for diffusion models, it generates an image of the Chinese character A in style font B;

[0075] 4.1: Construct a style refinement module;

[0076] 4.1.1: Style feature map f s The inputs are fed into two convolutional layers respectively to obtain f. s1 and f s2 Simultaneously, the style feature map is input into another convolutional layer to obtain f. s3 ;

[0077] 4.1.2: f s1 f s2 f s3 Change the shape to Where N = H × W;

[0078] 4.1.3: Then, calculate the style space attention weights. The calculation expression is

[0079]

[0080] in, Representing the feature map f s2 transpose,

[0081] Then, the feature map f is adjusted by matrix multiplication and element-level addition. s The output of the style refinement module is obtained, and the calculation expression is:

[0082]

[0083] in, Representative matrix A s Transpose of;

[0084] 4.2: Construct a style-content fusion module

[0085] 4.2.1: Content feature maps of each layer obtained by the content encoder A cross-attention mechanism is used with the feature maps output by the style refinement module. First, it is combined with the output of the first layer of the Unet network at the channel level, and then projected onto the feature map f output by the Q-style refinement module. ss Projected onto K and V respectively;

[0086] 4.2.2: Construct a bias term for the attention mechanism This represents the content feature map of layer 1. It is a fine-grained appearance encoder consisting of K transformer layers with zero convolutions. A zero convolution is a standard 1×1 convolutional layer with weights and biases initialized to zero.

[0087] 4.2.3: Calculating the expression for the attention mechanism

[0088]

[0089] Among them, K T Represents the transpose of matrix K

[0090] 4.3: Perform downsampling on the output of the style-content fusion module;

[0091] 4.4: After passing through four style-content fusion modules and downsampling operations, the data is input into the intermediate module;

[0092] 4.5: Subsequently, the output of the intermediate module undergoes four upsampling operations to obtain the predicted noise, which is then used to input x. t After removing the noise predicted at this moment, we get x. t-1 , will x t-1 Input the data into the model and repeat step 4.5. After continuous denoising, the generated font image is obtained.

[0093] 5) Calculate the mean square error of the predicted noise and the added noise, as well as the sum of the differences between the multi-layer feature maps of the generated image and the target image, to calculate the total loss function, expressed as follows:

[0094] L total =λ1L MSE +λ2L feature

[0095] Where λ1 and λ2 are 0.8 and 0.2 respectively, the model parameters are adjusted by backpropagation based on the total loss function;

[0096] 6) Input the new style reference image and the Chinese characters to be generated into the trained model to generate new style glyph images.

Claims

1. A few-sample font generation method based on a diffusion model, characterized in that, The method includes the following steps: 1) Construct a font image dataset; 2) Add noise multiple times to the target image to obtain a noisy image that follows a Gaussian distribution; 3) Use the regular script style of text A in the target generated image as the content image. Images of other characters in style B of the target image are used as style images. We use content encoders and style encoders to extract structural and semantic features; 4) Train the diffusion model to predict noise in noisy images, gradually removing intermediate hidden-coded noise. The training target is the original image without added noise; content image. and style images As a conditional denoising mechanism for the diffusion model, a style refinement module and a style-content fusion module are constructed in the diffusion model, and finally an image of the content Chinese character A of style font B is generated. 5) Calculate the mean square error of the predicted noise and the added noise, and adjust the model parameters by summing the differences between the multi-layer feature maps of the generated image and the target image; 6) After training, test the model by inputting the reference image of the style to be generated and the Chinese characters to be generated into the trained model to generate new style glyph images; where: Step 4) specifically includes: 4-1: Construct a style refinement module; 4-1-1: Style Feature Map The inputs are fed into two convolutional layers respectively, resulting in... and Simultaneously, the style feature map is input into another convolutional layer to obtain... ; 4-1-2: Will , , Change the shape to ,in Let C be the number of channels, and H and W be the length and width of the feature map, respectively. , which is the product of length and width; 4-1-3: Then, calculate the style space attention weights. The calculation expression is: ， in, Representation of feature map Transpose of; Then, the feature map is adjusted by matrix multiplication and element-level addition. The output of the style refinement module is obtained, and the calculation expression is: ， in, Representative matrix Transpose of; 4-2: Build a style-content fusion module 4-2-1: Content feature maps of each layer obtained by the content encoder and the feature map output by the style refinement module. Implement cross-attention mechanisms. First, input the noisy image into the model. The image represents the noisy image at time T. A channel attention mechanism is applied at time t, and the image is then projected onto the query matrix Q. The feature map output by the style refinement module... Project them onto the key matrix K and the value matrix V, respectively; 4-2-2: Constructing a bias term for the attention mechanism , This represents the content feature map of layer l. It is a fine-grained appearance encoder consisting of M transformer layers with zero convolutions. Zero convolution is a standard feature. The convolutional layer has its weights and biases initialized to zero. 4-2-3: Calculating the expression for the attention mechanism in, d represents the transpose of matrix K, d is the dimension of matrix Q, Q is the query matrix, K is the key matrix, and V is the value matrix; 4-3: Perform downsampling on the output of the style-content fusion module; 4-4: After passing through four style-content fusion modules and downsampling operations, the data is input into the intermediate module; 4-5: Subsequently, the output of the intermediate module undergoes four upsampling operations to obtain the predicted noise, which is then used to input the noise. Remove the noise predicted at this moment to obtain ,Will Input the data into the model and repeat steps 4-5. After continuous noise reduction, the generated font image is obtained.

2. The method for generating small sample fonts according to claim 1, characterized in that, Step 1) specifically includes: Given a font file with the .ttf extension, render the font image based on the Chinese characters to be generated.

3. The method for generating small sample fonts according to claim 1, characterized in that, Step 2) specifically includes: Adding noise multiple times to the images in the dataset yields purely noisy images, which are then used to obtain the training data. The noise-adding expression is: , where t is the time step Gaussian noise, It is the variance of the t-th noise addition, which is a constant that increases as t increases; This is the original image. This is the image after noise has been added; It is also a constant, and the calculation formula is: ; for The cumulative product from time 1 to time t.

4. The method for generating small sample fonts according to claim 1, characterized in that, Step 3) specifically includes: 3-1: Encode content images using a content encoder. ; 3-1-1: The content encoder consists of four convolutional blocks. Each convolutional block contains an Identity() normalization function, a ReLU activation function, a reflection padding function to maintain the consistency of the input and output dimensions, and a convolutional layer. The initial number of input channels for the convolutional layer is 1, and the number of output channels are 32, 64, 128, and 256, respectively. 3-2: Encode style images using a style encoder. ; 3-2-1: The style encoder consists of six convolutional blocks; 3-2-2: For the first five convolutional blocks, each convolutional block contains the Identity() normalization function, the ReLU activation function, the reflection padding function to keep the dimensions of the input and output consistent, and a convolutional layer; the initial input channel of the convolutional layer is 1, and the output channels are 64, 128, 256, 512 and 1024 respectively; 3-2-3: The last convolutional block contains an instance normalization function, a ReLU activation function, and a convolutional layer. The input and output channels of the convolutional layer are both 1024, and the kernel size is [missing value]. .

5. The method for generating small sample fonts according to claim 1, characterized in that, Step 5) specifically includes: 5-1: Calculate the mean square error of the predicted noise and the added noise. The calculation expression is as follows: ， in, The noise was added to the actual sound. The noise in the final prediction; 5-2: Calculate the sum of the differences between the multi-layer feature maps of the generated image and the target image. The calculation expression is as follows: ， in For the generated image, For the target image, VGG is a VGG16 network pre-trained on ImageNet; 5-3: Calculate the total loss function, the expression is as follows: + in These are the weight coefficients of the two loss functions, satisfying the condition. + =1.

6. The few-sample font generation method based on a diffusion model according to claim 1, characterized in that, Step 6) specifically includes: Given a new style reference image and the Chinese characters to be generated, input them into the trained model to generate new style glyph images.

Images