Small sample font generation method based on diffusion model
By constructing a font image dataset using a diffusion model-based few-sample font generation method and extracting information using a content encoder and a style encoder, combined with style refinement and fusion modules, the difficulties in generating complex characters and large-scale style transfer in existing technologies are solved, achieving efficient and low-cost font generation.
Patent Information
- Application Number
- CN202410978649.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-07-22
AI Technical Summary
Existing small-sample font generation methods suffer from problems such as unstable training, high annotation costs, and poor generation results when generating complex characters and handling large-scale style transfer. This is especially true in languages with a large number of characters, such as Chinese, Japanese, and Korean. Traditional methods rely on expert designers to manually render the glyph styles, resulting in high costs and low generation efficiency.
A few-sample font generation method based on a diffusion model is adopted. By constructing a font image dataset, noise is added to the target image multiple times. The content encoder and style encoder are used to extract structural and semantic information. The noise is gradually removed by combining the style refinement module and the content-style fusion module. The model parameters are updated using the total loss function, and finally, a new style glyph image is generated.
It enables simple and effective font generation, reduces generation costs, improves generation efficiency and style transfer quality, and is suitable for complex characters and large-scale style transfer.
Smart Images

Figure CN118898549A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a small sample font generation method based on a diffusion model. Background Art
[0002] Font design technology can benefit many important applications, such as ancient book restoration, data enhancement for optical character recognition, and new font creation. Therefore, it has significant commercial and cultural value. However, traditional font design relies heavily on expert designers to manually render the glyph style for each character, making font creation costly, especially for languages with a large number of characters, such as Chinese, Japanese, and Korean. Recently, with the development of deep learning, many studies have proposed small sample font generation methods, that is, automatically generating thousands of glyph images of the same style based on less than 5 reference style images.
[0003] Research on existing technologies generates fonts that have never been seen before by decoupling style and content representation. In order to enable the generator to capture local style features, the LF-Font model uses prior knowledge such as strokes and components, and adopts a component-based learning strategy to enhance the local style representation capability; some methods that do not rely on priors have also been proposed. The Fs-Font method aims to discover the spatial correspondence between content images and style images to learn local style details; CF-Font integrates various content features of different fonts and introduces an iterative style vector refinement strategy. However, these methods have the following problems:
[0004] These methods are all based on the Generative Adversarial Networks (GAN) framework, which may lead to unstable training due to the nature of adversarial training. LF-Font uses a method based on prior knowledge such as strokes and components. It requires component annotation for each content word, and selects style glyphs containing components as references during training, which leads to high annotation costs. Moreover, if the component model in the glyph is not learned, the model generation effect may be poor, which requires new preparation when applied to new glyphs, and has high generation costs. For CF-Font, a method that learns the spatial relationship between content and style images, its reference image selection strategy has a greater impact on the quality of the results. In addition, these methods still have difficulties in generating complex characters and handling large-scale style transfer. Summary of the invention
[0005] The purpose of the present invention is to propose a small sample font generation method based on a diffusion model in response to the deficiencies of the prior art. A font image dataset is constructed, and the target font image is continuously denoised to obtain pure noise. A content image and a style reference image are selected, and the content image and the style image are encoded using a content encoder and a style encoder to extract structural information and semantic information. Then, the noise of the noise image is predicted, denoising inference is performed, and a diffusion model is trained to gradually remove the noise of the intermediate latent code, and the training target is the original image without denoising. In the denoising process, the content image and the style reference image are used as auxiliary conditions for denoising, the style refinement module is used to strengthen the style features, and the content-style fusion module is used to promote better interaction between content information and style information. The total loss function of the model is calculated to update the model parameters, and finally a font image is generated based on the new font and Chinese characters.
[0006] The method is simple, effective and has good application prospects.
[0007] The object of the present invention is achieved in that:
[0008] A small sample font generation method based on a diffusion model is characterized in that the method first constructs a font image dataset; adds noise to a target image multiple times to obtain pure noise; then, uses a content encoder to encode a content image and a style image, and uses a style encoder to encode a style vector of a style reference image; then, uses a UNet network to predict the noise of the noise image, and uses the content image and the style reference image as auxiliary conditions for denoising. In order to better refine the style information, a style refinement module is used to enhance the style mode in the style image. In order to strengthen the fusion of style and content information, a style-content fusion module is proposed; then, the mean square error of the predicted noise and the added noise, as well as the sum of the difference between the generated image and the multi-layer feature map of the target image are calculated to adjust the model parameters; finally, a new style reference image and the Chinese character to be generated are given as input into the trained model to generate a new style font image; specifically, the following steps are included:
[0009] A method for generating a small sample font based on a diffusion model, characterized in that the method comprises the following steps:
[0010] 1) Build a font image dataset;
[0011] 2) Add noise to the target generated image multiple times to obtain a noise image that satisfies the Gaussian distribution;
[0012] 3) Take the Kaiti style of the target generated image as the content image x c , use the images of other words in the style B of the target image as the style image x s ,Use content encoder and style encoder to extract structural features and semantic features;
[0013] 4) Train the diffusion model to predict the noise of the noisy image and gradually remove the noise of the intermediate hidden code. The training target is the original image without noise. The content image x c and style image x s As a conditional aid to denoising in the diffusion model, a style refinement module and a style-content fusion module are constructed in the diffusion model, and finally an image of content Chinese character A with style font B is generated;
[0014] 5) Calculate the mean square error of the predicted noise and the added noise, as well as the sum of the differences between the generated image and the multi-layer feature map of the target image to adjust the model parameters;
[0015] 6) After the training is completed, the model is tested, and the reference image of the style to be generated and the Chinese characters to be generated are input into the trained model to generate a new style glyph image.
[0016] Further, the step 1) specifically includes:
[0017] Given a font file with a .ttf suffix, render the font image according to the Chinese characters to be generated.
[0018] Further, the step 2) specifically includes:
[0019] Add noise to the images in the data set multiple times to obtain pure noise images and obtain training data. The noise addition expression is: where t is the time step, ∈ is Gaussian noise, β t is the variance of the t-th noise addition, which is a constant and increases with the increase of t; x0 is the original image, x t is the image after adding noise; α t It is also a constant, and the calculation formula is: t =1-β t ;
[0020] is α t The cumulative multiplication from time 1 to time t;
[0021] Further, the step 3) specifically includes:
[0022] 3.1: Encode the content image x using the content encoder c Get the content feature map f c ;
[0023] 3.1.1: The content encoder consists of four convolutional blocks, each of which contains the Identity() normalization function, the ReLU activation function, the reflection padding function to keep the input and output dimensions consistent, and the convolutional layer; the initial number of input channels of the convolutional layer is 1, and the number of output channels is 32, 64, 128, and 256 respectively;
[0024] 3.2: Use style encoder to encode style image x s Get the style feature map f s ;
[0025] 3.2.1: The style encoder consists of six convolutional blocks;
[0026] 3.2.2: For the first five convolutional blocks, each convolutional block contains the Identity() normalization function, the ReLU activation function, the reflection padding function to keep the input and output dimensions consistent, and the convolutional layer; the initial input channel of the convolutional layer is 1, and the output channels are 64, 128, 256, 512, and 1024 respectively;
[0027] 3.2.3: The last convolution block contains the instance normalization function, the ReLU activation function and the convolution layer. The input and output channels of the convolution layer are both 1024, and the convolution kernel size is 1×1.
[0028] Further, the step 4) specifically includes:
[0029] 4.1: Construct style refinement module;
[0030] 4.1.1: The style feature map f s Input into two convolutional layers respectively to get f s1 and f s2 At the same time, the style feature map is input into another convolutional layer to obtain f s3 ;
[0031] 4.1.2: f s1 , f s2 , f s3 Change the shapes to in is a real number space, C is the number of channels, H and W are the length and width of the feature map respectively, and N = H × W is the product of length and width;
[0032] 4.1.3: Then, calculate the style space attention weight The calculation expression is
[0033]
[0034] in, Represents the feature map f s2 The transpose of
[0035] Then, the feature map f is adjusted by matrix multiplication and element-wise addition. s , get the output of the style refinement module, count
[0036] The calculation expression is
[0037]
[0038] in, Represents the matrix A s The transpose of
[0039] 4.2: Building a style-content fusion module
[0040] 4.2.1: Each layer of content feature map obtained by encoding the content encoder And the feature map f output by the style refinement module ss Perform a cross-attention mechanism. First, the noise image x input into the model T , represents the noise image at time T, the channel attention mechanism is performed at time t, and then projected to the query matrix Q, and the feature map f output by the style refinement module is projected to the key matrix K and the value matrix V respectively;
[0041] 4.2.2: Building a bias term for the attention mechanism Represents the content feature map of layer 1, It is a fine-grained appearance encoder consisting of M transformer layers with zero convolutions, where the zero convolution is a standard 1×1 convolutional layer, whose weights and biases are initialized to zero;
[0042] 4.2.3: Calculating the Attention Mechanism Expression
[0043]
[0044] Among them, K T represents the transpose of matrix K, d is the dimension of Q matrix, Q is the query matrix, K is the key matrix, and V is the value matrix;
[0045] 4.3: Downsample the output of the style-content fusion module;
[0046] 4.4: After four style-content fusion modules and downsampling operations, it is input into the intermediate module;
[0047] 4.5: Then, the output of the intermediate module undergoes four upsampling operations to obtain the predicted noise. The input x is removed from the predicted noise at this moment to obtain x t-1 , x t-1 Input into the model, repeat step 4.5, and after continuous denoising, the generated font image is obtained.
[0048] Further, the step 5) specifically includes:
[0049] 5.1: Calculate the mean square error of the predicted noise and the added noise. The calculation expression is
[0050] L MSE =||∈-∈ θ (x t ,t,x c ,x s )||,
[0051] Among them, ∈ is the real noise added, ∈ θ is the final predicted noise;
[0052] 5.2: Calculate the sum of the differences between the generated image and the multi-layer feature map of the target image. The calculation expression is
[0053]
[0054] Where x0 is the generated image, x target is the target image, VGG is the VGG16 network pre-trained based on ImageNet;
[0055] 5.3: Calculate the total loss function, the expression is
[0056] L total =λ1L MSE E+λ2L feature
[0057] Among them, λ1 and λ2 are the weight coefficients of the two loss functions, respectively, satisfying the condition λ1+λ2=1.
[0058] Further, the step 6) specifically includes:
[0059] Given a new style reference image and the Chinese characters to be generated, they are input into the trained model to generate a new style glyph image.
[0060] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:
[0061] 1) The present invention proposes a style refinement module. Learning the style pattern (stroke, texture, thickness) of the style image is very important for font generation. Therefore, channel attention mechanism for the style can better enhance the style characteristics;
[0062] 2) The present invention proposes a style-content fusion module, which projects the refined style features to K and V in the cross-attention mechanism, projects the content feature map of each layer to Q, and inputs the content feature map into the fine-grained appearance encoder as a bias item of the attention mechanism, so that the content features can be better learned. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] Figure 1 It is a flow chart of the present invention;
[0064] Figure 2 It is a schematic diagram of the specific operation of the embodiment. DETAILED DESCRIPTION
[0065] The present invention is further described below in conjunction with specific embodiments and drawings.
[0066] The present invention constructs a font dataset, continuously adds noise to the target font image to obtain pure noise, selects content images and style reference images, and encodes the content images and style images using content encoders and style encoders to extract structural information and semantic information. Then, the noise of the noise image is predicted, denoising inference is performed, and a diffusion model is trained to gradually remove the noise of the intermediate latent code, and the training target is the original image without noise. In the denoising process, the content image and the style reference image are used as auxiliary conditions for denoising, the style refinement module is used to strengthen the style features, the content-style fusion module is used to promote better interaction between content information and style information, the total loss function of the model is calculated to update the model parameters, and finally a font image is generated based on the new font and Chinese characters.
[0067] See also Figure 1 The present invention provides a method for generating a small sample font based on a diffusion model, which specifically comprises the following steps:
[0068] 1) Build a font image dataset;
[0069] 1.1: Collect font files with .ttf as the suffix, write the Chinese characters in the content into a txt file, and render it into a glyph image;
[0070] 2) Add noise to the images in the data set multiple times to obtain a noise image that satisfies the Gaussian distribution;
[0071] 2.1: The noise expression for the image is Where t is the time step, ∈ is Gaussian noise, and β is the variance of the t-th noise addition. x0 is the original image, x t is the image after adding noise. t The calculation formula is: t =1-β t
[0072]
[0073] 3) Take the Kaiti style of the target image’s text A as the content image x c , use the images of other words in the style B of the target image as the style image x s ,Use content encoder and style encoder to extract structural features and semantic features;
[0074] 4) Predict the noise of the noisy image, perform denoising inference, and train the diffusion model to gradually remove the noise of the intermediate hidden code. The training target is the original image without noise. Content image x c and style image x s As a condition of the diffusion model to assist in denoising, an image of the content Chinese character A is generated with style font B;
[0075] 4.1: Construct style refinement module;
[0076] 4.1.1: The style feature map f s Input into two convolutional layers respectively to get f s1 and f s2 At the same time, the style feature map is input into another convolutional layer to obtain f s3 ;
[0077] 4.1.2: f s1 , f s2 , f s3 Change the shapes to Where N = H × W;
[0078] 4.1.3: Then, calculate the style space attention weight The calculation expression is
[0079]
[0080] in, Represents the feature map f s2 The transpose of
[0081] Then, the feature map f is adjusted by matrix multiplication and element-wise addition s , the output of the style refinement module is obtained, and the calculation expression is
[0082]
[0083] in, Represents the matrix A s The transpose of
[0084] 4.2: Building a style-content fusion module
[0085] 4.2.1: Each layer of content feature map obtained by encoding the content encoder Perform cross-attention mechanism with the feature map output by the style refinement module. First, combine the output of the previous layer of the Unet network at the channel level, and then project it to Q, the feature map f output by the style refinement module ss Project to K and V respectively;
[0086] 4.2.2: Building a bias term for the attention mechanism Represents the content feature map of layer 1, It is a fine-grained appearance encoder consisting of K transformer layers with zero convolutions, where the zero convolution is a standard 1×1 convolutional layer, whose weights and biases are initialized to zero;
[0087] 4.2.3: Calculating the Attention Mechanism Expression
[0088]
[0089] Among them, K T Represents the transpose of matrix K
[0090] 4.3: Downsample the output of the style-content fusion module;
[0091] 4.4: After four style-content fusion modules and downsampling operations, it is input into the intermediate module;
[0092] 4.5: Then, the output of the intermediate module undergoes four upsampling operations to obtain the predicted noise, and the input x t Remove the noise predicted at this moment and get x t-1 , x t-1 Input into the model, repeat step 4.5, and after continuous denoising, the generated font image is obtained.
[0093] 5) Calculate the mean square error of the predicted noise and the added noise, as well as the sum of the differences between the generated image and the multi-layer feature map of the target image, and calculate the total loss function, which is expressed as
[0094] L total =λ1L MSE +λ2L feature
[0095] Among them, λ1 and λ2 are 0.8 and 0.2 respectively, and the model parameters are adjusted according to the back propagation of the total loss function;
[0096] 6) Give a new style reference image and the Chinese character to be generated and input them into the trained model to generate a new style glyph image.
Claims
1. A small sample font generation method based on a diffusion model, characterized in that: The method comprises the following steps: 1) Build a font image dataset; 2) Add noise to the target generated image multiple times to obtain a noise image that satisfies the Gaussian distribution; 3) Using the Kaiti style of the target generated image’s text A as the content image, and using the images of other characters in the target image’s style B as the style image, the content encoder and style encoder are used to extract structural features and semantic features; 4) Train the diffusion model to predict the noise of the noisy image and gradually remove the noise of the intermediate hidden code. The training target is the original image without noise. The content image and style image are used as conditions of the diffusion model to assist in denoising. A style refinement module and a style-content fusion module are constructed in the diffusion model. Finally, an image of content Chinese character A with style font B is generated. 5) Calculate the mean square error of the predicted noise and the added noise, as well as the sum of the differences between the generated image and the multi-layer feature map of the target image to adjust the model parameters; 6) After the training is completed, the model is tested, and the reference image of the style to be generated and the Chinese characters to be generated are input into the trained model to generate a new style glyph image.
2. The method for generating a small sample font according to claim 1, characterized in that: The step 1) specifically includes: Given a font file with a .ttf suffix, render the font image according to the Chinese characters to be generated.
3. The method for generating a small sample font according to claim 1, characterized in that: The step 2) specifically includes: Add noise to the images in the data set multiple times to obtain pure noise images and obtain training data. The noise addition expression is: where t is the time step, ∈ is Gaussian noise, β t is the variance of the t-th noise addition, which is a constant and increases with the increase of t; x0 is the original image, x t is the image after adding noise; α t It is also a constant, and the calculation formula is: t =1-β t ; is α t The cumulative multiplication from time 1 to time t.
4. The method for generating a small sample font according to claim 1, characterized in that: The step 3) specifically includes: 3.1: Use the content encoder to encode the content image to obtain the content feature map; 3.1.1: The content encoder consists of four convolutional blocks, each of which contains the Identity() normalization function, the ReLU activation function, the reflection padding function to keep the input and output dimensions consistent, and the convolutional layer; the initial number of input channels of the convolutional layer is 1, and the number of output channels is 32, 64, 128, and 256 respectively; 3.2: Use the style encoder to encode the style image to obtain the style feature map; 3.2.1: The style encoder consists of six convolutional blocks; 3.2.2: For the first five convolutional blocks, each convolutional block contains the Identity() normalization function, the ReLU activation function, the reflection padding function to keep the input and output dimensions consistent, and the convolutional layer; the initial input channel of the convolutional layer is 1, and the output channels are 64, 128, 256, 512, and 1024 respectively; 3.2.3: The last convolution block contains the instance normalization function, the ReLU activation function and the convolution layer. The input and output channels of the convolution layer are both 1024, and the convolution kernel size is 1×1.
5. The method for generating a small sample font according to claim 1, characterized in that: The step 4) specifically includes: 4.1: Construct style refinement module; 4.1.1: Input the style feature map into two convolutional layers respectively to obtain and. At the same time, input the style feature map into another convolutional layer to obtain; 4.1.2: Change the shapes of,, respectively, into in is a real number space, C is the number of channels, H and W are the length and width of the feature map respectively, and N = H × W is the product of length and width; 4.1.3: Then, calculate the style space attention weight The calculation expression is in, represents the transpose of the feature map; Then, the feature map is adjusted by matrix multiplication and element-level addition to obtain the output of the style refinement module. The calculation expression is: in, Represents the matrix A s The transpose of 4.2: Building a style-content fusion module 4.2.1: Each layer of content feature map obtained by encoding the content encoder And the feature map f output by the style refinement module ss Perform a cross-attention mechanism. First, the noise image x input into the model T , represents the noise image at time T, the channel attention mechanism is performed at time t, and then projected to the query matrix Q. The feature maps output by the style refinement module are projected to the key matrix K and the value matrix V respectively; 4.2.2: Building a bias term for the attention mechanism Represents the content feature map of layer 1, It is a fine-grained appearance encoder consisting of M transformer layers with zero convolutions, which is a standard 1×1 convolutional layer whose weights and biases are initialized to zero; 4.2.3: Calculating the Attention Mechanism Expression Among them, K T represents the transpose of matrix K, d is the dimension of Q matrix, Q is the query matrix, K is the key matrix, and V is the value matrix; 4.3: Downsample the output of the style-content fusion module; 4.4: After four style-content fusion modules and downsampling operations, it is input into the intermediate module; 4.5: Subsequently, the output of the middle module undergoes four upsampling operations to obtain the predicted noise. The noise predicted at this moment is removed from the input to obtain the font image. Step 4.5 is repeated to obtain the generated font image after continuous denoising.
6. The method for generating a small sample font according to claim 1, characterized in that: The step 5) specifically includes: 5.1: Calculate the mean square error of the predicted noise and the added noise. The calculation expression is L MSE =||∈-∈ θ (x t ,t,x c ,x s )||, Among them, ∈ is the actual noise added, and ∈θ is the final predicted noise; 5.2: Calculate the sum of the differences between the generated image and the multi-layer feature map of the target image. The calculation expression is Where x0 is the generated image, x target is the target image, VGG is the VGG16 network pre-trained based on ImageNet; 5.3: Calculate the total loss function, the expression is L Total =λ1L MSE +λ2L feature Among them, λ1 and λ2 are the weight coefficients of the two loss functions, respectively, satisfying the condition λ1+λ2=1.
7. The method for generating small sample fonts based on a diffusion model according to claim 1, characterized in that: The step 6) specifically includes: Given a new style reference image and the Chinese characters to be generated, they are input into the trained model to generate a new style glyph image.
Citation Information
Patent Citations
Font generation method and device, equipment and storage medium
CN118070752A
Single sample font generation method based on multi-scale style fusion and font control
CN118365744A
Method for creating a virtual image reflecting a style according to user requirements and computing device for the same method
KR102760577B1
Method and system for quantitative susceptibility mapping reconstruction in low-field magnetic resonance imaging (MRI) system
US20230305089A1
Ultrasound image denoising model establishing method and ultrasound image denoising method
WO2022083026A1
Cited By
Multi-style picture book generation method based on diffusion model
CN119228633A
Method and system for quickly generating few-sample Chinese fonts based on diffusion model
CN119692304A
Fine controllable image stylization method based on diffusion model
CN119762330A
A finely controllable image stylization method based on diffusion model
CN119762330B
Small sample infrared data generation method
CN119941548A
