Handwritten font generation method based on ResNeSt and diffusion model

Abstract

The invention discloses a handwritten font generation method based on comparative learning and a diffusion model, and the method comprises the steps: employing ResNeSt as a model to extract a backbone network of a feature map, and inputting the vector of the feature map into a Transformer encoder for global feature extraction; a CNN (Convolutional Neural Network) decoder and a Transform decoder are used to generate a handwritten font image; carrying out noise adding processing on the generated font image; then, using a diffusion model to predict noise from the image with noise, and finally, calculating an original image through a denoising formula; a contrast loss function of the font image generated by the diffusion model and the real image is calculated by using contrast learning, and the contrast loss function is used for promoting the generated image to be close to the real image; the problem that fonts generated by a current model are not vivid enough is solved, the quality of generated handwritten fonts is effectively improved, and the generated fonts and real fonts are more consistent in content and style.

Description

Technical Field

[0001] The invention relates to the field of handwriting font generation, and in particular to a handwriting font generation method based on ResNeSt and a diffusion model. Background Art

[0002] Handwritten font generation is a task with important practical applications and research value. It aims to imitate the writer's handwriting style to produce realistic handwritten fonts. It plays an important role in digital document processing, personalized font design, cultural protection and other fields. With the development of deep learning technology, automatically generating high-quality and realistic handwritten fonts is of great value in improving user experience and reducing labor costs.

[0003] Current handwritten font generation methods fail to fully capture the fine-grained features of fonts during feature extraction, resulting in insufficient fidelity of the generated fonts. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art and effectively improve the quality of generated handwritten fonts, the present invention proposes a handwritten font generation method based on ResNeSt and a diffusion model.

[0005] The technical solution of the present invention is implemented as follows: a handwriting font generation method based on ResNeSt and diffusion model, comprising the steps of:

[0006] S1, using ResNeSt as the backbone network of the model to extract feature maps, inputs the feature map vector into the Transformer encoder for global feature extraction, including the following steps:

[0007] S11, divide the input X into K groups according to the channel dimension C, and use a new hyperparameter R to represent the number of splits for further division of the feature map group, so the total number of feature groups is G = KR;

[0008] S12, a series of transformations {F1, F2, ... F G}, then the intermediate process of each group is represented by U i =F i (X), where i∈{1,2,...G};

[0009] S13, through the fusion operation, several basis arrays are combined into a combined representation, and the kth basis array can be represented as, where The shape of the output feature map can be represented by H, W and C, k∈[1,K];

[0010] S14, uses global average pooling across spatial dimensions to aggregate global contextual information with embedded channel statistics;

[0011] S15, concatenate the basis array representations along the channel dimension, and then obtain the final output through residual connection;

[0012] S16, passes the output to the Transformer encoder for global feature extraction;

[0013] S2, generates handwritten font images using CNN decoder and Transformer decoder;

[0014] S3, performing noise processing on the generated handwritten font image;

[0015] S4, predicts the noise from the noisy image using the diffusion model and calculates the original image;

[0016] S5, the handwritten font image generated by the diffusion model and the real image are compared using contrastive learning to calculate the contrast loss function between the two, which is used to make the generated image closer to the real image.

[0017] Furthermore, the noise reduction process in S3 includes the following steps:

[0018] The generated font image is processed with noise. Starting from the real data distribution z0, the diffusion model adds a small amount of noise in each step to generate a series of intermediate states z1, z2, …, z T , until the data becomes close to a pure noise distribution z T , the process of gradually adding noise to the sample data is described in formula (1):

[0019]

[0020] where z t is the noise transformed at time step t∈{0,1,…,T}, β s is a parameter that controls the noise intensity and increases with the increase of time t. Noise is gradually added to the sample data so that z T becomes a standard normal distribution, z T Denoted as z0 and noise variable ∈ t The linear combination of is shown in formula (2):

[0021]

[0022] in During the training process, the neural network minimizes z0 and z t The L2 loss between them is used to predict the added noise, Φ θ (z t ,t) is the mean predicted by the neural network, and the loss function is shown in formula (3):

[0023]

[0024] Among them, L diff is the loss function,

[0025] Furthermore, S4 comprises the steps of: in the inference process, using the initial noise z t , the trained neural network implements iterative denoising operation on the data sample z0, and the iterative denoising operation process is shown in formula (4):

[0026]

[0027] in Represents the noise variance at time step t in the model, defines the noise level at each time step in the reverse process, and accelerates the denoising process of the model by sampling a small number of fractional function evaluations.

[0028] Furthermore, S5 includes the steps of: using normalized temperature scaled cross entropy loss (NT-Xent) as contrast loss, and guiding the finally generated font to be closer to the real font by calculating the contrast loss between the font image generated after being processed by the diffusion model and the real image, and the contrast loss is shown in formula (5):

[0029]

[0030] Where sim(u,v) represents the cosine similarity, τ is the temperature parameter, G(z) is the font image generated after the diffusion model denoising operation, x is the real font image, and N is the size of a batch.

[0031] The beneficial effect of the present invention is that, compared with the prior art, the present invention provides a new method for handwritten font generation, which overcomes the problem of insufficient fidelity of fonts generated by the prior art and effectively improves the quality of the generated handwritten fonts. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of a handwriting font generation method based on ResNeSt and diffusion model of the present invention;

[0033] Figure 2 It is a structural schematic diagram of a handwriting font generation system based on ResNeSt and diffusion model of the present invention;

[0034] Figure 3 Schematic diagram of the system structure of image noise addition in the present invention;

[0035] Figure 4 This is a schematic diagram of the system structure for predicting noise in the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] See also Figure 1 and Figure 2 , a handwriting font generation method based on ResNeSt and diffusion model, comprising the steps of:

[0038] S1, using ResNeSt as the backbone network of the model to extract feature maps, and then inputting the feature map vector into the Transformer encoder for global feature extraction, including the following steps:

[0039] S11, first divide the input X into K groups according to the channel dimension C, and then use a new hyperparameter R to represent the number of splits to further divide the feature map group, so the total number of feature groups is C = KR;

[0040] S12, a series of transformations {F1, F2, ... F G}, then the intermediate process of each group is represented by U i =F i (X), where i∈{1,2,...G};

[0041] S13, through the fusion operation, several basis arrays can be formed into a combined representation, and the kth basis array can be represented as, where The shape of the output feature map can be represented by H, W and C, k∈[1,K];

[0042] S14, uses global average pooling across spatial dimensions to aggregate global contextual information with embedded channel statistics;

[0043] S15, finally concatenate the basis array representations along the channel dimension, and then obtain the final output through residual connection;

[0044] S16, passes the output to the Transformer encoder for global feature extraction;

[0045] S2, generates handwritten font images using CNN decoder and Transformer decoder;

[0046] S3, performing noise processing on the generated handwritten font image;

[0047] S4, predicts the noise from the noisy image using the diffusion model and calculates the original image;

[0048] S5, the handwritten font image generated by the diffusion model and the real image are compared using contrastive learning to calculate the contrast loss function between the two, which is used to make the generated image closer to the real image.

[0049] The noise reduction process in S3 includes the following steps:

[0050] The generated font image is subjected to noise processing, such as Figure 3 As shown in Figure 2, starting from the true data distribution z0, the diffusion model adds a small amount of noise at each step to generate a series of intermediate states z1, z2, …, z T , until the data becomes close to a pure noise distribution z T , the process of gradually adding noise to the sample data is described in formula (1):

[0051]

[0052] where z t is the noise transformed at time step t∈{0,1,…,T}, β s is a parameter that controls the noise intensity and increases with the increase of time t. Noise is gradually added to the sample data so that z T becomes a standard normal distribution, represented by z0 and the noise variable ∈ t The linear combination of is shown in formula (2):

[0053]

[0054] in During the training process, the neural network minimizes z0 and z t The L2 loss between is used to predict the added noise, such as Figure 4 As shown, Φ θ (z t ,t) is the mean predicted by the neural network, and the loss function is shown in formula (3):

[0055]

[0056] Among them, L diff is the loss function,

[0057] Furthermore, S4 comprises the steps of: in the inference process, using the initial noise z t , the trained neural network implements iterative denoising operation on the data sample z0, and the iterative denoising operation process is shown in formula (4):

[0058]

[0059] in Represents the noise variance at time step t in the model, defines the noise level at each time step in the reverse process, and accelerates the denoising process of the model by sampling a small number of fractional function evaluations.

[0060] The S5 comprises the steps of: using normalized temperature scaled cross entropy loss (NT-Xent) as contrast loss, and guiding the finally generated font to be closer to the real font by calculating the contrast loss between the font image generated after being processed by the diffusion model and the real image, and the contrast loss is shown in formula (5):

[0061]

[0062] Where sim(u,v) represents the cosine similarity, τ is the temperature parameter, G(z) is the font image generated after the diffusion model denoising operation, x is the real font image, and N is the size of a batch.

[0063] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A handwriting font generation method based on ResNeSt and diffusion model, comprising the steps of: S1, using ResNeSt as the backbone network of the model to extract feature maps, inputs the feature map vector into the Transformer encoder for global feature extraction, including the following steps: S11, divide the input X into K groups according to the channel dimension C, and use a new hyperparameter R to represent the number of splits for further division of the feature map group, so the total number of feature groups is G = KR; S12, a series of transformations {F1, F2, ... F G }, then the intermediate process of each group is represented by U i =F i (X), where i∈{1,2,...G}; S13, through the fusion operation, several basis arrays are combined into a combined representation, and the kth basis array can be represented as, where The shape of the output feature map can be represented by H, W and C, k∈[1,K]; S14, uses global average pooling across spatial dimensions to aggregate global contextual information with embedded channel statistics; S15, concatenate the basis array representations along the channel dimension, and then obtain the final output through residual connection; S16, passes the output to the Transformer encoder for global feature extraction; S2, generates handwritten font images using CNN decoder and Transformer decoder; S3, performing noise processing on the generated handwritten font image; S4, predicts the noise from the noisy image using the diffusion model and calculates the original image; S5, the handwritten font image generated by the diffusion model and the real image are compared using contrastive learning to calculate the contrast loss function between the two, which is used to make the generated image closer to the real image.

2. The handwritten font generation method based on ResNeSt and diffusion model as claimed in claim 1, wherein the noise reduction process in S3 comprises the steps of: The generated font image is processed with noise. Starting from the real data distribution z0, the diffusion model adds a small amount of noise in each step to generate a series of intermediate states z1, z2, …, z T , until the data becomes close to a pure noise distribution z T , the process of gradually adding noise to the sample data is described in formula (1): where z t is the noise transformed at time step t∈{0,1,…,T}, β s is a parameter that controls the noise intensity and increases with the increase of time t. Noise is gradually added to the sample data so that z T becomes a standard normal distribution, z T Denoted as z0 and noise variable ∈ t The linear combination of is shown in formula (2): in During the training process, the neural network minimizes z0 and z t The L2 loss between them is used to predict the added noise, Φ θ (z t ,t) is the mean predicted by the neural network, and the loss function is shown in formula (3): in, L diff is the loss function, 3. The handwriting font generation method based on ResNeSt and diffusion model as claimed in claim 1, wherein S4 comprises the step of: in the inference process, using the initial noise z t , the trained neural network implements iterative denoising operation on the data sample z0, and the iterative denoising operation process is shown in formula (4): in Represents the noise variance at time step t in the model, defines the noise level at each time step in the reverse process, and accelerates the denoising process of the model by sampling a small number of fractional function evaluations.

4. The handwritten font generation method based on ResNeSt and diffusion model as claimed in claim 1, wherein S5 comprises the step of: using normalized temperature scaled cross entropy loss (NT-Xent) as contrast loss, and calculating the contrast loss between the font image generated after being processed by the diffusion model and the real image, thereby guiding the finally generated font to be close to the real font, and the contrast loss is shown in formula (5): Where sim(u,v) represents the cosine similarity, τ is the temperature parameter, G(z) is the font image generated after the diffusion model denoising operation, x is the real font image, and N is the size of a batch.

Images