Method and system for fast generation of Chinese character fonts based on diffusion model with few samples

Abstract

The application discloses a few-sample Chinese character font rapid generation method and system based on a diffusion model, which comprises a low-resolution generation model module and a super-resolution model module, and can further comprise a diffusion model distillation module; a conditional implicit diffusion model with component-aware encoder, namely a low-resolution generation model, is constructed, which is used for generating high-quality low-resolution fonts; a super-resolution model based on a cascade diffusion model is constructed, which is used for expanding the low-resolution generation result to high resolution; the diffusion model distillation module can further distill the trained low-resolution generation model and the super-resolution model into a one-step generation model, so as to improve the font image generation efficiency. The application can extract and migrate the font style from limited input font image samples, obtain font image generation results which are significantly improved in the matching degree of style and the accuracy of font structure, and obtain high-quality vector fonts through a vectorization tool.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer graphics processing and artificial intelligence, and relates to Chinese font generation technology, in particular to a few-shot high-resolution Chinese font fast generation method and system based on a diffusion model, which can generate a complete Chinese font library for a new font given a limited input glyph sample. BACKGROUND

[0002] Font generation refers to the task of generating a complete font library for a new font given a limited input glyph sample. Automated font generation reduces the time-consuming and labor-intensive human burden, and therefore has wide application prospects and is widely concerned in the academic field. In particular, for a character system such as Chinese, Japanese or Korean, which contains tens of thousands of characters, a fast and effective few-shot font generation method not only speeds up the design of high-quality commercial font products, but also can be applied to personalized handwriting generation, scene text image editing, data augmentation for optical character recognition (OCR) and other application scenarios.

[0003] Existing font generation methods are generally divided into two categories: directly generating vector glyphs, or first generating high-quality raster images, and then vectorizing them into vector glyphs. This study focuses on the second method, i.e., developing a font generation system that takes a few-shot glyph image of the target style as input and outputs high-quality, high-resolution glyph images of the remaining characters in the font library. High-quality vector glyphs can be obtained through existing vectorization tools. Compared with the method of directly generating vector glyphs, although processing high-resolution images involves higher computational complexity and still requires human intervention in the vectorization process, the relatively more structured image modality is easier to process, and there are a large number of mature and advanced generation models available for use and reference. On the other hand, the method of directly generating vector glyphs can only handle simple shapes, which limits their application in font generation for character systems containing high topological complexity glyphs (such as Chinese). If a high-quality image modality generation result can be obtained, high-quality and high-complexity vector glyph results can be obtained through mature vectorization tools.

[0004] Despite some progress in recent years, few-shot font generation remains an unsolved and extremely challenging task due to the complexity of character structure and the creativity and diversity of font design. Most previous methods are built on the basis of generative adversarial networks (GANs) to train image-to-image translation networks in an adversarial manner. However, it is well known that GANs are difficult to train, difficult to scale, suffer from mode collapse, and lack diversity, limiting their potential to generate high-quality results. On the other hand, in the image modality, diffusion models have been shown to produce better results than state-of-the-art generative adversarial models and can be easily scaled to larger datasets. However, the iterative inference process of diffusion models makes them more time-consuming than generative adversarial models with single-step inference, hindering their application in real-world scenarios; in addition, it is also necessary to design how to properly integrate the prior knowledge of the font into the model architecture based on diffusion models.

[0005] Diffusion-based generative models have recently achieved unparalleled performance in many image synthesis tasks and generation tasks of other modalities such as video, audio, text, etc. Latent Diffusion Models, a variant of diffusion models that greatly reduces the computational cost of diffusion models, are widely used in various conditional generation tasks, including image editing, text-to-image generation, etc. Cascaded diffusion models, a variant of diffusion models that extends diffusion models to high-resolution generation, successively upsample the generated image to a higher resolution and have been successfully applied to various tasks that require high-resolution generation. Diff-Font (He, Haibin, et al. "Diff-Font: Diffusion Model for Robust One-Shot Font Generation." arXiv preprint arXiv:2212.05895. 2022.) and FontDiffuser (Yang, Zhenhua, et al. "FontDiffuser: One-Shot Font Generation via Denoising Diffusion with Multi-Scale Content Aggregation and Style Contrastive Learning." arXiv preprint arXiv:2312.12142. 2023.) have attempted to apply diffusion models to font generation tasks and have achieved impressive results. Although the above methods can generate clear results using the generation capabilities of diffusion models, the pre-trained style features or style learning strategies they use often fail when faced with unknown complex font styles. Moreover, they construct models based on pixel space and use multi-step sampling, which limits the practical application and scalability to higher resolutions of the method. SUMMARY

[0006] To overcome the deficiencies of the prior art described above, the present application proposes a font generation method and system based on a diffusion model, which supports one-step font generation and high-resolution generation, and realizes high-quality, high-speed, and high-resolution few-shot font generation.

[0007] Firstly, the present application adds prior knowledge about component splitting, which is very useful for font style learning and transfer. The present application uses attention mechanism to realize the reuse of component-level style information, and with the powerful generation ability of diffusion model, it learns how to effectively transfer diverse and complex font styles to the target glyph; in addition, in order to adapt to the smaller input sample size (such as less than 10) in practical application, the present application introduces a reference sample selection strategy, so that the model has the ability to extract global and local style information. Secondly, in order to further realize higher resolution generation, the present application designs a super-resolution process, so that the loss of detail style and shape at low resolution can be recovered in the super-resolution process, thereby narrowing the gap between image modalities and vector modalities, and realizing high-quality, high-speed, high-resolution few-shot font generation. Finally, in order to solve the problem of slow inference speed of diffusion model, the present application applies the score distillation sampling technology in the text-to-3D generation technology to realize efficient one-step generation with almost no quality loss.

[0008] For the convenience of explanation, the present application defines the following terms:

[0009] Token: The smallest unit of language processed by the attention mechanism.

[0010] Auto-encoder: Auto-encoder.

[0011] U-Net: A convolution-based deep learning model architecture proposed in the literature (Ronneberger, Olaf, et al. "U-net: Convolutional networks for biomedical image segmentation." Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. Springer International Publishing, 2015.).

[0012] Score Distillation Sampling (SDS): Score distillation sampling method proposed in the literature (Poole, Ben, et al. "Dreamfusion: Text-to-3d using 2d diffusion." arXiv preprint arXiv:2209.14988 (2022).).

[0013] The technical scheme provided by the present application is as follows:

[0014] A few-shot high-resolution Chinese character font fast generation method and system based on a diffusion model. The system proposed by the present application includes three parts: the first part includes a conditional implicit diffusion model with a component-aware encoder, which is used to generate high-quality low-resolution results (resolution 64x64); the second part includes a conditional super-resolution cascade diffusion model constructed based on a cascade diffusion model super-resolution technology, which is used for super-resolution to generate high-resolution glyphs; and the third part includes a diffusion model distillation technology, which is used to distill the trained conditional implicit diffusion model with a component-aware encoder and the conditional super-resolution cascade diffusion model into a one-step generation model.

[0015] The first part, a conditional diffusion model and a component-aware encoder are used to construct a conditional implicit diffusion model with a component-aware encoder, so as to realize high-quality low-resolution font generation.

[0016] According to the core idea of the diffusion model, the present application gradually adds noise to the real data until the data is converted into pure noise, and then estimates the distribution of the real data by learning a network that gradually converts the pure noise back to the real data. Specifically, given a character content reference image x c rendered using a standard font (such as Helvetica), and k style reference images from the target font, the goal is to generate a glyph image that has the style of the target font and matches the target character content. The present application uses a component-aware encoder C to encode the information of the input content and the style reference image, and finally obtains the glyph content and style encoding features y=C(x c ,X s ), and uses the principle of implicit diffusion model to estimate the distribution of the target generated picture using the conditional implicit diffusion model ∈ θ , i.e. the constructed low-resolution generation model, so as to finally achieve the purpose of generating the target picture according to the style features (as a conditional input). Let x0 be the glyph target picture, z0 be the implicit expression corresponding to the hidden space defined by the auto-encoder Auto-encoder of the glyph target picture encoded into the conditional implicit diffusion model, and the corresponding diffusion model training loss is as follows:

[0017]

[0018] , wherein ∈ is Gaussian noise, y is the glyph content and style encoding feature, z t is the result of adding noise to z0 to a noise intensity t, i.e. t z t 0+σ t ∈, α t and σ tis a sequence of non-zero variables, t is the noise strength, taking values between 0 and T, T = 1000. is the expectation; || || is the L2 distance.

[0019] In addition, in the scene sampled by the multi-step diffusion model, the classifier-free guidance method of guiding the sample to move to the high-density area of the distribution determined by the condition is often used to trade diversity for higher quality. The classifier-free guidance method defined in the present application for content and style two types of condition inputs corresponds to the noise estimated as follows, where two coefficients s c and s s respectively control the association strength of the generated result with the input content and style conditions respectively; is expressed as:

[0020]

[0021] where is the result of replacing x c with a full zero vector of the same size, is the result of replacing X s with a full zero vector of the same size, is the noise estimation result of the classifier-free guidance.

[0022] Since the style of the font exists in different structural levels of the glyph image, different characters share different common style information, and in an ideal case, the model is expected to pay attention to different relevant parts of the style reference at different structural levels of the input content. The present application uses a multi-head cross-attention mechanism to achieve this intention. Specifically, the component-aware encoder is composed of a style encoder E s , a content encoder E c and a cross-attention module. The style encoder encodes all k style reference images into style features i = 1, …, k, respectively, and the content encoder encodes the content reference image into a content feature Each style feature is represented as h s × w s tokens of dimension d s , and the content feature is represented as h c × w c tokens of dimension d c ; and cross-attention operation is performed, expressed as:

[0023]

[0024] where is the projection matrix corresponding to the i-th attention mechanism; y is the glyph content and style encoding feature; Attention is the attention mechanism, softmax is a function that maps an arbitrary real-valued vector to a probability distribution, Q is the query matrix, K is the key matrix, and V is the value matrix; is the f c the flattened features, is the f are concatenated and flattened. In this way, through the attention mechanism, each fine-grained position on the content feature map learns to focus on the relevant position on the style feature map, and the relevant style features are combined to form the final conditional input.

[0025] To guide this learning process, we provide the style encoder with pre-designed and selected style references corresponding to the components and strokes contained in the content characters. The present application defines a character set containing 775 commonly used components and strokes that can cover all Chinese characters contained in the Chinese characters in the GB2312 standard as a style reference set, as a model training set. For any character in the training set, the corresponding style reference is selected as the character containing the component or stroke corresponding to the character in the style reference set, so as to guide the model to learn the style extraction and migration of the component and stroke level.

[0026] The second part is to construct a conditional super-resolution cascade diffusion model based on the cascade diffusion model super-resolution technology to realize high-resolution font generation.

[0027] The motivation for turning the generation process to a higher resolution area is that at low resolution, a large amount of information will be lost, such as style details or dense strokes, so an upsampling process is needed to recover these lost information, and two is that through upsampling, the gap between rasterized images and vector glyphs can be narrowed to achieve a higher quality vector generation result.

[0028] The present application uses the conditional super-resolution cascade diffusion model to continuously upsample the high-quality low-resolution font result obtained in the first part to a higher resolution, i.e. from 64 to 256, and then from 256 to 1024, thereby generating a high-resolution glyph image. The present application does not directly apply other existing image super-resolution models, but uses the same model as the first part, i.e. the implicit diffusion model and the component-aware encoder, and makes three improvements to the super-resolution model: 1) the present application replaces the content reference image with a low-resolution image, and replaces the input noise z tThe low-resolution image is inserted into the model in a connected manner; in this way, similar to the role of the cross-attention module in the style feature condition input in the low-resolution model, the present application guides the attention of each position in the low-resolution image to the relevant position in the style reference to restore the lost details; 2) In order to carry out the noise condition enhancement operation which is essential to bridge the gap between the generated image and the training image, the present application encodes the low-resolution image into the latent space, adds noise to it with Gaussian noise, and then decodes it back to the image space as an enhanced low-resolution image input; In addition, the present application also adds an additional condition s to the U-Net part of the conditional implicit diffusion model with the part-aware encoder in the first part, that is, the noise level added in the noise condition enhancement process; 3) In order to solve the problem of increased memory occupation caused by high resolution, the present application adds additional down-sampling layers to the style and content encoders in the part-aware encoder. Since the style and content encoders have mastered the knowledge of extracting and migrating the required font style, the present application uses the weights of the low-resolution generation model to initialize the super-resolution model.

[0029] The third part is to realize single-step sampling high-speed font generation based on 3D distillation technology.

[0030] Image diffusion models usually need dozens of steps of sampling to produce effective and high-quality output results. For the model obtained in the first part, 10 steps of 1storder SDE sampler can produce a good enough result, but 10 steps of reasoning still cannot achieve fast and real-time generation. The present application proposes to use the Score Distillation Sampling (SDS) based strategy to distill the conditional implicit diffusion model with the part-aware encoder and the conditional super-resolution cascade diffusion model into a one-step generation model.

[0031] Score Distillation Sampling (SDS) was originally proposed for the text-to-3D generation task, that is, using the knowledge of a pre-trained 2D text-to-image diffusion model to guide the 3D model to render realistic 2D images from random angles, thereby realizing the learning of the 3D model. This distillation process can also be used to reduce the sampling steps of the model, that is, replacing the 3D model with a one-step sampling diffusion model to generate more realistic images guided by the multi-step diffusion model.

[0032] The distillation process includes a frozen teacher model ∈ θ (z t ,t,y) and a one-step generated student model, the weights of the student model are initialized to ∈ φ (z t ,t,y) = ∈ θ (z tT, y). The noise strength t in the student model is fixed as T to ensure the model starts from pure noise in the inference process. During the training process, the student model will move the samples to be denoised z T = a T z0+ s T and the glyph content and style encoding features y to generate samples Similar to the way SDS method processes 2D images rendered from 3D models, the goal is to optimize the parameters f such that looks close to samples from the teacher model. To do this, we need to move the samples towards the high probability density region defined implicitly by the teacher model for each noise strength t, which is equivalent to optimizing the following objective:

[0033]

[0034] where L SDS is the optimization objective; is the expectation; a t and s t are a series of predefined non-zero variables; D KL is the KL divergence; is the marginal distribution of the diffusion model forward process (i.e. ); p θ (z t | y, t) is the marginal distribution defined implicitly by the teacher model at noise strength t, and w is a constant. Since the estimated noise added to the data samples is equivalent to the score function of the estimated noisy data distribution, the approximate gradient of the above optimization objective can be expressed as:

[0035]

[0036] where is the gradient operator; L SDS is the optimization objective of SDS; is the expectation; s is Gaussian noise.

[0037] The noise estimate (s θ (z t , t, y)) in the above can adopt the classifier-free guidance noise estimate Only the weights of the diffusion model are updated in the optimization process, i.e. the weights of the component perceptual encoder are fixed during the training process.

[0038] Based on the above-mentioned diffusion model-based few-shot high-resolution Chinese character font rapid generation system constructed by the present application, the few-shot high-resolution Chinese character font rapid generation method comprises the following steps:

[0039] First, construct a style reference selection mapping;

[0040] Construct a style reference selection mapping. In specific implementation, based on the 1032 components and 339 strokes defined for Chinese characters, a character set that covers all components and strokes contained in Chinese characters in the GB2312 standard is constructed as a style reference set. The style reference for model training is selected from this set.

[0041] For any character c in the training set, a component-level character reference mapping and a stroke-level character reference mapping are established. The former consists of all characters in the style reference set that contain the components contained in c, and the latter consists of all characters in the style reference set that contain the strokes contained in c.

[0042] In the second step, the style reference corresponding to each target image during training is selected based on the style reference selection mapping, and the implicit diffusion model is trained to generate low-resolution glyph image results;

[0043] This step first builds a conditional implicit diffusion model with a component-aware encoder, which consists of a style encoder E s , a content encoder E c and a cross attention module. The style encoder encodes all k style reference images into style features i=1,…,k, the content encoder encodes the content reference image into content features Each style feature is represented as a dimension d s h s ×w s tokens, representing content features as a dimension of d c h c ×w c tokens and perform cross-attention operations.

[0044] We use classifier-free guidance to sample the constructed conditional implicit diffusion model with a component-aware encoder, and define a classifier-free guidance method for conditional inputs of multiple style features. We design and train the implicit diffusion model using a loss function, obtaining a trained conditional implicit diffusion model with a component-aware encoder. This trained conditional implicit diffusion model with a component-aware encoder is then used to generate high-quality low-resolution results.

[0045] The third step is to train a conditional super-resolution cascade diffusion model using the parameters of the trained low-resolution generative model (i.e., the conditional implicit diffusion model with the component-aware encoder) as the initial parameters.

[0046] Using the conditional super-resolution cascade diffusion model, the low-resolution results are continuously upsampled to higher resolutions, i.e., from 64 to 256 and then from 256 to 1024, thereby generating high-resolution glyph images.

[0047] The model in the first step, i.e. the implicit diffusion model with component-aware encoder, is used as the initialization parameters, and three improvements are made to the conditional implicit diffusion model with component-aware encoder for the task of super-resolution:

[0048] 1) For the corresponding low-resolution image, replace the content reference image with the low-resolution image, and insert it into the U-Net part of the conditional implicit diffusion model with component-aware encoder in the form of connecting the low-resolution image with the input noise z t ;

[0049] 2) Encode the low-resolution image into the latent space, add Gaussian noise to the corresponding latent representation of the low-resolution image in the latent space, and then decode it back to the image space as an enhanced low-resolution image input;

[0050] In addition, an additional condition s is added to the U-Net part of the conditional implicit diffusion model with component-aware encoder, i.e. the noise level added in the noise condition enhancement process;

[0051] 3) Additional down-sampling layers are added to the style and content encoders in the component-aware encoder to address the increased memory usage problem caused by high resolution.

[0052] In the fourth step, the conditional implicit diffusion model with component-aware encoder and the conditional super-resolution cascade diffusion model are shortened to single-step sampling using 3D distillation technology.

[0053] In implementation, the diffusion model distillation is based on the Score Distillation Sampling (SDS) strategy.

[0054] Let x0 be the font target picture, z0 be the latent representation corresponding to the latent space defined by the autoencoder in the conditional implicit diffusion model, α t and σ t be a series of predefined non-zero variables. Taking the distillation of the conditional implicit diffusion model with component-aware encoder as an example (the distillation process of the conditional super-resolution cascade diffusion model can be similarly derived), the distillation process includes a frozen teacher model ∈ θ (z t ,t,y) (i.e. the conditional implicit diffusion model with component-aware encoder) and a one-step generated student model, where z t = α t z0+σ t ∈ is the latent representation to be denoised, t is the noise intensity, and y is the font content and style encoding features. The weights of the student model are initialized by the teacher model as ∈ φ (z t ,t,y) = ∈θ (z t ,T,y). The noise intensity t in the student model is fixed to the maximum value T supported by the teacher model.

[0055] During the training process of the student model, the student model is based on the implicit expression z to be denoised T =α T z0+σ T ∈ and glyph content and style encoding features y to generate samples In order to optimize the model parameter φ, the sample Looks close to samples from the teacher model, optimizing the following objective:

[0056]

[0057] Among them, L SDS To optimize the goal; is the expectation; t and σ t is a series of predefined non-zero variables; D KL is the KL divergence; is the forward process of the diffusion model (i.e. ) marginal distribution, p θ (z t |y,t) is the marginal distribution implicitly defined by the teacher model at noise intensity t, and w is a constant. The gradient of the optimization objective is approximately:

[0058]

[0059] in, is the gradient operator; L SDS The optimization goal of SDS; is the expectation; ∈ is Gaussian noise.

[0060] The noise estimation can be performed using a version without classifier guidance. During the optimization process, only the weights of the diffusion model are updated, i.e., the weights of the perceptual encoder of the fixed component are fixed during training.

[0061] By inputting a limited number of glyph samples of a new font and following the above steps, a small number of high-resolution Chinese fonts can be quickly generated based on the diffusion model, thus generating a complete Chinese character library of the font.

[0062] Compared with the prior art, the present invention has the following beneficial technical effects:

[0063] The application provides a fast generation method and system of a few-sample high-resolution Chinese character font based on a diffusion model. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 is a flow block diagram of the font generation method provided by the application.

[0065] Figure 2 is a reference selection process schematic diagram of the application.

[0066] Figure 3 is a generation result comparison of the application and each existing method on unseen fonts and seen characters.n_ref is the input style reference sample size.

[0067] Figure 4 is a generation result comparison of the application and each existing method on unseen fonts and unseen characters.n_ref is the input style reference sample size.

[0068] Figure 5 is a detail style visualization result recovered by the super-resolution model in the application.

[0069] Figure 6 is a vector font result obtained by applying a vectorization tool to the high-resolution font image result in the application.

[0070] Figure 7 is a generation result comparison of the single-step model (after distillation) and its teacher model (before distillation) of the application on unseen fonts and seen characters.

[0071] Figure 8 is a generation efficiency comparison of the application and each existing method.

[0072] Figure 9 is a comparison of the application and each existing method when applied to font color and texture style transfer.

[0073] Figure 10 is a generation result of the application applied to cross-language generation. DETAILED DESCRIPTION

[0074] The application will be further described by examples in connection with the drawings, but the scope of the application is not limited in any way.

[0075] The application provides a few-shot high-resolution Chinese font fast generation technology and system based on diffusion model, which consists of three parts, the process and network structure are shown in the accompanying Figure 1 Fig. 1, wherein the first part realizes high-quality generation based on conditional diffusion model and component-aware encoder (Fig. 2, stage A), the second part realizes high-resolution font generation based on cascaded diffusion model super-resolution technology (Fig. 2, stage B1), and the third part realizes single-step sampling high-speed font generation based on 3D distillation technology (Fig. 2, stage B2). Figure 1 Figure 1 Figure 1

[0076] The specific implementation includes the following steps:

[0077] 1) Construct a style reference selection map, as shown in the accompanying Figure 2 Fig. 3.

[0078] In order to guide the style learning process of different levels of glyphs, the application provides the style encoder with pre-designed and selected style references corresponding to the components and strokes contained in the content characters. In EasyFont (Lian, Zhouhui, et al. "EasyFont: a style learning-based system to easily build your large-scale handwriting fonts." ACM Transactions on Graphics (TOG) 38.1 (2018)), m = 1032 components and n = 339 fine-grained strokes are defined for Chinese characters, and several character sets of different sizes covering different proportions of Chinese characters are constructed. We use the OptSet set defined in the literature, which contains 775 commonly used Chinese characters and can cover all the components and strokes contained in the Chinese characters in the GB2312 standard; set this set as the style reference set, and the style reference for model training is selected from it. For any character c in the training set, let c contain m c components comp c,i ∈{1,…,m},i=1,…,m c , let the i-th component contain n c,i strokes stroke c,i,j ∈{1,…,n},j=1,…,n c,i ​​​Since Chinese characters are usually decomposed into at most 6 components, the number of style references k is set to 6. We establish a component-level character reference map and a stroke-level character reference map, which maps each component comp in character c. c,i Assign a subreference set For component-level character reference mapping, All included components comp in the style reference collection c,i Character composition, stroke-level character reference mapping, The style reference collection contains any {stroke c,i,j ,j=1,…,n c,i} in the character composition. In the training process, the present invention A character is randomly selected from the style reference set for character c, and the remaining k positions are randomly filled. The present invention uses these two mappings with probabilities 1-p and p. During training, OptSet is used as the style reference set; during testing, a new mapping is constructed based on the given reference set.

[0079] 2) Select the style reference corresponding to each target image during training according to the style reference selection mapping, train the implicit diffusion model to generate low-resolution glyph image results, and set p to 0.1.

[0080] 3) Using the trained conditional implicit diffusion model parameters with component-aware encoder as initial parameters, train the super-resolution model.

[0081] 4) Use the 3D distillation technique introduced above to shorten the conditional implicit diffusion model with component perception encoder and the conditional super-resolution cascade diffusion model to single-step sampling, fixing the guidance coefficient s of the classifier guidance sampling c =2.0 and s s =2.0.

[0082] 5) Use the Image Tracer tool in Adobe Illustrator as a vectorization tool to convert the high-resolution glyph images into high-quality vector glyphs.

[0083] During testing, that is, when the user uses it, the style reference image of the target font and the target generated characters are input. The present invention sends the style reference and the target characters rendered in a standard font (such as bold) into the system, and obtains the vector glyph results through a low-resolution generation model, a super-resolution model and a vectorization tool.

[0084] Attachment Figure 3This is a comparison of the generation results of the present invention and existing methods on unseen fonts and seen characters. n_ref is the input style reference sample size. The samples generated by the present invention have fine and sharp details and are consistent with the reference images of various styles. Using different input sample sizes, the model can extract different levels of style information from the input reference sample set. Figure 4 This is a comparison of the results of the present invention and the existing methods on unseen fonts and unseen characters. n_ref is the input style reference sample size. The present invention can also generate high-quality results on characters that have not been seen in training. Figure 5 This is the detailed style visualization result restored by the super-resolution model in this invention. The model in this invention can extract and restore style details from high-resolution style references. Figure 6 This is the vector font result obtained by applying a vectorization tool (Adobe Illustrator) to the high-resolution font image result in the present invention. Since the quality of the high-resolution font image generation result is high enough, the present invention can obtain a high-quality vector font result with precise control points. Figure 7 The figure shows the comparison of the generation results of the single-step model of the present invention and its teacher model on unseen fonts and seen characters. The single-step student model obtained by the present invention faithfully imitates the output of the multi-step teacher model, while the teacher model cannot generate effective results in the single-step generation setting, which proves the effectiveness of the distillation method of the present invention. Figure 8 The single-step model obtained by the present invention is significantly faster than other diffusion-based models and is comparable to the methods based on generative adversarial networks. Figure 9 The present invention is compared with existing methods when applied to font color and texture style transfer. The present invention performs better in terms of content matching and style consistency in all tested styles. Figure 10 This is the generated result of the present invention applied to cross-language generation, that is, using a model trained on Chinese characters to transfer the style of Chinese characters to English and Korean characters.

[0085] The above description clearly and completely describes the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. It should be understood that the examples described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts are within the scope of protection of the present invention.

Claims

1. A method for fast generation of Chinese font based on diffusion model with few samples, characterized in that, By constructing and training a conditional implicit diffusion model with component-aware encoder, i.e., a low-resolution generation model, a high-quality low-resolution font result is generated; based on a cascade diffusion model super-resolution technology, a conditional super-resolution cascade diffusion model, i.e., a super-resolution model, is constructed, and the obtained high-quality low-resolution font result is super-resolved, so that a high-resolution vector glyph font is generated; comprising the following steps: In a first step, a style reference selection mapping is constructed; specifically, based on the components and strokes defined for Chinese characters, a character set covering the components and strokes contained in the Chinese characters in the Chinese encoding standard is constructed as a style reference set, which is used as a model training set; For any character in the model training set, a component-level character reference mapping and a stroke-level character reference mapping are established; wherein the component-level character reference mapping is composed of all characters in the style reference set containing the components contained in the character, and the stroke-level character reference mapping is composed of all characters in the style reference set containing the strokes contained in the character; In a second step, a style reference corresponding to each target picture during training is selected according to the style reference selection mapping, a conditional implicit diffusion model with component-aware encoder, i.e., a low-resolution generation model, is constructed and trained, and is used to generate a low-resolution glyph picture result; The component-aware encoder in the low-resolution generation model includes a style encoder, a content encoder, and a cross-attention module; the style encoder encodes all style reference images into style features, and the content encoder encodes the content reference image into content features; the cross-attention module is used to represent each style feature and content feature as a plurality of tokens and then perform cross-attention operation; The constructed low-resolution generation model is sampled using classifier-free guidance, and a classifier-free guidance method with multiple style feature conditional inputs is adopted; a loss function is designed and used to train the low-resolution generation model, and a trained low-resolution generation model is obtained; a high-quality low-resolution font result is generated by using the trained low-resolution generation model; The low-resolution generation model is sampled using classifier-free guidance, which is represented as: wherein, is the noise estimation result without classifier guidance; s c and s s are the correlation strength coefficients of the control generation result and the input content and style condition, respectively; x c is the character content reference image, X s is the style reference image from the target font; is the result of replacing x c with a full zero vector of the same size, is the result of replacing X s with a full zero vector of the same size; A 3D distillation technology is used to perform diffusion model distillation between the low-resolution generation model and the super-resolution model, so as to shorten the sampling to a single step, and obtain a one-step generation model; In a third step, the conditional super-resolution cascade diffusion model, i.e., the super-resolution model, is trained with the parameters of the trained low-resolution generation model as initial parameters; the trained super-resolution model is used to continuously upsample the low-resolution font result to a higher resolution, thereby generating a high-resolution glyph image; Through the above steps, the fast generation of Chinese font based on diffusion model with few samples is realized.

2. The method of claim 1, wherein the method further comprises: determining a font style of the input Chinese character; and determining a font size of the input Chinese character. In the low-resolution generation model, a style encoder in the part-aware encoder is configured to encode all k style reference images into style features respectively, denoted as: The content encoder in the component-aware encoder is used to encode the content reference image into content features, denoted as f c ; Each style feature is represented as a plurality of tokens with a dimension, the content feature is represented as a plurality of tokens with a dimension, and cross-attention operation is performed, which is represented as: wherein y is a character content and style encoding feature; Attention is an attention mechanism; Q is a query matrix, K is a key matrix, and V is a value matrix; are the projection matrices corresponding to the i-th attention mechanism, respectively; and softmax is a probability distribution mapping function. is a feature obtained by flattening the content features, is a feature obtained by concatenating and flattening the style features.

3. The method of claim 2, wherein the method further comprises: determining a font style of the input Chinese character; and determining a font size of the input Chinese character. In the second step, the loss function used to train the low-resolution generative model is represented as: wherein x0is the glyph target image; ∈ is a Gaussian noise, t is the noise intensity, ∈ θ represents the conditional implicit diffusion model; z t is the z0noised to the t corresponding implicit representation; z0is the implicit representation corresponding to the latent space defined by the autoencoder that encodes the glyph target image into the conditional implicit diffusion model; represents the expectation; ||| represents the L2 distance.

4. The method of claim 1, wherein the method further comprises: determining a font style of the input Chinese character; and determining a font size of the input Chinese character. In the third step, the low-resolution generation model is used to train the super-resolution model, including the following improvements: 1) by using the corresponding low-resolution image as the content reference image, and connecting it with the input noise, the low-resolution generation model is inserted into the convolutional deep learning model U-Net; 2) encode the low-resolution image to the latent space, add noise to the latent representation of the low-resolution image in the latent space using Gaussian noise, and then decode it back to the image space as an enhanced low-resolution image input; 3) add additional down-sampling layers to the style and content encoders in the component-aware encoder to address the increased memory footprint caused by high resolution.

5. The method of claim 4, wherein the method further comprises: determining a font style of the input Chinese character; and determining a font size of the input Chinese character. In improvement 2), an additional condition is added to the convolutional deep learning model U-Net in the low-resolution generation model, that is, a noise level is added in the noise condition enhancement process.

6. The method of claim 1, wherein the method further comprises: determining a font style of the input Chinese character; and determining a font size of the input Chinese character. The 3D distillation technology is specifically a strategy based on a score distillation sampling method SDS, which includes: The distillation process includes a frozen teacher model ∈ θ (z t ,t,y) and a one-step generated student model, the weight of the student model is initialized to ∈ φ (z t ,t,y) = ∈ θ (z t ,T,y); the noise intensity t in the student model is fixed to T; During the training process, the student model generates a sample according to the implicit expression z T and the glyph content and style encoding feature y Optimizing the parameter φ, so that The same as the sample from the teacher model; optimize the objective function L SDS is expressed as: where, is the forward process edge distribution for the diffusion model; p θ (z t is the edge distribution defined by the teacher model implicitly at noise intensity t; w is a constant; a t and σ t are predefined non-zero variables; D KL is the KL divergence; Gradient of the optimization objective is represented as: where the noise estimate ∈ θ Classifier-free noise estimation; only the weights of the diffusion model are updated during optimization, i.e., the weights of the perceptual encoder are fixed during training.

7. A system for fast font generation in Chinese characters with few samples based on a diffusion model, for implementing the method of claim 1, characterized in that, It comprises a low-resolution generation model module, a super-resolution model module, and a diffusion model distillation module; wherein, The low-resolution generation model module is used to generate low-resolution glyph picture results; the super-resolution model module is used to continuously up-sample the low-resolution font results to a higher resolution, thereby generating high-resolution vector glyph images; The diffusion model distillation module is used to perform diffusion model distillation on the low-resolution generation model and the super-resolution model to obtain a one-step generation model.

Images