Method and system for quickly generating few-sample Chinese fonts based on diffusion model

Abstract

The invention discloses a few-sample Chinese font rapid generation method and system based on a diffusion model, and the system comprises a low-resolution generation model module and a super-resolution model module, and can also comprise a diffusion model distillation module. Constructing a conditional implicit diffusion model with a component perception encoder, namely a low-resolution generation model for generating high-quality low-resolution fonts; constructing a super-resolution model based on a cascade diffusion model, wherein the super-resolution model is used for expanding a low-resolution generation result to a high-resolution generation result; the diffusion model distillation module can further distill the trained low-resolution generation model and the super-resolution model into a one-step generation model so as to improve the font image generation efficiency. According to the method, the font style can be extracted and migrated from a limited input font image sample, a font image generation result of which the style matching degree and the font structure accuracy are remarkably improved is obtained, and high-quality vector fonts can be obtained through a vectorization tool.

Description

Technical Field

[0001] The present invention belongs to the field of computer graphics processing technology and artificial intelligence technology, and relates to Chinese font generation technology, and in particular to a method and system for quickly generating Chinese fonts with high resolution using a small number of samples based on a diffusion model, which can generate a complete Chinese character library of a new font given a limited number of input glyph samples. Background Art

[0002] Font generation refers to the task of generating a complete font library given a limited number of input glyph samples of a new font. Automated font generation reduces the time-consuming and labor-intensive labor burden, so it has broad application prospects and has attracted widespread attention in academia; especially for writing systems such as Chinese, Japanese or Korean that contain tens of thousands of characters, fast and effective few-sample font generation methods can not only speed up the design of high-quality commercial font products, but can also be applied to personalized handwriting generation, scene text image editing, optical character recognition (OCR) data enhancement and other application scenarios.

[0003] Existing font generation methods are generally divided into two categories: directly generating vector glyphs, or first generating high-quality rasterized images and then vectorizing them into vector glyphs. This study focuses on the second type of method, which is to develop a font generation system that takes a few sample glyph images of the target style as input, outputs high-quality, high-resolution glyph images of the remaining characters in the font, and obtains high-quality vector glyphs through existing vectorization tools. Compared with the direct generation of vector glyphs, although processing high-resolution images involves higher computational complexity and still requires manual intervention in the vectorization process, image modalities with relatively more structured information are easier to process, and a large number of mature and advanced generation models are available for use and reference; on the other hand, the direct generation of vector glyphs can currently only process simple shapes, which limits their application in font generation of writing systems containing glyphs with high topological complexity (such as Chinese). If the image modality generation results of sufficiently high quality can be obtained, high-quality and high-complexity vector glyph results can be obtained through mature vectorization tools.

[0004] Although font generation technology has made some progress in recent years, few-shot font generation remains an unsolved and extremely challenging task due to the complexity of character structure and the creativity and diversity of font design. Most previous methods are based on generative adversarial networks (GANs), which train image-to-image translation networks in an adversarial manner. However, it is well known that generative adversarial networks are difficult to train and scale, suffer from mode collapse problems, and lack diversity, which limits their potential for generation quality. On the other hand, in the image modality, diffusion models have been shown to produce better results than state-of-the-art generative adversarial models and can be easily extended to larger datasets. However, the iterative reasoning process of the diffusion model makes it more time-consuming than the single-step reasoning generative adversarial model, which hinders its application in practical scenarios; in addition, based on the diffusion model, it is also necessary to design how to properly integrate the prior knowledge of glyphs into the model architecture.

[0005] Diffusion-based generative models have recently achieved unparalleled performance in many image synthesis tasks and generation tasks of other modalities such as video, audio, and text. The latent diffusion model variant that greatly reduces its computational cost has been widely used in various conditional generation tasks, including image editing, text-to-image generation, etc. The cascaded diffusion model variant that extends it to high-resolution generation upsamples the generated images to higher resolutions one by one, and has been successfully applied to a variety of tasks that require high-resolution generation. Diff-Font (He, Haibin, et al. "Diff-Font: Diffusion Model for Robust One-Shot Font Generation." arXiv preprintarXiv:2212.05895.2022.) and FontDiffuser (Yang, Zhenhua, et al. "FontDiffuser: One-Shot Font Generation via Denoising Diffusion with Multi-Scale Content Aggregation and Style Contrastive Learning." arXiv preprint arXiv:2312.12142.2023.) have attempted to apply diffusion models to the task of font generation and achieved impressive results. Although the above methods are able to generate clear results using the generation ability of the diffusion model, the pre-trained style features or style learning strategies they use often fail when facing unknown complex font styles, and they build models based on pixel space and use multi-step sampling, which limits the practical application of the methods and their scalability to higher resolutions. Summary of the invention

[0006] In order to overcome the deficiencies of the above-mentioned prior art, the present invention proposes a font generation method and system based on a diffusion model, which supports one-step font generation and high-resolution generation, and realizes high-quality, high-speed, and high-resolution font generation with a small number of samples.

[0007] First, the present invention adds prior knowledge about component decomposition, which is very useful for learning and transferring font styles. The present invention uses the attention mechanism to realize the reuse of component-level style information, and with the help of the powerful generation ability of the diffusion model, learns how to effectively transfer diverse and complex font styles to the target glyphs; in addition, in order to adapt to the smaller input sample size (such as less than 10) in practical applications, the present invention introduces a reference sample selection strategy so that the model has the ability to extract global and local style information at the same time. Secondly, in order to further achieve higher resolution generation, the present invention designs a super-resolution process so that the loss of detailed style and shape at low resolution can be restored in the super-resolution process, thereby narrowing the gap between image modality and vector modality, thereby achieving high-quality, high-speed, and high-resolution few-sample font generation. Finally, in order to solve the problem of slow reasoning speed of the diffusion model, the present invention applies the fractional distillation sampling technology in the text-to-3D generation technology to achieve efficient one-step generation with almost no quality loss.

[0008] For the convenience of explanation, the present invention agrees on the following definitions of terms:

[0009] Token: The smallest language unit processed by the attention mechanism.

[0010] Auto-encoder: Autoencoder.

[0011] U-Net: A convolution-based deep learning model architecture proposed in the literature (Ronneberger, Olaf, et al. "U-net: Convolutional networks for biomedical image segmentation." Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. Springer International Publishing, 2015.).

[0012] Score Distillation Sampling (SDS): The score distillation sampling method proposed in the literature (Poole, Ben, et al. "Dreamfusion: Text-to-3d using 2d diffusion." arXiv preprint arXiv:2209.14988(2022).).

[0013] The technical solution provided by the present invention is as follows:

[0014] A method and system for fast generation of Chinese fonts with few samples and high resolution based on diffusion model. The system proposed by the present invention includes three parts: the first part includes a conditional implicit diffusion model with component-aware encoder, which is used to generate high-quality low-resolution results (resolution 64×64); the second part includes a conditional super-resolution cascade diffusion model based on cascade diffusion model super-resolution technology, which is used to perform super-resolution and generate high-resolution glyph fonts; the third part includes a diffusion model distillation technology, which is used to distill the trained conditional implicit diffusion model with component-aware encoder and the conditional super-resolution cascade diffusion model into a one-step generation model.

[0015] In the first part, a conditional implicit diffusion model with a component-aware encoder is constructed based on the conditional diffusion model and the component-aware encoder to achieve high-quality low-resolution font generation.

[0016] According to the core idea of ​​the diffusion model, the present invention gradually adds noise to the real data until the data is converted into pure noise, and then estimates the distribution of the real data by learning a network that gradually converts the pure noise back to the real data. Specifically, given a character content reference image x rendered in a standard font (such as boldface), c , and k style reference images from the target font The goal is to generate a glyph image with a target font style that matches the target character content. The present invention uses a component-aware encoder C to encode the information of the input content and style reference image, and finally obtains the glyph content and style encoding feature y=C(x c ,X s ), and using the principle of implicit diffusion model, use the conditional implicit diffusion model ∈ θ (i.e. the constructed low-resolution generative model) estimates the distribution of the target generated image, and finally achieves the purpose of generating the target image according to the style features (as conditional input). Let x0 be the glyph target image, z0 be the implicit expression corresponding to the latent space defined by the auto-encoder Auto-encoder in the conditional implicit diffusion model, and the corresponding diffusion model training loss is as follows:

[0017]

[0018] Among them, ∈ is Gaussian noise, y is the font content and style encoding feature, z t is the result of adding noise z0 to the noise intensity t, that is, z t =α t z0+σ t ∈, α t and σ tis a series of non-zero variables, t is the noise intensity, ranging from 0 to T, T=1000. is the expectation; |||| is the L2 distance.

[0019] In addition, in the scenario of multi-step diffusion model sampling, classifier-free guidance, a method of guiding samples to move to high-density areas of distribution determined by conditions, is often used to use diversity in exchange for higher quality. The present invention defines a classifier-free guidance method for two types of conditional inputs, content and style, and the corresponding noise is estimated as follows, where the two coefficients s c and s Respectively control the association strength between the generated results and the input content and style conditions; expressed as:

[0020]

[0021] in To x c The result of replacing it with an all-zero vector of the same size, To X s The result of replacing it with an all-zero vector of the same size, Noise estimation results without classifier guidance.

[0022] Since the style of the font exists at different structural levels of the glyph image, different characters share different common style information. Ideally, the model is expected to focus on different relevant parts of the style reference at different structural levels of the input content. The present invention adopts a multi-head cross-attention mechanism to achieve this intention. Specifically, the component-aware encoder consists of a style encoder E s , a content encoder E c and a cross attention module. The style encoder encodes all k style reference images into style features i=1,…,k, the content encoder encodes the content reference image into content features Each style feature is represented as a dimension d s h s × s tokens, representing the content features as a dimension of d c h c × c tokens; and perform a cross-attention operation, expressed as:

[0023]

[0024] in is the projection matrix corresponding to the i-th attention mechanism; y is the glyph content and style encoding features; Attention is the attention mechanism, softmax is a function that maps any real vector to a probability distribution, Q is the query matrix, K is the key matrix, and V is the value matrix; To f c The flattened features are For the general The features are concatenated and flattened. In this way, through the attention mechanism, each fine-grained position on the content feature map learns to pay attention to the relevant position on the style feature map, and the relevant style features are combined to form the final conditional input.

[0025] In order to guide this learning process, we provide the style encoder with a pre-designed and selected style reference corresponding to the components and strokes contained in the content characters. The present invention defines a character set containing 775 commonly used components and strokes contained in Chinese characters in all Chinese character encoding standards (GB2312 standard) as a style reference collection, as a model training set. For any character in the training set, the corresponding style reference is selected as the character in the style reference set that contains the corresponding components or strokes of the character, so as to guide the model to learn the style extraction and migration at the component and stroke level in a targeted manner.

[0026] In the second part, a conditional super-resolution cascade diffusion model is constructed based on the cascade diffusion model super-resolution technology to achieve high-resolution font generation.

[0027] The motivation for shifting the generation process to higher resolution areas is that, first, at low resolution, a lot of information will be lost, such as style details or dense strokes, so an upsampling process is needed to restore this lost information. Second, upsampling can narrow the gap between rasterized images and vector glyphs, achieving higher quality vector generation results.

[0028] The present invention uses a conditional super-resolution cascade diffusion model to continuously upsample the high-quality low-resolution font results obtained in the first part to a higher resolution, that is, from 64 to 256, and then from 256 to 1024, so as to generate a high-resolution font image. The present invention does not directly apply other existing image super-resolution models, but uses the same model as the first part, that is, the implicit diffusion model and the component perception encoder, and makes three improvements to the super-resolution model: 1) The present invention replaces the content reference image with the low-resolution image and compares it with the input noise z t1) The invention connects the low-resolution image to the model in a connected manner; in this way, similar to the effect of the cross-attention module in the conditional input of the style feature on the low-resolution model, the invention guides the attention at each position of the low-resolution image to the relevant position in the style reference to restore the lost details; 2) In order to perform the noise conditional enhancement operation that is crucial to bridging the gap between the generated image and the training image, the invention encodes the low-resolution image into the latent space, adds noise to it with Gaussian noise, and then decodes it back to the image space as the enhanced low-resolution image input; in addition, the invention also adds an additional condition s to the U-Net part of the conditional implicit diffusion model with the component-aware encoder in the first part, that is, the noise level added during the noise conditional enhancement process; 3) In order to cope with the problem of increased memory usage caused by high resolution, the invention adds an additional downsampling layer to the style and content encoder in the component-aware encoder. Since the style and content encoders have mastered the knowledge of extracting and migrating the required font style, the invention uses the weights of the low-resolution generation model to initialize the super-resolution model.

[0029] The third part realizes single-step sampling high-speed font generation based on 3D distillation technology.

[0030] Image diffusion models usually require dozens of sampling steps to produce effective, high-quality output results. For the model obtained in the first part, sampling 10 steps with a 1st-order SDE sampler can get a good enough result, but 10 steps of reasoning still cannot achieve fast, real-time generation. The present invention proposes to use a strategy based on the score distillation sampling method Score Distillation Sampling (SDS) to distill the conditional implicit diffusion model with component perception encoder and the conditional super-resolution cascade diffusion model into a one-step generation model.

[0031] Score Distillation Sampling (SDS) was originally proposed for text-to-3D generation tasks, which uses the knowledge of a pre-trained 2D text-to-image diffusion model to guide the 3D model to render realistic 2D images from random angles, thereby achieving 3D model learning. This distillation process can also be used to reduce the sampling steps of the model, that is, replacing the 3D model with a single-step sampling diffusion model, and generating more realistic images under the guidance of a multi-step diffusion model.

[0032] The distillation process consists of a frozen teacher model ∈ θ (z t ,t,y) and a student model generated in one step. The weight of the student model is initialized by the teacher model as ∈ φ (z t ,t,y)=∈ θ (z t,T,y). The noise intensity t in the student model is fixed to T to ensure that the model starts with pure noise during inference. During training, the student model is trained based on the implicit expression z to be denoised. T =α T z0+σ T ∈ and the glyph content and style encoding features y to generate samples Similar to the way the SDS method processes 2D images rendered from 3D models, the goal is to optimize the parameter φ so that It looks close to the samples from the teacher model. To do this, it is necessary to move the samples to the high probability density area implicitly defined by the teacher model for each noise intensity t, which is equivalent to optimizing the following objective:

[0033]

[0034] Among them, L SDS To optimize the goal; is the expectation; t and σ t is a series of predefined non-zero variables; D KL is the KL divergence; is the forward process of the diffusion model (i.e. ) marginal distribution, p θ (z t |y,t) is the marginal distribution implicitly defined by the teacher model at noise intensity t, and w is a constant. Since estimating the noise added to the data sample is equivalent to estimating the score function of the noisy data distribution, the approximate gradient of the above optimization objective can be expressed as:

[0035]

[0036] in, is the gradient operator; L SDS The optimization goal of SDS; is the expectation; ∈ is Gaussian noise.

[0037] The noise estimate (∈ θ (z t ,t,y)) can be estimated by noise without classifier guidance Only the weights of the diffusion model are updated during the optimization process, i.e., the weights of the component perceptual encoder are fixed during the training process.

[0038] Based on the above-mentioned diffusion model-based small sample high-resolution Chinese font rapid generation system of the present invention, the small sample high-resolution Chinese font rapid generation method comprises the following steps:

[0039] The first step is to build a style reference selection mapping;

[0040] Constructing a style reference selection mapping. In specific implementation, based on the 1032 types of components and 339 types of strokes defined for Chinese characters, a character set that can cover all components and strokes contained in Chinese characters in the GB2312 standard is constructed as a style reference set, and the style reference for model training is selected from it;

[0041] For any character c in the training set, a component-level character reference mapping and a stroke-level character reference mapping are established. The former consists of all characters in the style reference set that contain the components contained in c, and the latter consists of all characters in the style reference set that contain the strokes contained in c.

[0042] The second step is to select the style reference corresponding to each target image during training according to the style reference selection mapping, and train the implicit diffusion model to generate low-resolution glyph image results;

[0043] This step first builds a conditional implicit diffusion model with a component-aware encoder, which consists of a style encoder E s , a content encoder E c and a cross attention module. The style encoder encodes all k style reference images into style features i=1,…,k, the content encoder encodes the content reference image into content features Each style feature is represented as a dimension d s h s × s tokens, representing the content features as a dimension of d c h c × c tokens and perform cross-attention operations.

[0044] The constructed conditional implicit diffusion model with component-aware encoder is sampled using classifier-free guidance, and a classifier-free guidance method for multiple style feature conditional inputs is defined. The implicit diffusion model is designed and trained using a loss function to obtain a trained conditional implicit diffusion model with component-aware encoder. High-quality low-resolution results are generated through the trained conditional implicit diffusion model with component-aware encoder.

[0045] The third step is to train the conditional super-resolution cascade diffusion model using the parameters of the trained low-resolution generative model (i.e., the conditional implicit diffusion model with component-aware encoder) as the initial parameters;

[0046] Using the conditional super-resolution cascade diffusion model, the low-resolution results are continuously upsampled to higher resolutions, i.e., from 64 to 256 and then from 256 to 1024, thereby generating high-resolution glyph images.

[0047] The model in the first step, that is, the implicit diffusion model and the component-aware encoder, is used as the initialization parameters, and three improvements are made to the conditional implicit diffusion model with the component-aware encoder for the super-resolution task:

[0048] 1) The corresponding low-resolution image is obtained by replacing the content reference image with the low-resolution image and comparing it with the input noise z t The connected method is inserted into the U-Net part of the conditional implicit diffusion model with component-aware encoder;

[0049] 2) Encode the low-resolution image into the latent space, add Gaussian noise to the implicit expression corresponding to the low-resolution image in the latent space, and then decode it back to the image space as the enhanced low-resolution image input;

[0050] In addition, an additional condition s is added to the U-Net part of the conditional implicit diffusion model with component-aware encoder, which is the noise level added during the noise condition enhancement process;

[0051] 3) Additional downsampling layers are added to the style and content encoders in the part-aware encoder to cope with the increased memory usage caused by high resolution.

[0052] In the fourth step, the conditional implicit diffusion model with component-aware encoder and the conditional super-resolution cascade diffusion model are shortened to a single-step sampling using 3D distillation.

[0053] In the specific implementation, diffusion model distillation is performed based on the Score Distillation Sampling (SDS) strategy.

[0054] Let x0 be the glyph target image, z0 be the implicit expression corresponding to the latent space defined by the autoencoder in the conditional implicit diffusion model, and α t and σ t is a series of predefined non-zero variables. Taking the distillation of the conditional implicit diffusion model with component-aware encoder as an example (the distillation conditional super-resolution cascade diffusion model process can be similarly derived), the distillation process includes a frozen teacher model ∈ θ (z t ,t,y) (i.e., a conditional implicit diffusion model with a component-aware encoder) and a one-step generative student model, where z t =α t z0+σ t ∈ is the implicit expression to be denoised, t is the noise intensity, and y is the glyph content and style encoding feature. The weight of the student model is initialized by the teacher model as ∈ φ (z t ,t,y)=∈θ (z t ,T,y). The noise intensity t in the student model is fixed to the maximum value T supported by the teacher model.

[0055] In the process of training the student model, the student model is based on the implicit expression z to be denoised T =α T z0+σ T ∈ and the glyph content and style encoding features y to generate samples In order to optimize the model parameter φ, the sample Looks close to samples from the teacher model, optimizing the following objective:

[0056]

[0057] Among them, L SDS To optimize the goal; is the expectation; t and σ t is a series of predefined non-zero variables; D KL is the KL divergence; is the forward process of the diffusion model (i.e. ) marginal distribution, p θ (z t |y,t) is the marginal distribution implicitly defined by the teacher model at noise intensity t, and w is a constant. The gradient of the optimization objective is approximated as:

[0058]

[0059] in, is the gradient operator; L SDS The optimization goal of SDS; is the expectation; ∈ is Gaussian noise.

[0060] The noise estimation can be performed with a version without classifier guidance. Only the weights of the diffusion model are updated during the optimization process, i.e., the weights of the component perceptual encoder are fixed during the training process.

[0061] By inputting a limited number of glyph samples of a new font and following the above steps, rapid generation of a small number of high-resolution Chinese fonts based on a diffusion model can be achieved, thereby generating a complete Chinese character library of the font.

[0062] Compared with the prior art, the beneficial technical effects of the present invention are:

[0063] The present invention provides a method and system for rapidly generating Chinese fonts with a small number of samples and high resolution based on a diffusion model, which utilizes an attention mechanism to reuse component-level style information, and effectively transfers diverse and complex font styles to target glyphs based on diffusion model learning; in addition, the present invention introduces a reference sample selection strategy so that the model has the ability to extract both global and local style information. Secondly, the present invention applies the fractional distillation sampling technology in the text-to-3D generation literature to achieve efficient one-step generation with almost no quality loss, solve the problem of slow reasoning speed of the diffusion model, and improve the speed and efficiency of font generation. Finally, the present invention designs a super-resolution process so that the loss of detailed style and shape at low resolution can be restored in the super-resolution process, thereby narrowing the gap between the image modality and the vector modality, and further achieving higher resolution font generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a flowchart of the font generation method provided by the present invention.

[0065] Figure 2 It is a schematic diagram of the reference selection process of the present invention.

[0066] Figure 3 This is a comparison of the generation results of the present invention and the existing methods on the unseen fonts and seen characters. n_ref is the input style reference sample size.

[0067] Figure 4 This is a comparison of the generation results of the present invention and the existing methods on unseen fonts and unseen characters. n_ref is the input style reference sample size.

[0068] Figure 5 It is the detail style visualization result restored by the super-resolution model in the present invention.

[0069] Figure 6 It is the vector font result obtained by applying a vectorization tool to the high-resolution font image result in the present invention.

[0070] Figure 7 It is a comparison of the generation results of the single-step model of the present invention (after distillation) and its teacher model (before distillation) on unseen fonts and seen characters.

[0071] Figure 8 This is a comparison of the generation efficiency of the present invention and the existing methods.

[0072] Fig. 9 This is a comparison between the present invention and the existing methods when applied to font color and texture style transfer.

[0073] Fig.10 It is the generation result of the present invention applied to cross-language generation. DETAILED DESCRIPTION

[0074] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings, but the scope of the present invention is not limited in any way.

[0075] The present invention provides a technology and system for fast generation of Chinese fonts with high resolution from a small number of samples based on a diffusion model. The technology and system consist of three parts. The process and network structure are shown in the attached figure. Figure 1 As shown in the figure, the first part achieves high-quality generation based on the conditional diffusion model and component-aware encoder (see Appendix Figure 1 The second part is based on the cascade diffusion model super-resolution technology to achieve high-resolution font generation (Appendix Figure 1 The third part is based on 3D distillation technology to achieve single-step sampling high-speed font generation (Appendix Figure 1 Middle stage B2).

[0076] The specific implementation includes the following steps:

[0077] 1) Build style reference selection mapping, as shown in the attached Figure 2 shown.

[0078] In order to guide the style learning process of different levels of glyphs, the present invention provides the style encoder with pre-designed and selected style references corresponding to the components and strokes contained in the content characters. In EasyFont (Lian, Zhouhui, et al. "EasyFont: a style learning-based system to easily build your large-scale handwriting fonts." ACM Transactions on Graphics (TOG) 38.1 (2018)), m = 1032 types of components and n = 339 types of fine-grained strokes are defined for Chinese characters, and several character sets of different sizes covering different proportions of Chinese characters are constructed. We use the OptSet set defined in the document, which contains 775 commonly used Chinese characters and can cover all components and strokes contained in the Chinese characters in the GB2312 standard; let this set be the style reference set, and the style reference for model training is selected from it. For any character c in the training set, let c contain m c Componentscomp c,i ∈{1,…,m},i=1,…,m c , let the i-th component contain n c,i stroke c,i,j ∈{1,…,n},j=1,…,n c,iSince Chinese characters are usually decomposed into at most 6 components, the number of style references k is set to 6. We establish a component-level character reference mapping and a stroke-level character reference mapping, which maps each component comp in character c. c,i Assign a subreference set For component-level character reference mapping, All the included components comp in the style reference collection c,i The character composition of the stroke-level character reference mapping, The style reference collection contains any {stroke c,i,j ,j=1,…,n c,i}. In the training process, the present invention A character is randomly selected from the style reference set of character c, and then the remaining k positions are randomly filled. The present invention uses these two mappings with probabilities 1-p and p. During the training process, OptSet is used as the style reference set; during the testing process, a new mapping is constructed based on the given reference set.

[0079] 2) According to the style reference selection mapping, the style reference corresponding to each target image during training is selected, and the implicit diffusion model is trained to generate low-resolution glyph image results, and p is set to 0.1.

[0080] 3) Using the trained conditional implicit diffusion model parameters with component-aware encoder as initial parameters, train the super-resolution model.

[0081] 4) Use the 3D distillation technique introduced above to shorten the conditional implicit diffusion model with component-aware encoder and the conditional super-resolution cascade diffusion model to single-step sampling, and fix the guidance coefficient s of the classifier guidance sampling c =2.0 and s s =2.0.

[0082] 5) Use the Image Tracer tool in Adobe Illustrator as a vectorization tool to convert high-resolution glyph images into high-quality vector glyphs.

[0083] During testing, that is, when the user uses it, the style reference image of the target font and the target generated characters are input. The present invention sends the style reference and the target characters rendered in the standard font (such as bold) into the system, and obtains the vector glyph results through the low-resolution generation model, super-resolution model and vectorization tool.

[0084] Attached Figure 3This is a comparison of the generation results of the present invention and the existing methods on unseen fonts and seen characters. n_ref is the input style reference sample size. The samples generated by the present invention have fine and sharp details and are consistent with the reference images of various styles. Using different input sample sizes, the model can extract different levels of style information from the input reference sample set. Figure 4 This is a comparison of the results of the present invention and the existing methods on unseen fonts and unseen characters. n_ref is the input style reference sample size. The present invention can also generate high-quality results on characters that have not been seen in training. Figure 5 is the detailed style visualization result restored by the super-resolution model in the present invention. The model in the present invention can extract and restore style details from high-resolution style references. Figure 6 This is the vector font result obtained by applying a vectorization tool (Adobe Illustrator) to the high-resolution font image result in the present invention. Since the quality of the high-resolution font image generation result is high enough, the present invention can obtain a high-quality vector font result with precise control points. Figure 7 The single-step student model of the present invention and its teacher model generate results on unseen fonts and seen characters. The single-step student model obtained by the present invention faithfully imitates the output of the multi-step teacher model, while the teacher model cannot generate effective results in the single-step generation setting, which proves the effectiveness of the distillation method in the present invention. Figure 8 The single-step model obtained by the present invention is significantly faster than other diffusion-based models and is comparable to the methods based on generative adversarial networks. Fig. 9 The present invention is compared with the existing methods when applied to font color and texture style transfer. The present invention performs better in content matching and style consistency of all tested styles. Fig.10 It is the generation result of the present invention applied to cross-language generation, that is, using the model trained on Chinese characters to transfer the style of Chinese characters to English and Korean characters.

[0085] The above is a clear and complete description of the technical solutions in the embodiments of the present invention in combination with the drawings in the embodiments of the present invention. It is understandable that the examples described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

Claims

1. A method for fast generation of Chinese fonts with few samples based on diffusion model, characterized in that: By constructing and training a conditional implicit diffusion model with a component-aware encoder, i.e., a low-resolution generation model, a high-quality low-resolution font result is generated; based on the cascade diffusion model super-resolution technology, a conditional super-resolution cascade diffusion model, i.e., a super-resolution model, is constructed to super-resolve the obtained high-quality low-resolution font result, thereby generating a high-resolution vector glyph font; the steps include: The first step is to build a style reference selection mapping; The second step is to select the style reference corresponding to each target image during training according to the style reference selection mapping, build a conditional implicit diffusion model with a component-aware encoder, i.e., a low-resolution generative model, and train it to generate low-resolution glyph image results; The component-aware encoder in the low-resolution generative model includes a style encoder, a content encoder, and a cross-attention module; The style encoder encodes all style reference images into style features respectively, and the content encoder encodes the content reference images into content features; The cross-attention module is used to represent each style feature and content feature as multiple tokens and then perform cross-attention operations; The constructed low-resolution generative model is sampled using a classifier-free guide, and a classifier-free guide method for inputting multiple style feature conditions is adopted; the low-resolution generative model is designed and trained using a loss function to obtain a trained low-resolution generative model; high-quality low-resolution font results are generated through the trained low-resolution generative model; The third step is to use the parameters of the trained low-resolution generative model as initial parameters to train a conditional super-resolution cascade diffusion model, i.e., a super-resolution model. The trained super-resolution model is used to continuously upsample the low-resolution font results to a higher resolution, thereby generating a high-resolution glyph image. Through the above steps, the rapid generation of Chinese fonts with a small number of samples based on the diffusion model can be achieved.

2. The method for fast generation of Chinese fonts with a small number of samples based on a diffusion model as claimed in claim 1, characterized in that: In the first step, based on the components and strokes defined for Chinese characters, a character set covering the components and strokes included in Chinese characters in the Chinese encoding standard is constructed as a style reference set, which is used as a model training set; For any character in the model training set, a component-level character reference mapping and a stroke-level character reference mapping are established; wherein the component-level character reference mapping is composed of all characters in the style reference set that contain the components contained in the character, and the stroke-level character reference mapping is composed of all characters in the style reference set that contain the strokes contained in the character.

3. The method for fast generation of Chinese fonts with a small number of samples based on a diffusion model as claimed in claim 1, characterized in that: In the low-resolution generative model, the style encoder in the component-aware encoder is used to encode all k style reference images into style features, which are expressed as: The content encoder in the component-aware encoder is used to encode the content reference image into content features, denoted as f c ; Represent each style feature as multiple tokens with dimensions, represent the content feature as multiple tokens with dimensions, and perform cross-attention operations, expressed as: Among them, y is the glyph content and style encoding features; Attention is the attention mechanism; Q is the query matrix, K is the key matrix, and V is the value matrix; are the projection matrices corresponding to the i-th attention mechanism; softmax is the probability distribution mapping function; To flatten the content features, The features are obtained by concatenating and flattening the style features.

4. The method for fast generation of Chinese fonts with a small number of samples based on a diffusion model as claimed in claim 3, characterized in that: In the second step, the loss function used to train the low-resolution generative model is It is expressed as: Among them, x0 is the glyph target image; ∈ is Gaussian noise, t is the noise intensity, ∈ θ represents the conditional implicit diffusion model; z t is the implicit expression corresponding to adding noise to z0 to t; z0 is the implicit expression corresponding to the latent space defined by the autoencoder in the conditional implicit diffusion model that encodes the glyph target image; represents expectation; |||| represents L2 distance.

5. The method for fast generation of Chinese fonts with a small number of samples based on a diffusion model as claimed in claim 4, characterized in that: In the second step, the low-resolution generative model is sampled using a classifier-free guide, expressed as: in, is the noise estimation result without classifier guidance; s c and s are the correlation strength coefficients controlling the generated results, the input content and the style conditions respectively; x c is the character content reference image, X s is a stylistic reference image from the target font; To x c The result of replacing it with an all-zero vector of the same size, To X s The result of replacing it with an all-zero vector of the same size.

6. The method for fast generation of Chinese fonts with a small number of samples based on a diffusion model as claimed in claim 5, characterized in that: In the third step, the low-resolution generative model is used to train the super-resolution model, including the following improvements: 1) By using the corresponding low-resolution image as the content reference image and concatenating it with the input noise, it is inserted into the convolutional deep learning model U-Net in the low-resolution generative model; 2) Encode the low-resolution image into the latent space, use Gaussian noise to add noise to the implicit expression corresponding to the low-resolution image in the latent space, and then decode it back to the image space as the enhanced low-resolution image input; 3) Add additional downsampling layers to the style and content encoders in the part-aware encoder to cope with the increased memory usage caused by high resolution.

7. The method for fast generation of Chinese fonts with a small number of samples based on a diffusion model as claimed in claim 6, characterized in that: In improvement 2), additional conditions are added to the convolutional deep learning model U-Net in the low-resolution generation model, that is, the noise level is added in the noise condition enhancement process.

8. The method for fast generation of Chinese fonts with a small number of samples based on a diffusion model as claimed in claim 6, characterized in that: Furthermore, the 3D distillation technology is used to perform diffusion model distillation on the low-resolution generative model and the super-resolution model, shortening it to a single-step sampling to obtain a one-step generative model.

9. The method for fast generation of Chinese fonts with a small number of samples based on a diffusion model as claimed in claim 8, characterized in that: The 3D distillation technology is specifically a strategy based on the score distillation sampling method SDS, including: The distillation process consists of a frozen teacher model ∈ θ (z t ,t,y) and a student model generated in one step, the weight of the student model is initialized by the teacher model as ∈ φ (z t ,t,y)=∈ θ (z t ,T,y); the noise intensity t in the student model is fixed to T; During the training process, the student model is based on the implicit expression z to be denoised T Generate samples with glyph content and style encoding features y Optimize the parameter φ so that Same as the samples from the teacher model; optimize the objective function L SDS It is expressed as: in, is the marginal distribution of the forward process of the diffusion model; p θ (z t |y,t) is the marginal distribution implicitly defined by the teacher model at noise intensity t; w is a constant; α t and σ t is a predefined non-zero variable; D KL is the KL divergence; Optimize the gradient of the objective It is expressed as: The noise estimate ∈ θ Noise estimation without classifier guidance is adopted; only the weights of the diffusion model are updated during the optimization process, that is, the weights of the component perception encoder are fixed during the training process.

10. The method for fast generation of Chinese fonts with a small number of samples based on a diffusion model as claimed in claim 1, characterized in that: Based on the method described, a diffusion model-based fast generation system of Chinese fonts with a small number of samples is implemented, including a low-resolution generation model module, a super-resolution model module and a diffusion model distillation module; wherein, The low-resolution generation model module is used to generate low-resolution font image results; the super-resolution model module is used to continuously upsample the low-resolution font results to a higher resolution, thereby generating a high-resolution vector font image; The diffusion model distillation module is used to perform diffusion model distillation on the low-resolution generative model and the super-resolution model to obtain a one-step generative model.

Images