Font generation method and system based on prior guidance knowledge distillation and style perception

Abstract

The invention belongs to the field of computer vision, and discloses a font generation method and system based on prior guidance knowledge distillation and style perception, and the method comprises two training stages: a first stage, training a teacher model which takes a content character image, a style reference image and character prior information as condition input; in the second stage, through knowledge distillation, a student model which only takes content and style images as input is trained, so that the performance of the student model is close to that of a teacher model; the invention provides a multi-directional style perception Mamba module used for extracting comprehensive and robust style features, and a hierarchical multi-modal feature fusion module used for efficiently integrating various inputs of a teacher model. Through the method, the guiding effect of priori knowledge is internalized into a student model, so that the model can generate a high-quality font image with accurate content and consistent style even if the model lacks priori information during reasoning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a font generation method and system based on priori guided knowledge distillation and style perception. BACKGROUND

[0002] Font generation aims to automatically synthesize a character image with a new style according to the content of a source character image and the style of a reference character image. This technology has great application value in the fields of advertisement design, ancient book restoration, and personalized font customization, and can significantly reduce the labor cost of font design, especially for languages such as Chinese and Korean, which have a large character set and complex structure. An ideal font generation method needs to meet two core requirements: the content of the generated character should be strictly consistent with the source character, and the artistic style should be highly faithful to the reference character.

[0003] Early studies often model the font generation task as an image-to-image translation problem, such as the Zi2Zi model using a generative adversarial network (GAN) for conversion. However, this kind of method relies heavily on large-scale paired training samples (i.e., multiple style images of the same character), and cannot generalize to “unseen styles” that do not appear in the training set, which greatly limits its practical application. Subsequently, the paradigm of content and style decoupling became mainstream, such as the EMD and SA-VAE models, which extract content features from content images and style features from reference images, and then fuse them to generate. This decoupling idea enables the model to handle font generation of unseen styles, which is a major advance in the field.

[0004] To further improve the generation quality, existing technologies mainly develop along two paths: Network model-centered methods: strive to design superior network architectures to enhance the ability of feature extraction, decoupling, and fusion. For example, NTF-Font views font generation as a continuous change process within the neural transform domain; FontDiffuser improves the integrity of the content and the fidelity of the style by introducing multi-scale content features and style contrast learning. Although these methods have achieved certain results, they still have problems such as inconsistent local style features, stroke sticking or missing, etc. when facing complex strokes or unique style fonts.

[0005] Methods using character prior information: By introducing the inherent structure information of characters (such as strokes, component radicals), the model is assisted to generate. This kind of method can be further divided into two kinds: one is to use prior information as a supervision signal, for example, MX-Font uses multiple expert networks to learn different local features, and is assisted by a classifier for supervision; CG-GAN designs a component-level discriminator to achieve more fine-grained discrimination. However, due to the fuzzy boundary between the components of Chinese characters and other characters, and the nested structure, it is extremely difficult to accurately locate and identify the information of each component. The other is to use prior information as the input condition of the model, for example, Diff-Font fuses stroke information to guide generation, but its scope of application is limited to visible characters; IF-Font uses ideographic description sequence (IDS) as content input, although it introduces the spatial relationship between components and can generate invisible characters, but IDS cannot accurately describe the relative position, distance and size ratio between components, resulting in unbalanced component layout in the generation result. In addition, this kind of method has a fatal defect: its performance is highly dependent on the availability of prior information. In the open scene of cross-language generation or lack of prior information, its effect cannot be guaranteed.

[0006] At the same time, the style encoder of the existing model also has limitations. The style of the font is multi-dimensional, including global features such as font size and inclination, as well as local details such as stroke thickness, stroke shape, and whether to connect the strokes. The commonly used style encoder based on convolutional neural network (CNN) has insufficient ability to capture the global and long-range dependency style features of characters due to its inherent local receptive field. The encoder based on self-attention mechanism has a global receptive field, but its computational complexity is high, and it treats all features equally, which may ignore the local fine texture that is crucial to the style.

[0007] In summary, the existing font generation methods either have poor performance under complex styles due to insufficient model capabilities, or are limited in application scenarios due to excessive reliance on prior information, and the comprehensive and efficient extraction of style features still needs to be improved. SUMMARY

[0008] In view of the problems in the prior art that the generation effect is poor (such as inconsistent style, missing strokes) when facing complex styles, and strong dependence on character prior information and limited application, the present application proposes a font generation method based on prior guided knowledge distillation and style perception Mamba module. The present application aims to combine an efficient style encoder through a two-stage training framework, so that the model can not only use prior information to improve the performance upper limit, but also maintain high-quality generation ability when lacking prior information, thereby improving the generation quality and the generalization of the model.

[0009] In a first aspect, the present application provides a font generation method based on priori guidance knowledge distillation and style perception Mamba module, comprising the following steps: obtaining a content character image, a style reference image, and character priori information corresponding to the content character image; encoding the content character image, the style reference image, and the character priori information respectively to obtain content features, style features, and priori features; performing multi-modal fusion on the content features, the style features, and the priori features to generate conditional features; based on the conditional features, guiding the denoising process of a diffusion model to train a teacher model; using the teacher model as a guide model and initializing a student model; obtaining the content character image and the style reference image as inputs of the student model; using a knowledge distillation strategy to train the student model through a distillation loss function, wherein the distillation loss function is used to minimize the difference between the student model and the teacher model in noise prediction and intermediate feature representation.

[0010] As an optional implementation of the first aspect of the present application, in the step of encoding the content character image, the style reference image, and the character priori information respectively to obtain content features, style features, and priori features, the step of encoding the style reference image comprises: inputting the style reference image into a style encoder, wherein the style encoder comprises a multi-directional style perception Mamba module; the multi-directional style perception Mamba module divides the input features into a first branch and a second branch along the channel; the first branch sequentially performs linear layer expansion, depth separable convolution, activation function processing, and style perception group direction scanning operation on the features, and performs normalization; the second branch sequentially performs linear layer expansion and activation function processing on the features; the processing results of the first branch and the processing results of the second branch are multiplied element by element, and are projected back to the original dimension through a linear layer to obtain the style features.

[0011] As an optional implementation manner of the first aspect of the present application, in the step of performing, by the first branch, the linear layer expansion, the depth separable convolution, the activation function processing and the style-aware grouped direction scanning operation on the feature in sequence, the step of performing the style-aware grouped direction scanning operation specifically comprises: dividing the input feature of the first branch into eight feature groups uniformly along the channel dimension; assigning each of the feature groups with an independent scanning direction, wherein the eight scanning directions include horizontal scanning, vertical scanning, diagonal scanning and their respective flipped scanning; performing a selective scanning operation on each of the feature groups along the corresponding scanning direction thereof respectively to obtain eight directional scanning results; and splicing and merging the eight directional scanning results along the channel dimension to generate mixed style features that fuse multi-directional information.

[0012] As an optional implementation manner of the first aspect of the present application, the step of performing multi-modal fusion on the content feature, the style feature and the prior feature is implemented by a hierarchical multi-modal feature fusion module, specifically comprising: a first fusion stage: taking the content feature as a query, taking the style feature as a key and a value, and generating a preliminary fusion feature by a first multi-head cross attention mechanism; and a second fusion stage: taking the preliminary fusion feature as a query, taking the prior feature as a key and a value, and generating a final conditional feature by a second multi-head cross attention mechanism.

[0013] As an optional implementation manner of the first aspect of the present application, the distillation loss function comprises: a basic loss term for calculating the difference between the noise predicted by the student model and the real noise; a noise distillation loss term for calculating the difference between the noise predicted by the student model and the noise predicted by the teacher model; and a feature distillation loss term for calculating the difference between the intermediate feature maps extracted by the student model in the U-Net architecture of the diffusion model and the intermediate feature maps extracted by the teacher model at the corresponding layer.

[0014] As an optional implementation manner of the first aspect of the present application, the method further comprises a font generation inference stage, and the inference stage comprises two modes: mode one: when character prior information is provided, the teacher model is used for font generation, and the content character image, the style reference image and the character prior information are used as input conditions; and mode two: when no character prior information is provided, the student model is used for font generation, and only the content character image and the style reference image are used as input conditions; both mode one and mode two use a classifier-free guiding strategy for noise prediction, and a DPM-Solver++ sampling algorithm is used for accelerated sampling.

[0015] As an optional implementation of the first aspect of the application, the diffusion model is a latent diffusion model, and the denoising process of the guided diffusion model specifically includes: mapping the original image to a latent space through an encoder to obtain a latent representation; performing a forward noise adding process and a backward denoising process in the latent space; in the backward denoising process, inputting the conditional feature into a U-Net architecture to guide the model to iteratively denoise the noisy latent representation; and mapping the denoised latent representation back to an image space through a decoder to generate a target font image.

[0016] In a second aspect, the embodiments of the present application also provide a font generation system based on priori guided knowledge distillation and style perception, which comprises: The training teacher model module is configured to obtain a content character image, a style reference image, and character priori information corresponding to the content character image; encode the content character image, the style reference image, and the character priori information respectively to obtain a content feature, a style feature, and a priori feature; perform multi-modal fusion on the content feature, the style feature, and the priori feature to generate a conditional feature; and guide a denoising process of a diffusion model based on the conditional feature to train a teacher model. The training student model module is configured to initialize a student model by taking the teacher model as a guide model; obtain the content character image and the style reference image as inputs of the student model; and train the student model by using a knowledge distillation strategy and a distillation loss function, where the distillation loss function is used to minimize the difference between the student model and the teacher model in noise prediction and intermediate feature representation.

[0017] In a third aspect, the embodiments of the present application provide an electronic device, which comprises a processor, a memory, and a program or instructions stored on the memory and executable on the processor, and the program or instructions are executed by the processor to implement the steps of the method according to the first aspect.

[0018] In a fourth aspect, the embodiments of the present application provide a readable storage medium, which stores a program or instructions, and the program or instructions are executed by a processor to implement the steps of the method according to the first aspect.

[0019] Compared with the prior art, the present application has the following remarkable beneficial effects: 1. Improving the accuracy and generalization ability of style modeling: through the innovative two-stage knowledge distillation strategy, the present application integrates the advantages of character priori information into model training, and the final student model is free from the dependence on priori information during reasoning. This makes the model not only utilize priori knowledge to improve the understanding of complex character structure and style, but also be flexibly applied in cross-language and open scenarios without priori information.

[0020] 2. Richer and more robust style representation is achieved: The multi-directional style-aware Mamba module designed in the present application, especially its core eight-directional group-wise scanning mechanism, can comprehensively extract global (such as font size) and local (such as stroke connection) style features of character images from multiple spatial perspectives. Compared with traditional CNN and self-attention mechanism, this module has improved both comprehensiveness and efficiency of feature extraction.

[0021] 3. Efficient multi-modal information fusion and knowledge transfer is achieved: The hierarchical multi-modal feature fusion module proposed in the present application can orderly and efficiently integrate features of three modalities of content, style and priori information. At the same time, the loss function design in the knowledge distillation stage not only supervises the final output, but also supervises the intermediate features, providing more fine-grained learning signals for the student model, ensuring the effectiveness of knowledge transfer, so that it can still achieve high-quality generation level in the absence of priori input. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 Fig. 1 is a flowchart of a font generation method based on priori guided knowledge distillation and style perception according to an embodiment of the present application; Figure 2 Fig. 2 is a general framework diagram of a font generation model (KDMFont model) based on priori guided knowledge distillation and style perception according to an embodiment of the present application; Figure 3 Fig. 3 is a schematic diagram of a style-aware group-wise directional scanning (SAGDS) mechanism according to an embodiment of the present application; Figure 4 Fig. 4 is a qualitative comparison diagram of the present method and other five basic models on visible style, invisible character and invisible style invisible character datasets; Figure 5 Fig. 5 is an effect comparison diagram of the knowledge distillation strategy when the present method does not input IDS information in the cross-language font generation stage; Figure 6 Fig. 6 is a generation effect diagram of the network-based method, the priori-based method and the present method; Figure 7 Fig. 7 is a schematic diagram of different feature extraction methods; Figure 8 Fig. 8 is a structural schematic diagram of a font generation system based on priori guided knowledge distillation and style perception according to an embodiment of the present application.

[0023] The following specific embodiments will further illustrate the present application in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0026] Example 1 Please see Figure 1 The flowchart below illustrates a font generation method based on prior guided knowledge distillation and style awareness, as provided in an embodiment of the present invention.

[0027] The method proposed in this invention is based on a stable diffusion model. The stable diffusion model comprises two processes: forward noise addition and backward denoising. Forward noise addition primarily involves gradually adding noise to the image according to Markov chain rules until it becomes isotropic pure noise. Backward denoising primarily involves gradually recovering the original image from the noise. To further conserve computational resources, this invention employs a latent diffusion model (LDM). Given an image X, the encoder trained in the stable diffusion model is first used... Encode it to obtain Then add noise to it to get Based on the input condition c, predict the added noise. During the test, samples were randomly taken from a Gaussian distribution. Then, the trained model is used to predict the corresponding noise, and finally, the result is obtained. and using the decoder Decode it back into image X. This noise-adding process can be represented as: in X represents the original image. This indicates the added noise. , It is a predefined variance scheduler.

[0028] The denoising process can be briefly written as: where denotes the predicted noise, is a hyperparameter, . denotes the latent variable corresponding to time t, denotes the latent variable corresponding to time t-1.

[0029] Referring to Figure 2 , a general framework diagram of a font generation model (KDMFont model) based on prior guided knowledge distillation and style perception is proposed for an embodiment of the present application. The general framework includes three stages: (1) preliminary training stage: the model input conditions include three parts, content character image, style reference image and character prior information (IDS) corresponding to the content character image, style reference image; the multi-directional style perception Mamba module is used to construct a style encoder to extract multi-directional style features; the hierarchical multi-modal feature fusion module (HMMFI) fuses the extracted conditional features. (2) Knowledge distillation stage: the model of the previous stage is knowledge distilled to adapt to the condition without IDS. (3) Inference stage: the model inference process has two options, one is to use IDS and the other is not to use IDS.

[0030] Specifically, the preliminary training stage is mainly for training a teacher model, corresponding to steps S1 to S4.

[0031] S1: obtaining a content character image, a style reference image and character prior information corresponding to the content character image; S2: respectively encoding the content character image, the style reference image and the character prior information to obtain content features, style features and prior features; As shown in Figure 2 , the input conditions include a content character image, a style reference image and character prior information IDS corresponding to the content character image, which are respectively input into corresponding encoders (content encoder, style encoder and IDS encoder) to obtain corresponding features . Among them, the content encoder is composed of four convolutional blocks and two residual blocks, the IDS encoder is composed of an encoding layer and four Transformers, and the style encoder is composed of a multi-directional style perception Mamba module.

[0032] As shown in Figure 2 , the multi-directional style perception Mamba module divides the input features into a first branch and a second branch along the channel; In the first branch, the input features first pass through a linear layer ( The channels are expanded. Then, they are sequentially passed through depthwise separable convolutions (…). The process employs the SiLU activation function and the Style-Aware Grouped Directional Scanning (SAGDS) operation proposed in this invention to extract multi-directional mixed style features. Finally, it passes through a LayerNorm layer for normalization to obtain the processing result of the second branch. ; In the second branch, input features Similarly, it first goes through a linear layer ( The feature channels are expanded, and then passed through a SiLU activation function to obtain the processing result of the second branch. ; The result of the first branch is element-wise multiplied with the result of the second branch, and then processed through a linear layer ( Projecting back to the original dimension yields the style features; The above process can be represented as: in This represents the product of elements.

[0033] Furthermore, the stylistic features of a character are often quite complex; for example, font size, stroke thickness, and whether strokes are connected all affect the style of the font. Therefore, this invention proposes Style-Aware Grouped Oriented Scanning (SAGDS) in the first branch, such as... Figure 3 As shown, this approach aims to better extract local and global style. A character stroke includes not only horizontal and vertical parts but also some diagonal parts. To fully explore the stylistic features of a character, SAGDS's scanning method includes eight feature groups evenly divided along the channel dimension: horizontal scanning, vertical scanning, diagonal scanning, and a flipped scanning. This design not only effectively extracts the global style of the character but also enhances the modeling ability of local region information. Therefore, inspired by UltraLight VM-UNet, this invention introduces the above-mentioned grouped scanning strategy to save parameters. The input channel is divided into eight parts, each corresponding to a separate scanning direction. Specifically, assuming a given input feature... First, its dimensions are reshaped to (B, L, C), where L = H x W, and B represents the batch size, H represents the feature height, and W is the feature width. Then, it is divided into 8 parts along the channel. , , i∈(1,8), then Input selective scan layer to get the corresponding , and finally all are spliced and reshaped back to the original dimension (B, C, H, W) to get the final output result. The operation process of SAGDS can be represented as: where ChannelSplit represents splitting along the channel dimension, SelectiveScan represents selective scanning, Merge represents merging along the channel dimension, representing a hybrid style feature that fuses multi-directional information.

[0034] S3: Multi-modal fusion of the content feature, style feature and prior feature to generate a conditional feature; S4: Guiding the denoising process of the diffusion model based on the conditional feature to train a teacher model.

[0035] After obtaining each feature, we fuse the three features through a hierarchical multi-modal feature fusion module (HMMFI). Specifically, the hierarchical multi-modal feature fusion module is based on a two-layer multi-head cross-attention mechanism fusion stage.

[0036] In the first fusion stage, the content feature serves as the query Query and interacts with the style feature through the first multi-head cross-attention mechanism to extract style information related to the current content, thereby generating a preliminary fusion feature .

[0037] In the second stage, the preliminary fusion feature serves as the query Query to further query the IDS-based semantic feature , introducing character prior information, and through the second multi-head cross-attention mechanism, finally obtaining the fused conditional feature .

[0038] The finally fused conditional feature will be input as a condition into the U-Net to guide the denoising process of the model. This process can be represented as where respectively represent the content feature, IDS feature and style feature, and CrossAttn represents the cross-attention mechanism.

[0039] The loss function used in this process is where represents the real noise, Indicates prediction noise. This represents the basic loss term.

[0040] Specifically, the knowledge distillation stage mainly involves training the student model, corresponding to the following steps S5 to S7.

[0041] S5: Use the teacher model as the guidance model and initialize the student model; S6: Obtain the content character image and style reference image as input to the student model; S7: Employ a knowledge distillation strategy to train the student model using a distillation loss function, which minimizes the differences between the student model and the teacher model in noise prediction and intermediate feature representation.

[0042] The model obtained in the previous stage is used as the teacher model to guide the training of the student model in this stage. The architecture of the student model is identical to that of the teacher model, except that the hierarchical multimodal feature fusion module is replaced with a single cross-attention module. Furthermore, the student model only needs to input content and style images. In this stage, only the student model is trained, while the parameters of the teacher model remain frozen. The loss function used in this stage is: in and These are hyperparameters used to balance various losses. Represents real noise. This represents the noise in the teacher model's predictions. This represents the noise in the student model's predictions. These are the content image and the style image, respectively. Noise distillation loss term. The aim is to enhance the student model's ability to mimic the teacher model's behavior by minimizing the difference between the student and teacher models in noise prediction. Furthermore, feature distillation loss is introduced. This loss is calculated based on intermediate feature representations in the U-Net architecture. Specifically, and These represent the feature maps extracted from the i-th layer of the teacher model and the student model, respectively. Compared to supervised methods that rely solely on noise prediction, It provides the student model with finer-grained and more learnable intermediate feature signals, enabling it to achieve high-quality font generation even in the absence of character priors (such as IDS) input.

[0043] Specifically, the reasoning phase includes the following two modes: Mode one: when the character prior information is provided, the teacher model is used for font generation, and the content character image, style reference image and character prior information are used as input conditions; Mode two: when the character prior information is not provided, the student model is used for font generation, and only the content character image and style reference image are used as input conditions; Both mode one and mode two use a classifier-free guiding strategy for noise prediction, and use a DPM-Solver++ sampling algorithm for accelerated sampling.

[0044] Therefore, there are two options in this stage, one is to input IDS using the teacher model for generation, and the other is not to input IDS using the student model for generation.

[0045] (1) If IDS is input, the noise prediction process is: wherein , , is a predefined variance schedule, s represents a guiding scale, is a hyperparameter, . IDS represents the character corresponding to the IDS, represents the condition is empty.

[0046] (2) If IDS is not input, the noise prediction process is: wherein , , is a predefined variance schedule, s represents a guiding scale, is a hyperparameter, .

[0047] In this embodiment, a large number of empirical studies are also carried out, and the results show that the method has superior performance in the font generation task, which fully verifies its effectiveness.

[0048] For example Figure 4As shown, the qualitative comparison of the method on the visible style, invisible character and invisible style invisible character dataset and other five basic models. Among them, the MX-Font model disassembles the character into a preset component, and learns its feature representation by using a multi-expert mechanism, thereby improving the fineness of character generation at the local detail level. The CF-Font model introduces a content feature fusion module (CFM), which adaptively fuses content features between multiple base fonts, and combines iterative style vector optimization, effectively alleviating the difficulty of decoupling content and style, thereby significantly improving the overall quality and structural consistency of the generated characters. The NTF-Font model models the font generation as a continuous neural transformation process, and realizes smoother style migration with the expression ability of the transformation domain. The IF-Font model innovatively uses the description sequence of ideographic characters as content input, and completes character generation based on an autoregressive model, showing strong modeling ability. The FontDiffuser model combines multi-scale content features and style contrast learning under the diffusion model framework, effectively improving the performance of generated characters in local details and global style.

[0049] As shown in Figure 5 To verify the effectiveness of the knowledge distillation strategy in the present application, some cross-language font generation is performed without inputting IDS information in this cross-language font generation stage.

[0050] As shown in Figure 6 The deficiencies of the network-based method and the priori method are shown, and the generation effect of the present method is shown. The first column is the network-based method, the second column is the method using priori information, and the third column is the present method and the target character.

[0051] As shown in Figure 7 The comparison of different feature extraction methods is shown, (a) in Figure 7 (b) in Figure 7 (b) in Figure 7 (c) in the present method, the global receptive field of the multi-directional style perception Mamba module can be combined, and the local features can be captured well through scanning in 8 directions.

[0052] Embodiment 2 Please refer to Figure 8 The structure diagram of a font generation system based on priori guided knowledge distillation and style perception is shown in the second embodiment of the present application. The system includes the following key modules: The training teacher model module 100 is configured to obtain a content character image, a style reference image, and character prior information corresponding to the content character image; encode the content character image, the style reference image, and the character prior information respectively to obtain content features, style features, and prior features; perform multi-modal fusion on the content features, the style features, and the prior features to generate conditional features; and guide a denoising process of a diffusion model based on the conditional features to train a teacher model. The training student model module 200 is configured to use the teacher model as a guide model and initialize a student model; obtain a content character image and a style reference image as inputs of the student model; and train the student model by using a knowledge distillation strategy and a distillation loss function, where the distillation loss function is used to minimize differences between the student model and the teacher model in noise prediction and intermediate feature representation.

[0053] The font generation system based on prior guided knowledge distillation and style perception in the embodiments of the present application can be a device, a component, an integrated circuit, or a chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Illustratively, the mobile electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a wearable device, an Ultra-mobile Personal Computer (UMPC), a netbook, or a Personal Digital Assistant (PDA), and the non-mobile electronic device can be a server, a Network Attached Storage (NAS), a Personal Computer (PC), and the like, and the embodiments of the present application are not limited in this regard.

[0054] The font generation system based on prior guided knowledge distillation and style perception in the embodiments of the present application can be a device with an operating system. The operating system can be an Android operating system, an IOS operating system, or other possible operating systems, and the embodiments of the present application are not limited in this regard.

[0055] The font generation system based on prior guided knowledge distillation and style perception provided in the embodiments of the present application can implement the method embodiments. Figure 1 The font generation method based on prior guided knowledge distillation and style perception implemented in the method embodiments is not repeated here to avoid repetition.

[0056] Optionally, the embodiment of the present application further provides an electronic device, comprising a processor, a memory, a program or instructions stored in the memory and executable on the processor, which realizes each process of the above-mentioned embodiment of the font generation method based on prior guiding knowledge distillation and style perception and achieves the same technical effects. To avoid repetition, it will not be repeated here.

[0057] The embodiment of the present application further provides a readable storage medium, which stores a program or instructions, which realizes each process of the above-mentioned embodiment of the font generation method based on prior guiding knowledge distillation and style perception and achieves the same technical effects when executed by a processor. To avoid repetition, it will not be repeated here.

[0058] The processor is the processor in the electronic device in the above-mentioned embodiments. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0059] It should be noted that in this paper, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to the order of functions shown or discussed, but can also include the execution of functions in a substantially simultaneous manner or in the opposite order according to the functions involved, for example, the described method can be executed in an order different from the described order, and various steps can also be added, omitted or combined. In addition, the features described with reference to some examples can be combined in other examples.

[0060] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, also can be through hardware, but many cases the former is the better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the contribution to the prior art can be embodied in the form of software products, the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), including a number of instructions to make a terminal (may be a mobile phone, computer, server, air conditioner, or network equipment, etc.) executes the method described in various embodiments of the present application.

[0061] The embodiments of the present application are described above in conjunction with the drawings, but the present application is not limited to the above-mentioned specific embodiments, the above-mentioned specific embodiments are only illustrative, but not limited, those skilled in the art can make many forms without departing from the purpose of the present application and the scope of the claims under the inspiration of the present application, all belong to the protection of the present application.

Claims

1. A font generation method based on priori guidance knowledge distillation and style perception, characterized in that, The method comprises the following steps: obtaining a content character image, a style reference image and character prior information corresponding to the content character image; encoding the content character image, the style reference image and the character prior information respectively to obtain content features, style features and prior features; performing multi-modal fusion on the content features, the style features and the prior features to generate conditional features; training a teacher model based on the conditional features by guiding a denoising process of a diffusion model; using the teacher model as a guide model and initializing a student model; obtaining the content character image and the style reference image as inputs of the student model; training the student model by using a knowledge distillation strategy and a distillation loss function, wherein the distillation loss function is used to minimize the difference between the student model and the teacher model in noise prediction and intermediate feature representation.

2. The method of claim 1, wherein, In the step of encoding the content character image, the style reference image and the character prior information respectively to obtain content features, style features and prior features, the step of encoding the style reference image comprises the following steps: inputting the style reference image into a style encoder, wherein the style encoder comprises a multi-directional style perception Mamba module; the multi-directional style perception Mamba module divides input features into a first branch and a second branch along a channel dimension; the first branch sequentially performs linear layer expansion, depth separable convolution, activation function processing and style perception grouped direction scanning operation on the features, and performs normalization; the second branch sequentially performs linear layer expansion and activation function processing on the features; performing element multiplication on the processing results of the first branch and the processing results of the second branch, and projecting back to the original dimension through a linear layer to obtain the style features.

3. The method of claim 2, wherein, In the step of sequentially performing linear layer expansion, depth separable convolution, activation function processing and style perception grouped direction scanning operation on the features by the first branch, the step of style perception grouped direction scanning operation comprises the following steps: uniformly dividing the input features of the first branch into eight feature groups along the channel dimension; assigning each feature group an independent scanning direction, wherein the eight scanning directions include horizontal scanning, vertical scanning, diagonal scanning and their respective flipped scanning; performing selective scanning operation on each feature group along its corresponding scanning direction to obtain eight directional scanning results; splicing and merging the eight directional scanning results along the channel dimension to generate mixed style features with fused multi-directional information.

4. The method of claim 1, wherein, The step of performing multi-modal fusion on the content features, the style features and the prior features is implemented by a hierarchical multi-modal feature fusion module, and specifically comprises the following steps: a first fusion stage: taking the content features as queries, the style features as keys and values, and performing interaction through a first multi-head cross-attention mechanism to generate preliminary fusion features; a second fusion stage: taking the preliminary fusion features as queries, the prior features as keys and values, and performing interaction through a second multi-head cross-attention mechanism to generate final conditional features.

5. The method of claim 1, wherein, The distillation loss function comprises: a base loss term for calculating a difference between the noise predicted by the student model and the real noise; a noise distillation loss term for calculating a difference between the noise predicted by the student model and the noise predicted by the teacher model; a feature distillation loss term for calculating a difference between the intermediate feature maps extracted by the student model in the U-Net architecture of the diffusion model and the intermediate feature maps extracted by the teacher model at a corresponding layer.

6. The method of claim 1, wherein, The method further includes a font generation inference phase, which includes the following two modes: Mode one: when character prior information is provided, the teacher model is used for font generation, and the content character image, the style reference image, and the character prior information are taken as input conditions; Mode two: when no character prior information is provided, the student model is used for font generation, and only the content character image and the style reference image are taken as input conditions; Both mode one and mode two use a classifier-free guidance strategy for noise prediction, and a DPM-Solver++ sampling algorithm for accelerated sampling.

7. The method of claim 1, wherein, The diffusion model is a latent diffusion model, and the denoising process of the guided diffusion model specifically includes: mapping an original image to a latent space through an encoder to obtain a latent representation; performing a forward noise adding process and a backward denoising process in the latent space; in the backward denoising process, inputting the conditional features into a U-Net architecture to guide the model to iteratively denoise the noisy latent representation; mapping the denoised latent representation back to an image space through a decoder to generate a target font image.

8. A font generation system based on priori guided knowledge distillation and style perception, characterized in that, The system includes: a teacher model training module configured to obtain a content character image, a style reference image, and character prior information corresponding to the content character image; encode the content character image, the style reference image, and the character prior information to obtain content features, style features, and prior features; perform multi-modal fusion on the content features, the style features, and the prior features to generate conditional features; and guide a denoising process of a diffusion model based on the conditional features to train a teacher model; a student model training module configured to initialize a student model using the teacher model as a guide model; obtain a content character image and a style reference image as input of the student model; and train the student model using a knowledge distillation strategy through a distillation loss function, where the distillation loss function is configured to minimize a difference between the student model and the teacher model in noise prediction and intermediate feature representation.

9. An electronic device, comprising: A processor, a memory, and a program or instructions stored on the memory and executable on the processor are included, and the program or instructions are executed by the processor to implement the steps of the font generation method based on prior guided knowledge distillation and style perception according to any one of claims 1-7.

10. A readable storage medium, characterized by, A program or instructions are stored on the readable storage medium, and the program or instructions are executed by the processor to implement the steps of the font generation method based on prior guided knowledge distillation and style perception according to any one of claims 1-7.

Images