Font generation method and system based on prior guidance knowledge distillation and style perception

Abstract

The present application belongs to the field of computer vision, and discloses a font generation method and system based on priori guidance knowledge distillation and style perception, which comprises two training stages: in the first stage, a teacher model is trained with content character image, style reference image and character prior information as conditional input; in the second stage, through knowledge distillation, a student model is trained with only content and style image as input, so that its performance approximates to that of the teacher model. The present application proposes a multi-directional style perception Mamba module for extracting comprehensive and robust style features, and a hierarchical multi-modal feature fusion module for efficiently integrating various inputs of the teacher model; through the method, the guiding role of prior knowledge is internalized into the student model, so that the model can generate high-quality font images with accurate content and consistent style even in the absence of prior information during reasoning.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a font generation method and system based on prior guided knowledge distillation and style perception. Background Technology

[0002] Font generation aims to automatically synthesize character images with a new style based on the content of a source character image and the style of a reference character image. This technology has significant application value in advertising design, ancient book restoration, and personalized font customization, especially for languages ​​with large character sets and complex structures such as Chinese and Korean, where it can significantly reduce the manual cost of font design. An ideal font generation method must simultaneously meet two core requirements: the content of the generated characters should be strictly consistent with the source characters, and their artistic style should be highly faithful to the reference characters.

[0003] Early research often modeled font generation as an image-to-image translation problem, such as the Zi2Zi model which uses Generative Adversarial Networks (GANs) for translation. However, these methods heavily rely on large-scale paired training samples (i.e., images of the same character in multiple styles) and cannot generalize to "invisible styles" not present in the training set, which greatly limits their practical applications. Subsequently, the paradigm of decoupling content and style became mainstream, such as the EMD and SA-VAE models. These models extract content features from the content image and style features from the reference image, respectively, and then fuse them to generate the font. This decoupling idea enables the model to handle font generation with invisible styles, representing a significant advancement in the field.

[0004] To further improve the quality of generated data, existing technologies are mainly developing along two paths:

[0005] Network model-centric approaches focus on designing superior network architectures to enhance feature extraction, decoupling, and fusion capabilities. For example, NTF-Font treats font generation as a continuous process within a neural transform domain; FontDiffuser improves content integrity and style fidelity by introducing multi-scale content features and style contrast learning. Despite these achievements, problems such as inconsistent local style features, stroke overlap, or missing strokes are still prevalent when dealing with fonts with complex strokes or unique styles.

[0006] Methods utilizing prior character information: These methods introduce inherent structural information about characters (such as strokes and radical components) to assist model generation. This type of method can be further divided into two categories: one uses prior information as a supervisory signal. For example, MX-Font uses multiple expert networks to learn different local features and supplements this with a classifier for supervision; CG-GAN designs a component-level discriminator to achieve finer-grained discrimination. However, due to the blurred boundaries and nested structures between character components, such as Chinese characters, accurately locating and identifying the information of each component is extremely difficult. The other type uses prior information as input conditions for the model. For example, Diff-Font integrates stroke information to guide generation, but its applicability is limited to visible characters; IF-Font uses Ideographic Sequences (IDS) as content input. Although it introduces spatial relationships between components and can generate invisible characters, IDS cannot accurately describe the relative positions, distances, and size ratios between components, leading to an unbalanced component layout in the generated results. Furthermore, these methods generally suffer from a fatal flaw: their performance is highly dependent on the availability of prior information. In open scenarios where cross-language generation or prior information is lacking, its effectiveness cannot be guaranteed.

[0007] Meanwhile, existing style encoders also have limitations. Font style is multi-dimensional, encompassing global features such as font size and slant, as well as local details such as stroke thickness, the shape of the beginning and ending strokes, and whether strokes are connected. Currently used style encoders based on convolutional neural networks (CNNs) are insufficient in capturing global and long-range dependent style features of characters due to their inherent local receptive fields. While encoders based on self-attention mechanisms have global receptive fields, their computational complexity is high, and they treat all features equally, potentially ignoring crucial local fine textures that are essential to style.

[0008] In summary, existing font generation methods either suffer from insufficient model capabilities leading to poor performance under complex styles, or their over-reliance on prior information limits their application scenarios. Furthermore, there is still room for improvement in the comprehensive and efficient extraction of style features. Summary of the Invention

[0009] To address the shortcomings of existing technologies, such as poor generation performance when dealing with complex styles (e.g., inconsistent styles, missing strokes) and strong reliance on prior character information, which limits their application, this invention proposes a font generation method based on prior-guided knowledge distillation and a style-aware Mamba module. This invention aims to improve generation quality and model generalization by employing a two-stage training framework combined with an efficient style encoder. This allows the model to leverage prior information to enhance its performance ceiling while maintaining high-quality generation capabilities even in the absence of prior information.

[0010] In a first aspect, the present invention provides a font generation method based on prior guided knowledge distillation and style-aware Mamba modules, comprising the following steps:

[0011] Obtain the content character image, the style reference image, and the character prior information corresponding to the content character image;

[0012] The content character image, style reference image, and character prior information are encoded respectively to obtain content features, style features, and prior features;

[0013] The content features, style features, and prior features are fused using a multimodal method to generate conditional features;

[0014] Based on the aforementioned conditional features, the denoising process of the diffusion model is guided to train and obtain the teacher model;

[0015] The teacher model is used as the guidance model, and the student model is initialized.

[0016] The content character image and style reference image are obtained as input to the student model;

[0017] A knowledge distillation strategy is employed to train the student model using a distillation loss function, which minimizes the differences between the student model and the teacher model in noise prediction and intermediate feature representation.

[0018] As an optional implementation of the first aspect of this application, the step of encoding the content character image, style reference image, and character prior information to obtain content features, style features, and prior features specifically includes: inputting the style reference image into a style encoder, the style encoder including a multi-directional style-aware Mamba module; the multi-directional style-aware Mamba module dividing the input features into a first branch and a second branch along the channel; the first branch sequentially performing linear layer expansion, depthwise separable convolution, activation function processing, and style-aware grouping direction scanning operation on the features, and then normalizing them; the second branch sequentially performing linear layer expansion and activation function processing on the features; and the processing results of the first branch and the second branch are element-wise multiplied and projected back to the original dimension through a linear layer to obtain the style features.

[0019] As an optional implementation of the first aspect of this application, in the steps of the first branch sequentially performing linear layer augmentation, depthwise separable convolution, activation function processing, and style-aware grouped directional scanning operation on the features, the style-aware grouped directional scanning operation specifically includes: uniformly dividing the input features of the first branch into eight feature groups along the channel dimension; assigning an independent scanning direction to each feature group, the eight scanning directions including horizontal scanning, vertical scanning, diagonal scanning, and their respective flipped scanning; performing a selective scanning operation on each feature group along its corresponding scanning direction to obtain scanning results in eight directions; and concatenating and merging the scanning results in eight directions along the channel dimension to generate a hybrid style feature that integrates multi-directional information.

[0020] As an optional implementation of the first aspect of this application, the step of multimodal fusion of content features, style features, and prior features is implemented through a hierarchical multimodal feature fusion module, specifically including: a first fusion stage: using the content features as a query and the style features as keys and values, interacting through a first multi-head cross-attention mechanism to generate preliminary fused features; a second fusion stage: using the preliminary fused features as a query and the prior features as keys and values, interacting through a second multi-head cross-attention mechanism to generate the final conditional features.

[0021] As an optional implementation of the first aspect of this application, the distillation loss function includes: a base loss term for calculating the difference between the noise predicted by the student model and the actual noise; a noise distillation loss term for calculating the difference between the noise predicted by the student model and the noise predicted by the teacher model; and a feature distillation loss term for calculating the difference between the intermediate feature map extracted by the student model in the U-Net architecture of the diffusion model and the intermediate feature map extracted by the teacher model in the corresponding layer.

[0022] As an optional implementation of the first aspect of this application, the method further includes a font generation inference stage, which includes the following two modes: Mode 1: when character prior information is provided, the teacher model is used for font generation, and the content character image, style reference image, and character prior information are used as input conditions; Mode 2: when character prior information is not provided, the student model is used for font generation, and only the content character image and style reference image are used as input conditions; both Mode 1 and Mode 2 use a classifier-free guidance strategy for noise prediction and use the DPM-Solver++ sampling algorithm for accelerated sampling.

[0023] As an optional implementation of the first aspect of this application, the diffusion model is a latent diffusion model, and the denoising process of the guided diffusion model specifically includes: mapping the original image to the latent space through an encoder to obtain a latent representation; performing a forward denoising process and a backward denoising process in the latent space; inputting the conditional features into the U-Net architecture during the backward denoising process to guide the model to iteratively denoise the noisy latent representation; and mapping the denoised latent representation back to the image space through a decoder to generate a target font image.

[0024] Secondly, embodiments of this application also provide a font generation system based on prior guided knowledge distillation and style awareness, the system comprising:

[0025] A teacher model training module is used to acquire content character images, style reference images, and character prior information corresponding to the content character images; encode the content character images, style reference images, and character prior information respectively to obtain content features, style features, and prior features; perform multimodal fusion on the content features, style features, and prior features to generate conditional features; and guide the denoising process of the diffusion model based on the conditional features to train the teacher model.

[0026] The student model training module is used to use the teacher model as the guiding model and initialize the student model; it acquires content character images and style reference images as inputs to the student model; it employs a knowledge distillation strategy and trains the student model using a distillation loss function, which minimizes the differences between the student model and the teacher model in noise prediction and intermediate feature representation.

[0027] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the method described in the first aspect.

[0028] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0029] Compared with the prior art, the present invention has the following significant advantages:

[0030] 1. Improved accuracy and generalization ability of style modeling: Through an innovative two-stage knowledge distillation strategy, this invention integrates the advantages of prior character information into model training, while the resulting student model is free from dependence on prior information during inference. This enables the model to not only leverage prior knowledge to enhance its understanding of complex character structures and styles, but also to flexibly apply it in open scenarios involving cross-language communication and the absence of prior information.

[0031] 2. Achieves richer and more robust style representation: The multi-directional style-aware Mamba module designed in this invention, especially its core eight-directional group scanning mechanism, can comprehensively extract global (e.g., character size) and local (e.g., stroke connection) style features of character images from multiple spatial perspectives. Compared with traditional CNNs and self-attention mechanisms, this module improves both the comprehensiveness and efficiency of feature extraction.

[0032] 3. Achieved efficient multimodal information fusion and knowledge transfer: The hierarchical multimodal feature fusion module proposed in this invention can orderly and efficiently integrate features from three modalities: content, style, and prior information. Simultaneously, the loss function design in the knowledge distillation stage not only supervises the final output but also the intermediate features, providing the student model with finer-grained learning signals, ensuring the effectiveness of knowledge transfer, and enabling it to achieve high-quality generation even in the absence of prior input. Attached Figure Description

[0033] Figure 1 This is a flowchart of a font generation method based on prior guided knowledge distillation and style awareness according to an embodiment of the present invention;

[0034] Figure 2 This is a general framework diagram of a font generation model based on prior guided knowledge distillation and style awareness (KDMFont model) proposed in an embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of the Style-Aware Grouped Direction Scanning (SAGDS) mechanism according to an embodiment of the present invention;

[0036] Figure 4 This is a qualitative comparison chart of our method with five other basic models on the visible style, invisible character, and invisible style invisible character datasets;

[0037] Figure 5 This is a comparison chart showing the effectiveness of the knowledge distillation strategy when no IDS information is input during the cross-language font generation stage of this method.

[0038] Figure 6 The images show the generation results of network-based methods, prior-based methods, and the proposed method.

[0039] Figure 7This is a schematic diagram illustrating different feature extraction methods;

[0040] Figure 8 This is a schematic diagram of the structure of a font generation system based on prior guided knowledge distillation and style perception according to an embodiment of the present invention.

[0041] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0044] Example 1

[0045] Please see Figure 1 The flowchart below illustrates a font generation method based on prior guided knowledge distillation and style awareness, as provided in an embodiment of the present invention.

[0046] The method proposed in this invention is based on a stable diffusion model. The stable diffusion model comprises two processes: forward noise addition and backward denoising. Forward noise addition primarily involves gradually adding noise to the image according to Markov chain rules until it becomes isotropic pure noise. Backward denoising primarily involves gradually recovering the original image from the noise. To further conserve computational resources, this invention employs a latent diffusion model (LDM). Given an image X, the encoder trained in the stable diffusion model is first used... Encode it to obtain Then add noise to it to get Based on the input condition c, predict the added noise. During the test, samples were randomly taken from a Gaussian distribution. Then, the trained model is used to predict the corresponding noise, and finally, the result is obtained. and using the decoder Decode it back into image X. This noise-adding process can be represented as:

[0047]

[0048] in X represents the original image. This indicates the added noise. , It is a predefined variance scheduler.

[0049] The noise reduction process can be summarized as follows:

[0050]

[0051] in This represents the predicted noise. For hyperparameters, . Represents the latent variable corresponding to time t. This represents the latent variable at time t-1.

[0052] Please see Figure 2 This is a diagram of the overall framework of a font generation model (KDMFont model) based on prior guided knowledge distillation and style awareness proposed in this embodiment of the invention. The overall framework includes three stages: (1) Preliminary training stage: The model input conditions include three parts: content character image, style reference image and character prior information (IDS) corresponding to the content character image, and style reference image; the multi-directional style awareness Mamba module is used to construct the style encoder and extract multi-directional style features; the hierarchical multimodal feature fusion module (HMMFI) fuses the extracted conditional features. (2) Knowledge distillation stage: The model in the previous stage is subjected to knowledge distillation so that it can adapt to the situation without IDS. (3) Inference stage: There are two choices in the model inference process: one is to use IDS and the other is not to use IDS.

[0053] Specifically, the initial training phase mainly involves training the teacher model, corresponding to the following steps S1 to S4.

[0054] S1: Obtain the content character image, the style reference image, and the character prior information corresponding to the content character image;

[0055] S2: Encode the content character image, style reference image, and character prior information respectively to obtain content features, style features, and prior features;

[0056] like Figure 2As shown, the input conditions include a content character image, a style reference image, and character prior information (IDS) corresponding to the content character image. These are then processed by their respective encoders (content encoder, style encoder, and IDS encoder) to obtain the corresponding features. The content encoder consists of four convolutional blocks and two residual blocks, the IDS encoder consists of one encoding layer and four Transformer layers, and the style encoder consists of a multi-directional style-aware Mamba module.

[0057] like Figure 2 As shown, the multi-directional style-aware Mamba module divides the input features into the first branch along the channels. Second branch ;

[0058] In the first branch, the input features first pass through a linear layer ( The channels are expanded. Then, they are sequentially passed through depthwise separable convolutions (…). The process employs the SiLU activation function and the Style-Aware Grouped Directional Scanning (SAGDS) operation proposed in this invention to extract multi-directional mixed style features. Finally, it passes through a LayerNorm layer for normalization to obtain the processing result of the second branch. ;

[0059] In the second branch, input features Similarly, it first goes through a linear layer ( The feature channels are expanded, and then passed through a SiLU activation function to obtain the processing result of the second branch. ;

[0060] The result of the first branch is element-wise multiplied with the result of the second branch, and then processed through a linear layer ( Projecting back to the original dimension yields the style features;

[0061] The above process can be represented as:

[0062]

[0063] in This represents the product of elements.

[0064] Furthermore, the stylistic features of a character are often quite complex; for example, font size, stroke thickness, and whether strokes are connected all affect the style of the font. Therefore, this invention proposes Style-Aware Grouped Oriented Scanning (SAGDS) in the first branch, such as... Figure 3As shown, this approach aims to better extract local and global style. A character stroke includes not only horizontal and vertical parts but also some diagonal parts. To fully explore the stylistic features of a character, SAGDS's scanning method includes eight feature groups evenly divided along the channel dimension: horizontal scanning, vertical scanning, diagonal scanning, and a flipped scanning. This design not only effectively extracts the global style of the character but also enhances the modeling ability of local region information. Therefore, inspired by UltraLight VM-UNet, this invention introduces the above-mentioned grouped scanning strategy to save parameters. The input channel is divided into eight parts, each corresponding to a separate scanning direction. Specifically, assuming a given input feature... First, its dimensions are reshaped to (B, L, C), where L = H x W, and B represents the batch size, H represents the feature height, and W is the feature width. Then, it is divided into 8 parts along the channel. , , i∈(1,8), then Input selective scan layer to obtain the corresponding Finally, all of them The components are concatenated and their dimensions are reshaped back to the original dimensions (B, C, H, W) to obtain the final output. The SAGDS operation process can be represented as follows:

[0065]

[0066] ChannelSplit represents partitioning along the channel dimension, SelectiveScan represents selective scanning, and Merge represents merging along the channel dimension. This indicates a hybrid style feature that integrates information from multiple directions.

[0067] S3: Perform multimodal fusion of the content features, style features, and prior features to generate conditional features;

[0068] S4: Based on the aforementioned conditional features, guide the denoising process of the diffusion model to train and obtain the teacher model.

[0069] After obtaining the individual features, we fuse the three features using a hierarchical multimodal feature fusion module (HMMFI). Specifically, this hierarchical multimodal feature fusion module is based on a two-layer multi-head cross-attention mechanism fusion stage.

[0070] In the first integration phase, content features As a query and style feature Interacting through a first multi-head cross-attention mechanism to extract style information relevant to the current content, thereby generating preliminary fusion features. .

[0071] In the second stage, initial integration features As a query, further queries can be performed based on semantic features of IDS. By incorporating prior character information and employing a second multi-head cross-attention mechanism, the fused conditional features are ultimately obtained. .

[0072] The conditional features obtained by final fusion These conditions are input into U-Net to guide the model's denoising process. This process can be represented as...

[0073]

[0074] in These represent content features, IDS features, and style features, respectively, while CrossAttn represents the cross-attention mechanism.

[0075] The loss function used in this process is:

[0076]

[0077] in Represents real noise. Indicates prediction noise, This represents the basic loss term.

[0078] Specifically, the knowledge distillation stage mainly involves training the student model, corresponding to the following steps S5 to S7.

[0079] S5: Use the teacher model as the guidance model and initialize the student model;

[0080] S6: Obtain the content character image and style reference image as input to the student model;

[0081] S7: Employ a knowledge distillation strategy to train the student model using a distillation loss function, which minimizes the differences between the student model and the teacher model in noise prediction and intermediate feature representation.

[0082] The model obtained in the previous stage is used as the teacher model to guide the training of the student model in this stage. The architecture of the student model is identical to that of the teacher model, except that the hierarchical multimodal feature fusion module is replaced with a single cross-attention module. Furthermore, the student model only needs to input content and style images. In this stage, only the student model is trained, while the parameters of the teacher model remain frozen. The loss function used in this stage is:

[0083]

[0084] in and These are hyperparameters used to balance various losses. Represents real noise. This represents the noise in the teacher model's predictions. This represents the noise in the student model's predictions. These are the content image and the style image, respectively. Noise distillation loss term. The aim is to enhance the student model's ability to mimic the teacher model's behavior by minimizing the difference between the student and teacher models in noise prediction. Furthermore, feature distillation loss is introduced. This loss is calculated based on intermediate feature representations in the U-Net architecture. Specifically, and These represent the feature maps extracted from the i-th layer of the teacher model and the student model, respectively. Compared to supervised methods that rely solely on noise prediction, It provides the student model with finer-grained and more learnable intermediate feature signals, enabling it to achieve high-quality font generation even in the absence of character priors (such as IDS) input.

[0085] Specifically, the reasoning phase includes the following two modes:

[0086] Mode 1: When character prior information is provided, the teacher model is used to generate the font, with the content character image, style reference image and character prior information as input conditions;

[0087] Mode 2: When no prior character information is provided, the student model is used for font generation, with only the content character image and style reference image as input conditions;

[0088] Both Mode 1 and Mode 2 employ a classifier-free guidance strategy for noise prediction and use the DPM-Solver++ sampling algorithm for accelerated sampling.

[0089] Therefore, there are two options at this stage: one is to use the teacher model to generate the input IDS, and the other is to use the student model to generate the input IDS.

[0090] (1) If IDS is input, the noise prediction process is as follows:

[0091]

[0092] in , , It is a predefined variance scheduler, where 's' represents the guiding scale. For hyperparameters, . Indicates the IDS corresponding to the character. This indicates that the condition is empty.

[0093] (2) If IDS is not input, the noise prediction process is as follows:

[0094]

[0095] in , , It is a predefined variance scheduler, where 's' represents the guiding scale. For hyperparameters, .

[0096] In this embodiment, extensive empirical studies were also conducted, and the results show that the proposed method exhibits superior performance in font generation tasks, fully verifying its effectiveness.

[0097] like Figure 4 The figure shows a qualitative comparison of our method with five other basic models on datasets of visible style, invisible characters, and invisible style invisible characters. The MX-Font model improves the precision of character generation at the local detail level by decomposing characters into predefined components and learning their feature representations using a multi-expert mechanism. The CF-Font model introduces a Content Feature Fusion (CFM) module, which adaptively fuses content features among multiple base fonts and combines iterative style vector optimization to effectively alleviate the problem of complete decoupling between content and style, thus significantly improving the overall quality and structural consistency of generated characters. The NTF-Font model models font generation as a continuous neural transformation process, leveraging the expressive power of the transform domain to achieve smoother style transfer. The IF-Font model innovatively uses the descriptive sequence of ideographic characters as content input and completes character generation based on an autoregressive model, demonstrating strong modeling capabilities. The FontDiffuser model, within the diffusion model framework, combines multi-scale content features and style contrast learning to effectively improve the performance of generated characters in both local details and global style.

[0098] like Figure 5 As shown, in order to verify the effectiveness of the knowledge distillation strategy in this invention, some cross-language font generation was performed. No IDS information was input during this cross-language font generation stage.

[0099] like Figure 6 As shown, the shortcomings of network-based methods and prior-based methods are illustrated, along with the generation effect of this method. The first column represents network-based methods, the second column represents methods using prior information, and the third column represents this method and the target character.

[0100] like Figure 7 As shown, a comparison of different feature extraction methods is presented. Figure 7 In the diagram, (a) represents a convolution-based extraction method with a local receptive field. Figure 7 (b) The attention-based approach has a global receptive field, and performing the same operation everywhere can easily overlook local details. Figure 7 (c) This method can simultaneously combine the global receptive field of the multi-directional style-aware Mamba module, and can capture local features well through scanning in 8 directions.

[0101] Example 2

[0102] Please see Figure 8 The second embodiment of this application presents a schematic diagram of a font generation system based on prior guided knowledge distillation and style awareness. The system includes the following key modules:

[0103] The teacher model training module 100 is used to acquire content character images, style reference images, and character prior information corresponding to the content character images; encode the content character images, style reference images, and character prior information respectively to obtain content features, style features, and prior features; perform multimodal fusion on the content features, style features, and prior features to generate conditional features; and guide the denoising process of the diffusion model based on the conditional features to train the teacher model.

[0104] The student model training module 200 is used to use the teacher model as the guiding model and initialize the student model; acquire content character images and style reference images as inputs to the student model; and train the student model using a knowledge distillation strategy and a distillation loss function, wherein the distillation loss function is used to minimize the difference between the student model and the teacher model in noise prediction and intermediate feature representation.

[0105] The font generation system based on prior guided knowledge distillation and style awareness in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), etc. This application embodiment does not impose specific limitations.

[0106] The font generation system based on prior guided knowledge distillation and style awareness in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0107] This application provides a font generation system based on prior guided knowledge distillation and style awareness, which can achieve... Figure 1 The various processes implemented in the method embodiment of a font generation method based on prior guided knowledge distillation and style awareness are not described in detail here to avoid repetition.

[0108] Optionally, embodiments of this application also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described embodiment of a font generation method based on prior guided knowledge distillation and style awareness, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0109] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described embodiment of a font generation method based on prior guided knowledge distillation and style awareness, and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0110] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0111] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0113] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A font generation method based on prior-guided knowledge distillation and style awareness, characterized in that, Includes the following steps: Obtain the content character image, the style reference image, and the character prior information corresponding to the content character image; The content character image, style reference image, and character prior information are encoded respectively to obtain content features, style features, and prior features; The specific steps involved in encoding the style reference image include: The style reference image is input to the style encoder, which includes a multi-directional style-aware Mamba module. The multi-directional style-aware Mamba module divides the input features into a first branch and a second branch along the channel. The first branch performs linear layer augmentation, depthwise separable convolution, activation function processing, and style-aware grouping direction scanning operations on the features in sequence, and then normalizes them. The style-aware grouping direction scanning operation specifically includes the following steps: uniformly dividing the input features of the first branch into eight feature groups along the channel dimension; assigning an independent scanning direction to each feature group, wherein the eight scanning directions include horizontal scanning, vertical scanning, diagonal scanning, and their respective flip scanning; performing a selective scanning operation on each feature group along its corresponding scanning direction to obtain scanning results in eight directions; and merging the scanning results in the eight directions along the channel dimension to generate a hybrid style feature that integrates multi-directional information. The second branch sequentially performs linear layer augmentation and activation function processing on the features; the processing results of the first branch and the second branch are element-wise multiplied and projected back to the original dimension through a linear layer to obtain the style features; The content features, style features, and prior features are fused using a multimodal method to generate conditional features; Based on the aforementioned conditional features, the denoising process of the diffusion model is guided to train and obtain the teacher model; The teacher model is used as the guidance model, and the student model is initialized. The content character image and style reference image are obtained as input to the student model; A knowledge distillation strategy is employed to train the student model using a distillation loss function, which minimizes the differences between the student model and the teacher model in noise prediction and intermediate feature representation.

2. The method according to claim 1, characterized in that, The step of multimodal fusion of the content features, style features, and prior features is implemented through a hierarchical multimodal feature fusion module, specifically including: First fusion stage: Using the content features as queries and the style features as keys and values, the system interacts through a first multi-head cross-attention mechanism to generate preliminary fusion features; The second fusion stage involves using the preliminary fusion features as queries and the prior features as keys and values, interacting through a second multi-head cross-attention mechanism to generate the final conditional features.

3. The method according to claim 1, characterized in that, The distillation loss function includes: The basic loss term is used to calculate the difference between the noise predicted by the student model and the actual noise. The noise distillation loss term is used to calculate the difference between the noise predicted by the student model and the noise predicted by the teacher model. The feature distillation loss term is used to calculate the difference between the intermediate feature map extracted by the student model in the U-Net architecture of the diffusion model and the intermediate feature map extracted by the teacher model in the corresponding layer.

4. The method according to claim 1, characterized in that, The method also includes a font generation inference stage, which includes the following two modes: Mode 1: When character prior information is provided, the teacher model is used to generate the font, with the content character image, style reference image and character prior information as input conditions; Mode 2: When no prior character information is provided, the student model is used for font generation, with only the content character image and style reference image as input conditions; Both Mode 1 and Mode 2 employ a classifier-free guidance strategy for noise prediction and use the DPM-Solver++ sampling algorithm for accelerated sampling.

5. The method according to claim 1, characterized in that, The diffusion model is a latent diffusion model, and the denoising process of the guided diffusion model specifically includes: The original image is mapped to the latent space through an encoder to obtain a latent representation; The forward noise addition process and the backward noise reduction process are performed in the potential space; During the backward denoising process, the conditional features are input into the U-Net architecture to guide the model to iteratively denoise the noisy latent representation; The latent representation obtained after denoising is mapped back to the image space through a decoder to generate the target font image.

6. A font generation system based on prior-guided knowledge distillation and style awareness, characterized in that, The system includes: A teacher training model module is used to acquire content character images, style reference images, and character prior information corresponding to the content character images; the content character images, style reference images, and character prior information are encoded respectively to obtain content features, style features, and prior features; wherein, encoding the style reference image specifically includes: inputting the style reference image to a style encoder, the style encoder including a multi-directional style-aware Mamba module; the multi-directional style-aware Mamba module divides the input features into a first branch and a second branch along the channel; the first branch sequentially performs linear layer expansion, depthwise separable convolution, activation function processing, and style-aware grouping direction scanning operation on the features, and performs normalization; wherein, the style-aware grouping direction scanning operation specifically includes: normalizing the input features of the first branch along the channel dimension. The system is divided into eight feature groups; each feature group is assigned an independent scanning direction, including horizontal scanning, vertical scanning, diagonal scanning, and their respective flip scanning; selective scanning is performed on each feature group along its corresponding scanning direction to obtain scanning results in eight directions; the scanning results in the eight directions are concatenated and merged along the channel dimension to generate a hybrid style feature that integrates multi-directional information; the second branch sequentially performs linear layer expansion and activation function processing on the features; the processing results of the first branch and the processing results of the second branch are element-wise multiplied and projected back to the original dimension through a linear layer to obtain the style feature; the content feature, style feature, and prior feature are multimodally fused to generate conditional features; based on the conditional features, the denoising process of the diffusion model is guided to train the teacher model; The student model training module is used to use the teacher model as the guiding model and initialize the student model; it acquires content character images and style reference images as inputs to the student model; it employs a knowledge distillation strategy and trains the student model using a distillation loss function, which minimizes the differences between the student model and the teacher model in noise prediction and intermediate feature representation.

7. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of a font generation method based on prior guided knowledge distillation and style awareness as described in any one of claims 1-5.

8. A readable storage medium, characterized in that, The program or instructions are stored on the readable storage medium, and when the program or instructions are executed by the processor, they implement the steps of the font generation method based on prior guided knowledge distillation and style awareness as described in any one of claims 1-5.

Images