Cross-lingual Font Generation Method Based on Style Contrastive Learning and Relative Position Perception

Abstract

The present invention belongs to the technical field of computer vision, and discloses a cross-language font generation method based on style contrast learning and relative position perception, including: establishing a new cross-language font generation model, which includes a content-style decoupling network, an image patch contrast learning module, and a relative position perception module; the model extracts content features from content character images through a content encoder, extracts style features from reference character images through a style encoder, and obtains local content features through a relative position perception module at the same time; then the content features, style features, and local content features are concatenated and sent to a decoder for decoding to obtain the generated character images; the image patch contrast learning module is also used to perform contrast learning on the local structure of strokes at the image patch level. The present invention has a good cross-language font generation effect after training.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and particularly relates to a cross - language font generation method based on style contrast learning and relative position perception. Background Art

[0002] In the prior art, due to the small number of samples of new - style fonts, the corresponding common font generation methods are mainly few - shot font generation methods, that is, generating new fonts by using a small number of data samples as references. The existing few - shot font generation mainly focuses on in - language font generation. Such few - shot font generation methods are mainly based on the strategy of content - style separation, and combine character priors or use network priors to guide the model to generate the required fonts.

[0003] The way of combining character priors mainly refers to introducing specific information related to characters to guide the model to better learn content features and style features, thereby improving the font generation effect. By introducing the component information of the corresponding language characters into the model to improve the performance of the model. Representative models in these existing methods have achieved more fine - grained generation by using various font prior information specific to a certain language during the generation process, thereby enhancing the integrity and accuracy of character images. However, the prior information of different language writing systems cannot be generalized. For example, the consonants, vowels and finals of Korean characters, and the radicals or stroke orders in Chinese characters. Therefore, a large amount of time is required for manual collection, which limits the ability of cross - language font generation under zero - shot or few - shot conditions.

[0004] Another few - shot font generation method in the prior art is different from directly using character priors. It uses the way of network priors to achieve font generation. This type of method mainly captures the implicit prior information of characters by pre - training a network. By introducing the pre - trained model, the learning ability of the model can be improved. However, for the cross - language font generation requirement, since the prior information often varies, the pre - trained model trained with a small number of samples of one language as the training set needs to be retrained when applied to other languages, thus reducing the generation efficiency. Although this type of method does not require the use of font prior information, the training process is relatively complex, and when performing cross - language font generation, the originally pre - trained model may be difficult to meet the new language requirements and needs to be pre - trained again.

[0005] Therefore, both of the two types of small-sample font generation methods in the existing technology have the problem that the font prior information used is different in different languages and characters, and therefore cannot meet the needs of cross-language font generation. It is often necessary to re-extract the corresponding prior information based on the new language and character for training. Otherwise, the effect of cross-language font generation is very poor. Re-collecting prior information for training will lead to low training and generation efficiency, and will affect the generation effect of the previous language and character. Summary of the Invention

[0006] The purpose of the present invention is to provide a cross-language font generation method based on style contrast learning and relative position perception, which is used to solve the technical problem that the font generation method with a small number of samples in the existing technology generates fonts based on the prior information specific to the language and characters, which leads to poor cross-language font generation effect due to the different prior information of different languages and characters, and the character integrity and style realism cannot meet the requirements.

[0007] The cross-language font generation method based on style contrast learning and relative position perception includes: establishing a new cross-language font generation model, which includes a content-style decoupling network, an image block contrast learning module and a relative position perception module; the cross-language font generation model extracts content features from the input content character image through a content encoder, and extracts style features from the reference character image providing the target style through a style encoder, while the relative position perception module divides the content character image into several parts, and then inputs these parts into the encoder to obtain corresponding local features, wherein the local features include both local content information and corresponding relative position information. The position information is obtained through a relative position encoder, and then these local features are combined according to the original division method to generate local content features; the content features, style features and local content features are then spliced and sent to a decoder for decoding to obtain a generated character image; the cross-language font generation model also uses an image block comparison learning module to compare and learn the local structure of strokes at the image block level to further improve the style of the character stroke level; the cross-language font generation model also uses a content discriminator and a style discriminator to respectively discriminate the content features and style features of the generated font; after training and optimizing the cross-language font generation model, it is directly applied to cross-language font generation, and the decoder outputs the result.

[0008] Preferably, the image block contrast learning module first inputs the generated character image and the target character image into the corresponding style encoder, and the two style encoders share parameters with the style encoder in the content-style decoupling network; LThe output of the layer is subjected to contrastive learning, and the features of the output of each layer are divided into several image blocks, each of which contains a complete local structure of the strokes. The features of each image block of the generated character image are input into the two-layer multi-layer perceptron to obtain the image block-level style features of the generated character image, and the features of each image block of the target character image are input into the two-layer multi-layer perceptron to obtain the image block-level style features of the target character image. During contrastive learning, the current image block is the image block-level style feature of the target character image, and the corresponding image block-level style feature of the generated character image is used as a positive sample, while the remaining image block-level style features of the generated character image are used as negative samples. Based on these positive and negative samples, the contrast loss corresponding to the current image block is calculated, and the total contrast loss is obtained by scanning all image blocks, thereby realizing the optimization of the cross-language font generation model.

[0009] Preferably, in the content-style decoupling network, the content encoder uses a content encoder of a small-sample unsupervised image conversion model, and the first four layers of the content encoder are convolution blocks; each convolution block uses a reflection padding to add pixels to the image boundary to retain the stroke details and edge details of the characters, uses instance normalization to reduce the differences in data within each sample, and uses a ReLU activation function to assist the network in learning various nonlinear features of the character image, and then fuses the information through a convolution layer; two residual blocks are added to the last two layers of the content encoder.

[0010] Preferably, in the content-style decoupling network, the reflection padding in the convolution block of the class encoder of the small-sample unsupervised image conversion model is replaced with zero padding to form a style encoder; six residual blocks are used in the decoder to fully fuse the features, and transposed convolution is used to upsample the image; the discriminator structure adopts the discriminator of PatchGAN.

[0011] Preferably, the losses used in the cross-language font generation model include image block level style contrast loss, adversarial loss and L1 loss, and total loss The loss function is as follows:

[0012] ,

[0013] in, is a hyperparameter, represents the loss of the content generator, represents the loss of the content discriminator, represents the loss of the style generator, represents the loss of the style discriminator, represents the L1 loss of the model, Represents the style contrast loss at the image patch level.

[0014] Preferably, the image block level style contrast loss Style contrast learning is implemented at the image patch level, and the loss function is:

[0015] ,

[0016] where represents the generated character image, y c represents the target character image, L represents the first L layers of the style encoder, W l represents the number of image patches in the current layer, represents the contrast loss, represents the patch-level style feature of the l th w patch of the target character image, represents the patch-level style feature of the l th w patch of the generated character image, represents the patch-level style features of all patches of the generated character image except the l th w patch, represents the expected value under the distribution of and y c in the sample data.

[0017] Preferably, the L1 loss function is used to constrain the pixel-level difference between the generated character and the target character, ensuring a high degree of consistency between the two at the pixel level. The corresponding loss function is: , represents the generated character image, y c represents the target character image, represents the expected value under the distribution of and y c in the sample data.

[0018] The present invention has the following advantages: The prior information used by this method to improve the training efficiency is the local information of the font image and the image patches used in contrastive learning, and the combination of the two aspects is good; moreover, neither of them depends on specific structural components (such as radicals) based on the type of characters. Therefore, it has a good cross-language font generation effect after training. At the same time, this method improves the generation effect of both the local structural content of the text image and the local style of the strokes from two aspects: the composition of the generation model and the contrastive learning module used in training. The former introduces relative position information to the content structure of the text to guide the correspondence between the local information of the character and the spatial position, avoiding many problems existing in the existing methods, such as: missing content, misalignment of the generated characters, and mismatch of the spatial position; the latter is combined with the style features collected from the reference character image, so that the training process includes both the style features at the image patch level and the multi-level style features. The style features at the image patch level can capture the fine-grained style more accurately, especially the style transfer implemented for the local structure of the strokes, and the use of multi-level style features can obtain a richer style representation. Therefore, this method can learn more detailed and comprehensive style differences during the contrastive learning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 FIG. is a flowchart of a cross-language font generation method based on style contrast learning and relative position perception according to the present invention.

[0020] Figure 2 FIG. is a schematic diagram of a content encoder, a style encoder, a decoder, and a relative position encoder in the present invention.

[0021] Figure 3 FIG. is a schematic diagram of the generation effect in cross-language font generation according to the present invention. Among them, Figure 3 The left side shows some invisible font styles (font styles not appearing in the training set), the generated Chinese characters, and the cross-language font generation results represented by Korean and Japanese; Figure 3 The middle (a) shows the conversion relationship of the style representation of the image patches in cross-language font generation; Figure 3 The right side (b) shows a schematic diagram of the correspondence between the local content information and the spatial relative position of the same character in different fonts. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] The following further describes the specific embodiments of the present invention in detail by describing the embodiments with reference to the accompanying drawings, so as to help those skilled in the art have a more complete, accurate, and in-depth understanding of the inventive concept and technical solution of the present invention.

[0023] As Figures 1 - 3 shown, the present invention provides a cross-language font generation method based on style contrast learning and relative position perception, including the following steps.

[0024] A novel cross - language font generation model (RP - Font, where R represents the relative position awareness module and P represents the patch - level contrastive learning module) is established. The novel font generation model is based on patch - level style contrastive learning and relative position awareness technology.

[0025] The novel cross - language font generation model includes a Style - Content Disentangled Network, a Patch - Level Style Contrastive Learning Module (abbreviated as the PSCL module), and a Relative Position Awareness Module (abbreviated as the RPA module).

[0026] The cross - language font generation model extracts content features from the input content character images through a content encoder x c and extracts style features from a few reference character images providing the target style through a style encoder f c . At the same time, it learns the correspondence between the local features of characters and their spatial positions through the Relative Position Awareness (RPA) module to obtain local content features. Then all the extracted features are sent together to a decoder (Mixer) to guide the position alignment of the font and the supplementation of content features, that is, the content features, style features, and local content features are concatenated and then sent to the decoder for decoding to obtain the generated character images . To further improve the style accuracy of the generated characters, the model further improves the character stroke - level style through the Patch - Level Style Contrastive Learning (PSCL) module. In addition, the Style - Content Disentangled Network of the model is established based on the architecture of a generative adversarial network and also includes a content discriminator and a style discriminator. Therefore, the model can also discriminate the content features and style features of the generated font through the content discriminator and the style discriminator respectively.

[0027] In the content-style decoupling network, the content encoder uses the content encoder of the few-shot unsupervised image translation model (FUNIT). The first four layers of the content encoder are convolutional blocks. Each convolutional block uses a reflection padding to add pixels to the image boundary to preserve the stroke details and edge details of the characters, uses instance normalization to reduce the data variance within each sample, and uses the ReLU activation function to assist the network in learning various non-linear features of the character images, such as stroke thickness, curvature, etc. Then, the information is fused through a convolutional layer. To prevent gradient vanishing and information loss, two residual blocks are added to the last two layers of the content encoder.

[0028] The content-style decoupling network makes some modifications to the class encoder of the few-shot unsupervised image translation model (FUNIT) to form the style encoder, that is, uses zero padding to replace the reflection padding in the convolutional blocks of the class encoder, which can avoid introducing too much noise and extract purer styles.

[0029] To make full use of the content features and style features, six residual blocks are used in the decoder to fully fuse the features, and at the same time, transposed convolution is used to upsample the image.

[0030] The discriminator (including the content discriminator and the style discriminator) adopts the discriminator of PatchGAN. The PatchGAN network can consider the influence of different parts of the image, rather than just providing an overall evaluation of the entire image, which is particularly important for the font generation task.

[0031] In most languages, the relative positions of the same character part in different fonts are generally unchanged. Therefore, learning the relative positions reduces the complexity compared to learning the entire character. This method uses this relative position information to guide the acquisition of the correspondence between the local information and the spatial position of the character. To capture this information, a relative position awareness (RPA) module is set in the cross-language font generation model. The relative position awareness module divides the content character image into several parts, captures the relative position information from them, and then uses the relative position information to guide the alignment of the positions of each part in the character image, and uses the local features of each part to supplement the content features.

[0032] In the relative position awareness module, the content character image x c is divided into four parts: the upper left , the upper right , the lower left and the lower right , and then input these four parts into the encoder to obtain the corresponding local features. The local features of these four parts include both local content information and corresponding relative position information. The relative position information is obtained through the relative position encoder, and then these local features are combined according to the original division method to generate local content features. f lc Local content features f lc It not only enhances content features, but also incorporates relative position information into the generation process, providing important guidance for the alignment of local parts of characters. Since this local content information is not a priori information unique to various languages and characters, it is universal and can be applied to different languages and characters.

[0033] Although the relative position perception module provides local content and relative position information, the segmentation process is unrelated to stroke information. Consequently, the local content information contains incomplete strokes, particularly those that disrupt stroke trajectories. This can lead to inaccurate stroke style deformation during style transfer. Therefore, to further improve the accuracy of generated character styles, the cross-language font generation model uses a patch contrastive learning (PSCL) module to further enhance character stroke-level style. Since font style is often reflected in the local structure of a character's strokes, the patch contrastive learning module performs contrastive learning at the patch level, rather than at the global character level.

[0034] The image block contrast learning module first generates the character image and target character image y c Input two corresponding style encoders (generated style encoder and target style encoder), which share parameters with the style encoder in the content-style decoupling network. L The output of the layer is compared and learned, and the style encoder is l The features of the layer output are divided into W l Each image block contains a complete stroke local structure, that is, the stroke local trajectory is not destroyed by the edge of the image block. No. w The features of the image blocks are input into a two-layer multi-layer perceptron N mlp Get the image block level style features of the generated character image , follow similar operations to transform the target character image y c No. w The features of the image blocks are input into the multilayer perceptron Nmlp Obtain the image patch-level style features of the target character image , which correspond to the image patch-level style features of the generated character image .

[0035] During contrastive learning, the current image patch is the image patch-level style feature of the target character image , and the corresponding image patch-level style feature of the generated character image is used as the positive sample, while the remaining image patch-level style features of the generated character image are used as the negative samples. Based on these positive and negative samples, the contrastive loss corresponding to the current image patch is calculated. By scanning all image patches, the total contrastive loss is obtained, thereby realizing the optimization of the cross-lingual font generation model. Through the style representation at the image patch level, the image patch contrastive learning module can learn the unique style representation of a given font. Since the style features utilized in this process are based on the complete style information of the local stroke trajectories, they are not affected by the unique stroke structures in different language scripts and can be well generalized to different language systems, thus achieving cross-lingual font generation performance with highly consistent styles.

[0036] To generate high-quality results in the cross-lingual font generation task and ensure the content integrity of the generated characters and the accuracy of the font styles, the loss functions used in this method during training include: the image patch-level style contrastive loss, which is used to guide the model to capture the image patch-level style features; the content adversarial loss and the style adversarial loss, which are used to optimize the model in the character content-style decoupling task, thereby effectively distinguishing and retaining the content features of the characters while also ensuring the accuracy of the style features; and the L1 loss function, which is used to constrain the pixel-level differences between the generated characters and the target characters, ensuring a high degree of consistency between the two at the pixel level.

[0037] The image patch-level style contrastive loss is used to achieve style contrastive learning at the image patch level to obtain a finer-grained symbol style representation. The loss function of the image patch-level style contrastive loss is as follows:[[]]

[0038] ,

[0039] where represents the generated character image, y c represents the target character image, L represents the first L layers of the style encoder, W l represents the number of image patches in the current layer, represents the contrastive loss, represents the lThe w image patch-level style features of the patch of the l layer of the generated character image, w image patch-level style features of the patch of the l layer of the generated character image, w except for the patch of the and y c expected value under the

[0040] The adversarial loss includes: the loss function of the content discriminator, which focuses on distinguishing the difference between the generated content and the real content; the loss function of the style discriminator, which focuses on distinguishing the difference between the generated style and the real style. The loss function of the content adversarial loss is as follows:

[0041] ,

[0042] ,

[0043] where x c is the input content image, represents the generated character image, y c represents the target character image, D c represents the content discriminator, represents the loss of the content generator, represents the and x c expected value under the represents the loss of the content discriminator, represents the y c and x c expected value under the

[0044] Similar to the content adversarial loss, the loss function of the style adversarial loss is as follows:

[0045] ,

[0046] ,

[0047] where x s is the image of the input reference character image, which is used to provide style information, represents the generated character image,y c Represents the target character image, D s Represents the style discriminator, Represents the loss of the style generator, Represents in the sample data and x s The expected value under the distribution, Represents the loss of the style discriminator, Represents in the sample data y c and x s The expected value under the distribution.

[0048] The L1 loss, which measures the difference between the generated image and the target image at the pixel level. The corresponding loss function is: , Represents the L1 loss of the model.

[0049] According to the above loss function, the losses used by the cross - language font generation model in the training stage include the style contrast loss at the image patch level, the adversarial loss, and the L1 loss. The total loss The loss function is as follows:

[0050] ,

[0051] Among them, is a hyperparameter. In subsequent experiments, its value can be default set to 1.

[0052] Based on the above model structure and loss function, the model is trained and optimized with a small number of reference character images. The cross - language font generation model obtained after training can be directly applied to cross - language font generation, and the results are output by the decoder.

[0053] Figure 3 Some invisible font styles (font styles not appearing in the training set), the generated Chinese characters, and the cross - language font generation results represented by Korean and Japanese are shown on the left; Figure 3 The conversion relationship of the style representation of the image patch in cross - language font generation is shown in (a) in the middle; Figure 3 The schematic diagram of the corresponding relationship between the local content information and the spatial relative position of the same character in different fonts is shown in (b) on the right, thereby reflecting the role and effect of the image patch contrast learning module and the relative position perception module in cross - language font generation.

[0054] The present invention has been described above in an exemplary manner with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited by the above-mentioned manner. As long as various non-substantive improvements are made by adopting the inventive concept and technical solution of the present invention, or the inventive concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A cross - language font generation method based on style contrast learning and relative position perception, characterized in that: Including: Building a new cross - language font generation model, which includes a content - style decoupling network, an image patch contrast learning module, and a relative position perception module; the cross - language font generation model extracts content features from the input content character image through a content encoder, and extracts style features from the reference character image providing the target style through a style encoder. At the same time, the relative position perception module divides the content character image into several parts, and then inputs these parts into the encoder to obtain corresponding local features. The local features include both local content information and corresponding relative position information. The relative position information is obtained through a relative position encoder, and then these local features are combined according to the original division method to generate local content features; then the content features, style features, and local content features are concatenated and sent to a decoder for decoding to obtain the generated character image; the cross - language font generation model also performs contrast learning on the stroke local structure at the image patch level through the image patch contrast learning module to further improve the style at the character stroke level; the cross - language font generation model also discriminates the content features and style features of the generated font through a content discriminator and a style discriminator respectively; After training and optimizing the cross - language font generation model, it is directly applied to cross - language font generation, and the result is output by the decoder; The image patch contrast learning module first inputs the generated character image and the target character image into the corresponding style encoder. These two style encoders share parameters with the style encoder in the content - style decoupling network; contrast learning is carried out using the output of the first L layers of the style encoder. The features output by each layer are divided into several image patches, and each image patch contains a complete stroke local structure. The features of each image patch of the generated character image are input into a two - layer multi - layer perceptron to obtain the image patch - level style features of the generated character image, and the features of each image patch of the target character image are input into a two - layer multi - layer perceptron to obtain the image patch - level style features of the target character image. During contrast learning, the current image patch is the image patch - level style feature of the target character image, and the corresponding image patch - level style feature of the generated character image is used as a positive sample, while the remaining image patch - level style features of the generated character image are used as negative samples. Based on these positive and negative samples, the contrast loss corresponding to the current image patch is calculated, and the total contrast loss is obtained by scanning all image patches, thereby realizing the optimization of the cross - language font generation model. Learning the unique style representation of a given font through style representation at the image patch level. The style features used are based on the complete style information of the stroke local trajectory. The cross - language font generation model obtained after training can be directly applied to cross - language font generation to achieve cross - language font generation under zero - sample conditions.

2. The cross-language font generation method based on style contrast learning and relative position perception according to claim 1, wherein: In the content-style decoupling network, the content encoder uses the content encoder of the few-shot unsupervised image transformation model. The first four layers of the content encoder are convolutional blocks; each convolutional block uses a reflection padding to add pixels to the image boundary to preserve the stroke details and edge details of the characters, adopts instance normalization to reduce the intra-sample data variability, uses the ReLU activation function to assist the network in learning various non-linear features of the character images, and then fuses the information through a convolutional layer; two residual blocks are added to the last two layers of the content encoder.

3. The cross-language font generation method based on style contrast learning and relative position perception according to claim 1, characterized in that: In the content-style decoupling network, the reflection padding in the convolutional blocks of the class encoder of the few-shot unsupervised image transformation model is replaced with zero padding to form the style encoder; six residual blocks are used in the decoder to fully fuse the features, and at the same time, transposed convolution is used to upsample the image; The structure of the discriminator adopts the discriminator of PatchGAN.

4. The cross-language font generation method based on style contrast learning and relative position perception according to claim 1, wherein: The losses used in the cross - language font generation model include the style contrast loss at the image patch level, the adversarial loss, and the L1 loss. The total loss The loss function is as follows: Among them, λ pscl is a hyperparameter, represents the loss of the content generator, represents the loss of the content discriminator, represents the loss of the style generator, represents the loss of the style discriminator, represents the L1 loss of the model, represents the style contrast loss at the image patch level.

5. The cross-language font generation method based on style contrast learning and relative position perception according to claim 4, wherein: Style contrast loss at the image patch level It is used to implement style contrast learning at the image patch level, and the loss function is: Among them, represents the generated character image, y c represents the target character image, L represents the first L layers of the style encoder, W l represents the number of image patches in the current layer, l(·) represents the contrastive loss, represents the patch-level style feature of the l-th layer and w-th patch of the target character image, represents the patch-level style feature of the l-th layer and w-th patch of the generated character image, represents the patch-level style feature of all patches except the l-th layer and w-th patch of the generated character image, represents in the sample data and y c the expected value under the distribution.

6. The cross - language font generation method based on style contrast learning and relative position perception according to claim 4, characterized in that: The L1 loss function is used to constrain the pixel-level difference between the generated character and the target character, ensuring a high degree of consistency between the two at the pixel level. The corresponding loss function is as follows: represents the generated character image, y c represents the target character image, represents in the sample data and y c the expected value under the distribution.

Images