Cross-language font generation method based on style contrast learning and relative position perception

Abstract

The invention belongs to the technical field of computer vision, and discloses a cross-language font generation method based on style comparative learning and relative position perception, comprising: establishing a novel cross-language font generation model comprising a content-style decoupling network, an image block comparative learning module and a relative position perception module; the model extracts content features from a content character image through a content encoder, extracts style features from a reference character image through a style encoder, and obtains local content features through a relative position sensing module; the content features, the style features and the local content features are spliced and then sent to a decoder to be decoded, and a generated character image is obtained; and an image block comparison learning module is used for carrying out comparison learning on the local structures of the strokes on the image block level. The cross-language font generation method has a good cross-language font generation effect after training.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a cross-language font generation method based on style contrast learning and relative position perception. Background Art

[0002] In the prior art, since there are few samples of new style fonts, the corresponding commonly used font generation methods are mainly few-sample font generation methods, that is, generating new fonts by using a small number of data samples as references. Existing few-sample font generation mainly focuses on font generation within a language. This type of few-sample font generation method is mainly based on the strategy of content-style separation, and combines character priors or uses network priors to guide the model to generate the required fonts.

[0003] The method of combining character priors mainly refers to introducing specific information related to characters to guide the model to better learn content features and style features, thereby improving the font generation effect. The performance of the model is improved by introducing the component information of the corresponding language and characters into the model. The representative models of these existing methods achieve more fine-grained generation by using various font prior information specific to a certain language to assist the model in the generation process, thereby enhancing the integrity and accuracy of the character image. However, the prior information of different language and writing systems cannot be universal, such as consonants, vowels and consonants in Korean, radicals or strokes in Chinese. Therefore, it takes a lot of time to collect manually, which limits the ability of cross-language font generation under zero or small sample conditions.

[0004] Another small sample font generation method in the prior art is different from directly using character priors. It uses network priors to achieve font generation. This type of method mainly captures implicit prior information of characters by pre-training a network. The introduction of a pre-trained model can improve the learning ability of the model. However, for cross-language font generation needs, since prior information often differs, the pre-trained model trained with a small number of samples of a language as a training set needs to be retrained when applied to other languages, thereby reducing the generation efficiency. Although this type of method does not require the use of font prior information, the training process is more complicated, and when generating cross-language fonts, the original pre-trained model may be difficult to meet the new language requirements and needs to be re-pretrained.

[0005] Therefore, both types of small-sample font generation methods in the prior art have the problem that the font prior information used is different in different languages ​​and characters, and therefore cannot meet the needs of cross-language font generation. It is often necessary to re-extract the corresponding prior information for training based on the new language and character. Otherwise, the effect of cross-language font generation is very poor. Re-collecting prior information for training will result in low training and generation efficiency, and will affect the generation effect of the previous language and character. Summary of the invention

[0006] The purpose of the present invention is to provide a cross-language font generation method based on style contrast learning and relative position perception, which is used to solve the technical problem that the font generation method with a small number of samples in the prior art generates fonts based on language-specific prior information, which leads to poor cross-language font generation effect due to the different prior information of different languages, and the character integrity and style realism cannot meet the requirements.

[0007] The cross-language font generation method based on style contrast learning and relative position perception includes: establishing a new cross-language font generation model, the cross-language font generation model includes a content-style decoupling network, an image block contrast learning module and a relative position perception module; the cross-language font generation model extracts content features from an input content character image through a content encoder, and extracts style features from a reference character image providing a target style through a style encoder, while the relative position perception module divides the content character image into several parts, and then inputs these parts into the encoder to obtain corresponding local features, wherein the local features include both local content information and corresponding relative position information, and the relative position information includes local content information and relative position information. The position information is obtained through a relative position encoder, and then these local features are combined according to the original division method to generate local content features; then the content features, style features and local content features are spliced ​​and sent to the decoder for decoding to obtain the generated character image; the cross-language font generation model also uses an image block comparison learning module to compare and learn the local structure of the strokes at the image block level to further improve the style of the character stroke level; the cross-language font generation model also uses a content discriminator and a style discriminator to respectively discriminate the content features and style features of the generated font; after training and optimizing the cross-language font generation model, it is directly applied to cross-language font generation, and the decoder outputs the result.

[0008] Preferably, the image block contrast learning module first inputs the generated character image and the target character image into corresponding style encoders, and the two style encoders share parameters with the style encoder in the content-style decoupling network; LThe output of the layer is subjected to contrastive learning, and the features of the output of each layer are divided into several image blocks, each of which contains a complete local structure of the strokes. The features of each image block of the generated character image are input into the two-layer multi-layer perceptron to obtain the image block-level style features of the generated character image. The features of each image block of the target character image are input into the two-layer multi-layer perceptron to obtain the image block-level style features of the target character image. During contrastive learning, the current image block is the image block-level style feature of the target character image, and the corresponding image block-level style features of the generated character image are used as positive samples, while the remaining image block-level style features of the generated character image are used as negative samples. The contrast loss corresponding to the current image block is calculated based on these positive and negative samples, and the total contrast loss is obtained by scanning all image blocks, thereby realizing the optimization of the cross-language font generation model.

[0009] Preferably, in the content-style decoupling network, the content encoder uses a content encoder of a small sample unsupervised image conversion model, and the first four layers of the content encoder are convolution blocks; each convolution block uses a reflection padding to add pixels to the image boundary to retain the stroke details and edge details of the characters, and instance normalization is used to reduce the differences in data within each sample. The ReLU activation function is used to assist the network in learning various nonlinear features of the character image, and then the information is fused through a convolution layer; two residual blocks are added to the last two layers of the content encoder.

[0010] Preferably, in the content-style decoupling network, the reflection padding in the convolution block of the class encoder of the small sample unsupervised image conversion model is replaced with zero padding to form a style encoder; six residual blocks are used in the decoder to fully fuse the features, and transposed convolution is used to upsample the image; the discriminator structure adopts the discriminator of PatchGAN.

[0011] Preferably, the losses used in the cross-language font generation model include style contrast loss, adversarial loss and L1 loss at the image block level, and the total loss The loss function is as follows: , in, is a hyperparameter, represents the loss of the content generator, represents the loss of the content discriminator, represents the loss of the style generator, represents the loss of the style discriminator, represents the L1 loss of the model, Represents the style contrast loss at the image patch level.

[0012] Preferably, the style contrast loss at the image block level It is used to implement style contrast learning at the image block level, and the loss function is: , in, represents the generated character image, y c represents the target character image, L represents the first L layers of the style encoder, W l Indicates the number of image blocks in the current layer, represents the contrast loss, The first l Tier w The block-level style features of the image, Represents the generated character image l Tier w The block-level style features of the image, Indicates that the generated character image is l Tier w The image block-level style features of all blocks outside the block, Indicates that the sample data and y c The expected value under the distribution.

[0013] Preferably, the L1 loss function is used to constrain the pixel-level difference between the generated character and the target character to ensure that the two have a high degree of consistency at the pixel level. The corresponding loss function is: , represents the generated character image, y c represents the target character image, Indicates that the sample data and y c The expected value under the distribution.

[0014] The present invention has the following advantages: the prior information used by the method to improve the training efficiency is the local information of the font image and the image block used for contrastive learning, and the two aspects have good integration; and both do not rely on specific structural components (such as radicals) based on the type of text, so after training, it has a good cross-language font generation effect. At the same time, the method improves the generation effect of the local structural content of the text image and the local style of the strokes from the two aspects of the composition of the generation model and the contrastive learning module used for training. The former introduces relative position information to the content structure of the text to guide the correspondence between the local information and the spatial position of the character, avoiding many problems existing in the existing methods, such as: the content of the generated characters is missing, dislocated, and the spatial position does not match; the latter is combined with the style features collected from the reference character image, so that the training process simultaneously includes the style features at the image block level and the multi-level style features. The style features at the image block level can more accurately capture the fine-grained style, especially the style transfer achieved for the local structure of the stroke, and the use of multi-level style features can obtain a richer style representation, so the method can learn more detailed and comprehensive style differences in the contrastive learning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 The present invention is a flow chart of a cross-language font generation method based on style contrast learning and relative position perception.

[0016] Figure 2 It is a schematic diagram of the content encoder, style encoder, decoder and relative position encoder in the present invention.

[0017] Figure 3 This is a schematic diagram of the generation effect of the present invention in cross-language font generation. Figure 3 The left side shows some invisible font styles (font styles that do not appear in the training set), generated Chinese characters, and cross-language font generation results represented by Korean and Japanese; Figure 3 The middle (a) shows the transformation relationship of the style representation of image patches in cross-language font generation; Figure 3 (b) on the right shows a schematic diagram of the correspondence between the local content information and the relative spatial position of the same character in different fonts. DETAILED DESCRIPTION

[0018] The specific implementation modes of the present invention are further explained in detail below by describing the embodiments with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0019] like Figure 1-Figure 3 As shown, the present invention provides a cross-language font generation method based on style contrast learning and relative position perception, including the following steps.

[0020] A new cross-language font generation model (RP-Font, R represents the relative position perception module, P represents the image block contrast learning module) is established. The new font generation model is based on image block style contrast learning and relative position perception technology.

[0021] The new cross-language font generation model includes a style-content decoupled network, a patch-level style contrastive learning module (PSCL module), and a relative position awareness module (RPA module).

[0022] The cross-language font generation model is constructed by using a content encoder to generate a character image from the input content. x c Content features f c , and a style encoder is used to select a few reference character images with the target style Extracting style features At the same time, the relative position awareness (RPA) module is used to learn the correspondence between the local features of the characters and their spatial positions, thereby obtaining local content features. All the extracted features are then sent to the decoder (Mixer) to guide the position alignment of the fonts and to supplement the content features. That is, the content features, style features, and local content features are spliced ​​and sent to the decoder for decoding to obtain the generated character image. In order to further improve the style accuracy of the generated characters, the model also uses the image patch contrastive learning (PSCL) module to further improve the style of the character stroke level. In addition, the content-style decoupling network of the model is built based on the architecture of the generative adversarial network, and also includes a content discriminator and a style discriminator. Therefore, the model can also use the content discriminator and style discriminator to discriminate the content features and style features of the generated fonts respectively.

[0023] In the content-style decoupling network, the content encoder uses the content encoder of the small sample unsupervised image conversion model (FUNIT), and the first four layers of the content encoder are convolution blocks. Each convolution block uses a reflection padding to add pixels to the image boundary to retain the stroke details and edge details of the characters, uses instance normalization to reduce the differences in the data within each sample, and uses the ReLU activation function to assist the network in learning various nonlinear features of the character image, such as stroke thickness, curvature, etc., and then fuses the information through a convolution layer. In order to prevent gradient disappearance and information loss, two residual blocks are added to the last two layers of the content encoder.

[0024] The content-style decoupling network makes some modifications to the class encoder of the small sample unsupervised image conversion model (FUNIT) to form a style encoder, that is, using zero padding to replace the reflection padding in the convolution block of the class encoder, which can avoid introducing too much noise and extract a purer style.

[0025] In order to make full use of content features and style features, six residual blocks are used in the decoder to fully fuse the features, and transposed convolution is used to upsample the image.

[0026] The structure of the discriminator (including content discriminator and style discriminator) adopts the discriminator of PatchGAN. The PatchGAN network can consider the influence of different parts of the image instead of just providing an overall evaluation of the entire image, which is particularly important for font generation tasks.

[0027] In most languages, the relative position of the same character part in different fonts is generally unchanged, so the learning of relative position reduces the complexity compared to the learning of the whole character. This method uses this relative position information to guide the acquisition of the correspondence between the local information and the spatial position of the character. In order to capture this information, a relative position awareness (RPA) module is set in the cross-language font generation model. The relative position awareness module divides the content character image into several parts and captures the relative position information therefrom. The relative position information is then used to guide the alignment of the positions of the various parts in the character image, and the local features of each part are used to supplement the content features.

[0028] In the relative position perception module, the content character image x c Divided into four parts: upper left , upper right , lower left and lower right , and then input these four parts into the encoder to obtain the corresponding local features. The local features of these four parts include both local content information and corresponding relative position information. The relative position information is obtained through the relative position encoder, and then these local features are combined according to the original division method to generate local content features. f lc Local content features f lc It can not only enhance content features, but also incorporate relative position information into the generation process, providing important guidance for the alignment of local parts of characters. Since this local content information is not a priori information unique to various languages ​​and characters, it is universal and can be applied to different languages ​​and characters.

[0029] Although the relative position perception module provides local content information and relative position information, the division process is unrelated to the stroke information, so there are incomplete strokes in the local content information, especially the destruction of the stroke line trajectory, which will cause inaccurate deformation of the stroke style during style transfer. Therefore, in order to further improve the style accuracy of the generated characters, the cross-language font generation model further improves the style of the character stroke level through the image block contrast learning (PSCL) module. Since the font style is usually reflected by some local stroke structures of the characters, the image block contrast learning module performs contrast learning at the image block level instead of contrast learning at the global character level.

[0030] The image patch contrast learning module first generates the character image and the target character image y c Input two corresponding style encoders (generated style encoder and target style encoder), which share parameters with the style encoder in the content-style decoupling network. L The output of the layer is compared and learned, and the style encoder is l The features of the layer output are divided into W l Each image block contains a complete stroke local structure, that is, the stroke local trajectory is not destroyed by the edge of the image block. No. w The features of the image blocks are input into a two-layer multi-layer perceptron N mlp Get the image block-level style features of the generated character image , follow similar operations to transform the target character image y c No. w The features of the image blocks are input into the multi-layer perceptron Nmlp Get the image block level style features of the target character image , and the image block-level style features of the generated character image correspond.

[0031] During contrastive learning, the current image block is the image block-level style feature of the target character image. , the image block-level style features of the corresponding generated character image As positive samples, the remaining image block-level style features of the generated character images The contrast loss corresponding to the current image block is calculated based on these positive and negative samples, and the total contrast loss is obtained by scanning all image blocks, thereby optimizing the cross-language font generation model. Through the style representation at the image block level, the image block contrast learning module can learn the unique style representation of a given font. Since the style features used in this process are based on the complete style information of the local stroke trajectory, they are not affected by the unique stroke structure in different languages ​​and can be well generalized to different language systems, thereby achieving cross-language font generation performance with highly consistent style.

[0032] In order to generate high-quality results in the cross-language font generation task while ensuring the content integrity of the generated characters and the accuracy of the font style, the loss functions used in this method during the training process include: style contrast loss at the image block level, which is used to guide the model to capture the style features at the image block level; content adversarial loss and style adversarial loss, which are used to optimize the model in the character content-style decoupling task, thereby effectively distinguishing and retaining the content features of the characters while also ensuring the accuracy of the style features; L1 loss function, which is used to constrain the pixel-level differences between the generated characters and the target characters, ensuring that the two have a high degree of consistency at the pixel level.

[0033] Patch-level style contrast loss , which is used to implement style contrast learning at the image block level to obtain a more fine-grained symbolic style representation. Style contrast loss at the image block level The loss function is: , in, represents the generated character image, y c represents the target character image, L represents the first L layers of the style encoder, W l Indicates the number of image blocks in the current layer, represents the contrast loss, The first l Tier w The block-level style features of the image, Represents the generated character image l Tier w The block-level style features of the image, Indicates that the generated character image is l Tier w The image block-level style features of all blocks outside the block, Indicates that the sample data and y c The expected value under the distribution.

[0034] The adversarial loss includes: the loss function of the content discriminator, which focuses on distinguishing the difference between the generated content and the real content; the loss function of the style discriminator, which focuses on distinguishing the difference between the generated style and the real style. The loss function of the content adversarial loss is as follows: , , in, x c is the input content image, represents the generated character image, y c represents the target character image, D c represents the content discriminator, represents the loss of the content generator, Indicates that the sample data and x c The expected value under the distribution, represents the loss of the content discriminator, Indicates that the sample data y c and x c The expected value under the distribution.

[0035] Similar to the content adversarial loss, the loss function of the content adversarial loss is as follows: , , in, x s is the input reference character image, used to provide style information, represents the generated character image, y c represents the target character image, D s represents the style discriminator, represents the loss of the style generator, Indicates that the sample data and x s The expected value under the distribution, represents the loss of the style discriminator, Indicates that the sample data y c and x s The expected value under the distribution.

[0036] L1 loss, which measures the difference between the generated image and the target image at the pixel level, and the corresponding loss function is: , Represents the L1 loss of the model.

[0037] According to the above loss function, the losses used in the cross-language font generation model of this method during the training phase include style contrast loss, adversarial loss and L1 loss at the image block level, and the total loss The loss function is as follows: , in, It is a hyperparameter. In subsequent experiments, its value can be set to 1 by default.

[0038] Based on the above model structure and loss function, the model is trained and optimized through a small number of reference character images. The cross-language font generation model obtained after training can be directly applied to cross-language font generation, and the results are output by the decoder.

[0039] Figure 3 The left side shows some invisible font styles (font styles that do not appear in the training set), generated Chinese characters, and cross-language font generation results represented by Korean and Japanese; Figure 3 The middle (a) shows the transformation relationship of the style representation of image patches in cross-language font generation; Figure 3 (b) on the right shows a schematic diagram of the correspondence between the local content information and the spatial relative position of the same character in different fonts, which reflects the role and effect of the image block contrast learning module and the relative position perception module in cross-language font generation.

[0040] The present invention is described above by way of example in conjunction with the accompanying drawings. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned method. As long as various non-substantial improvements are made using the inventive concept and technical solution of the present invention, or the inventive concept and technical solution are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A cross-language font generation method based on style contrast learning and relative position perception, characterized by: include: A new cross-language font generation model is established, which includes a content-style decoupling network, an image block contrast learning module and a relative position perception module; the cross-language font generation model extracts content features from an input content character image through a content encoder, and extracts style features from a reference character image providing a target style through a style encoder, and at the same time, the relative position perception module divides the content character image into several parts, and then inputs these parts into the encoder to obtain corresponding local features, the local features include both local content information and corresponding relative position information, the relative position information is obtained through the relative position encoder, and then these local features are combined according to the original division method to generate local content features; then the content features, style features and local content features are spliced ​​and sent to a decoder for decoding to obtain a generated character image; the cross-language font generation model also compares and learns the local structure of strokes at the image block level through the image block contrast learning module to further improve the style of the character stroke level; the cross-language font generation model also discriminates the content features and style features of the generated font through a content discriminator and a style discriminator respectively; After training and optimization, the cross-language font generation model is directly applied to cross-language font generation, and the decoder outputs the result.

2. The cross-language font generation method based on style contrast learning and relative position perception according to claim 1, characterized in that: The image patch contrast learning module first inputs the generated character image and the target character image into the corresponding style encoder. These two style encoders share parameters with the style encoder in the content-style decoupling network. L The output of the layer is subjected to contrastive learning, and the features of the output of each layer are divided into several image blocks, each of which contains a complete local structure of the strokes. The features of each image block of the generated character image are input into the two-layer multi-layer perceptron to obtain the image block-level style features of the generated character image. The features of each image block of the target character image are input into the two-layer multi-layer perceptron to obtain the image block-level style features of the target character image. During contrastive learning, the current image block is the image block-level style feature of the target character image, and the corresponding image block-level style features of the generated character image are used as positive samples, while the remaining image block-level style features of the generated character image are used as negative samples. The contrast loss corresponding to the current image block is calculated based on these positive and negative samples, and the total contrast loss is obtained by scanning all image blocks, thereby realizing the optimization of the cross-language font generation model.

3. The cross-language font generation method based on style contrast learning and relative position perception according to claim 1, characterized in that: In the content-style decoupling network, the content encoder uses the content encoder of the small sample unsupervised image conversion model. The first four layers of the content encoder are convolution blocks. Each convolution block uses a reflection padding to add pixels to the image boundary to retain the stroke details and edge details of the characters. Instance normalization is used to reduce the difference in data within each sample. The ReLU activation function is used to assist the network in learning various nonlinear features of the character image, and then the information is fused through a convolution layer. Two residual blocks are added to the last two layers of the content encoder.

4. The cross-language font generation method based on style contrast learning and relative position perception according to claim 1, characterized in that: In the content-style decoupling network, the reflection padding in the convolutional block of the class encoder of the small-sample unsupervised image conversion model is replaced with zero padding to form a style encoder; six residual blocks are used in the decoder to fully fuse the features, and transposed convolution is used to upsample the image; The discriminator structure adopts the discriminator of PatchGAN.

5. The cross-language font generation method based on style contrast learning and relative position perception according to claim 1, characterized in that: The losses used in the cross-language font generation model include style contrast loss, adversarial loss, L1 loss, and total loss at the image block level. The loss function is as follows: ， in, is a hyperparameter, represents the loss of the content generator, represents the loss of the content discriminator, represents the loss of the style generator, represents the loss of the style discriminator, represents the L1 loss of the model, Represents the style contrast loss at the image patch level.

6. The cross-language font generation method based on style contrast learning and relative position perception according to claim 5, characterized in that: Patch-level style contrast loss It is used to implement style contrast learning at the image block level, and the loss function is: ， in, represents the generated character image, y c represents the target character image, L represents the first L layers of the style encoder, W l Indicates the number of image blocks in the current layer, represents the contrast loss, The first l Tier w The block-level style features of the image, Represents the generated character image l Tier w The block-level style features of the image, Indicates that the generated character image is l Tier w The image block-level style features of all blocks outside the block, Indicates that the sample data and y c The expected value under the distribution.

7. The cross-language font generation method based on style contrast learning and relative position perception according to claim 5, characterized in that: The L1 loss function is used to constrain the pixel-level difference between the generated characters and the target characters to ensure that the two have a high degree of consistency at the pixel level. The corresponding loss function is: , represents the generated character image, y c represents the target character image, Indicates that the sample data and y c The expected value under the distribution.

Images