Writing brush calligraphy vector font bimodal fusion method based on texture variational auto-encoder

Through a dual-modal fusion network model based on texture variational autoencoder, combined with Transformer and CNN, the problems of poor font quality and difficult texture information retention in the existing technology are solved, and high-quality and realistic details are generated. Brush calligraphy fonts realize efficient representation of vector fonts and retain texture features.

CN118942100APending Publication Date: 2024-11-12XIAN UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411072497.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

It is difficult for the prior art to generate high-quality vector brush calligraphy fonts, and the existing vectorized models cannot effectively retain the texture information and continuous brush characteristics of brush calligraphy, resulting in poor quality of vector fonts and complex data representation.

Method used

Using a dual-mode fusion network model based on texture variational autoencoder, the encoding and decoding networks process the glyph image and sequence information respectively, and generate high-quality vector brush calligraphy fonts, using Deep learning structures such as Transformer and CNN are performed for feature extraction and fusion.

Benefits of technology

The generated vector brush calligraphy font can retain texture features and detailed information, ensure that the font macro structure is complete, the details are realistic, and the resolution can be adjusted arbitrarily, solving the problems of poor quality of vector fonts and complex data representation in the prior art.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942100A_ABST
    Figure CN118942100A_ABST
Patent Text Reader

Abstract

The invention discloses a brush calligraphy vector font bimodal fusion method based on a texture variational auto-encoder, and the method specifically comprises the steps: constructing a vector Chinese font data set, carrying out the preprocessing of Chinese fonts in the data set, and dividing the Chinese fonts into a training set and a test set; constructing a bimodal fusion network model based on a texture variational auto-encoder; the bimodal fusion network model comprises a texture generation module and a bimodal generation module; processing font images in the training set by using a bimodal fusion network model; constructing a loss function, and training the bimodal fusion network model; inputting the font images of the test set into the bimodal fusion network model for testing, and finally outputting the brush calligraphy vector fonts. According to the invention, the TVAE-DMF network model combines the advantages of a variational auto-encoder and bimodal fusion, local detail features of the writing brush calligraphy fonts can be learned in a generation result, and then the vector writing brush calligraphy fonts with texture features are obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a dual-modal fusion method of brush calligraphy vector fonts based on texture variational autoencoder. Background Art

[0002] There are two commonly used computer font description forms: bitmap and vector. To achieve the same effect as a vector font, the bitmap format requires the modification of many pixel colors and positions. Bitmaps not only take up a lot of file storage space, but also have obvious jagged outlines when enlarged. Vector fonts use an analytical representation with very few parameters, and the resolution can be scaled and expanded at will, which makes them more accurate and compact than bitmap fonts. Therefore, vector font formats are widely used in scenarios where there is a need for arbitrary scaling and transformation to display glyphs.

[0003] At present, the design of vector Chinese fonts needs to go through multiple links such as creative design, original manuscript, digital fitting, fine adjustment, vectorization, etc. There are some vector graphics editing software, such as Adobe Illustrator, CorelDRAW and FontLab, which can assist designers in making vector fonts, but they require more method definitions and manual fine-tuning, and the whole process is still cumbersome and time-consuming. Therefore, the study of brush calligraphy font vector methods based on deep learning has a good application prospect.

[0004] In recent years, the vigorous development of deep learning has brought new opportunities for Chinese font design, especially in image style transfer and stroke splitting and reorganization. However, it is impossible to directly generate vector fonts, and the bitmap-based font vectorization model will substitute its defects at the bitmap level into the vector result, which cannot be extended to practical application scenarios. Since the texture information contained in the brush calligraphy font is difficult to reproduce in the vector process, and its connected stroke characteristics are also difficult to express in the vector font generation task. Therefore, it is of great significance to analyze and optimize the existing vector font data format to design a vector font generation model that can retain the unique texture information of brush calligraphy. Summary of the invention

[0005] The purpose of the present invention is to provide a dual-modal fusion method for brush calligraphy vector fonts based on texture variational autoencoder, which solves the problems of poor quality of generated vector glyphs and complex data representation of vector Chinese fonts in the prior art.

[0006] The technical solution adopted by the present invention is a dual-modal fusion method of brush calligraphy vector fonts based on texture variational autoencoder, which is specifically implemented according to the following steps:

[0007] Step 1, construct a vector Chinese font dataset, pre-process the Chinese fonts in the dataset, and divide it into a training set and a test set;

[0008] Step 2, constructing a bimodal fusion network model based on texture variational autoencoder; the bimodal fusion network model includes a texture generation module and a bimodal generation module; using the bimodal fusion network model to process the glyph images in the training set;

[0009] Step 3: construct a loss function and train the bimodal fusion network model;

[0010] Step 4: Input the glyph images of the test set into the bimodal fusion network model for testing, and finally output the brush calligraphy vector font.

[0011] The present invention is also characterized in that:

[0012] In step 1, specifically:

[0013] Step 1.1, obtain the TTF format file of the Chinese font, use the FontForge tool combined with the Python script to read and traverse the TTF format file of the font from the character set file;

[0014] Step 1.2, convert the font library in TTF format into SFD format using a multi-process parallel processing script;

[0015] Step 1.3, extracting the glyph data of each character from the SFD format file, which includes the outline and shape of the data, and converting the extracted glyph data into a glyph image form;

[0016] Step 1.4, data enhancement is performed on the generated glyph image by rotation, scaling, and shearing;

[0017] Step 1.5, performing a relaxation representation on the character sequence in the glyph image processed in step 1.4 and calculating auxiliary Bezier points;

[0018] Step 1.6, constrain the bitmap format of the glyph image obtained in step 1.5 to a 128*128 viewing frame; based on the 128*128 viewing frame, implement a data enhancement strategy through random factors; and divide all glyph image data sets into training sets and test sets according to a 7:3 ratio.

[0019] In step 2, the texture generation module consists of an encoding network Encoder and a decoding network Decoder; the structure of the encoding network contains three fully connected layers, each of which is followed by a ReLU activation function; the bimodal generation module includes a TVAE-SEQ module and a TVAE-PIX module; among them, the encoder and decoder in the TVAE-PIX module are respectively composed of CNN and deconvolutional neural networks to output image features and generate target glyph images, and the TVAE-SEQ module is a sequence-to-sequence codec, which is a codec composed of a four-layer and six-layer Transformer encoder block.

[0020] In step 2, specifically:

[0021] Step 2.1, use the encoding network and decoding network in the texture generation module to perform dimension conversion on the glyph images in the training set; specifically:

[0022] Step 2.11, randomly crop the glyph images in the training set in step 1 to generate 10,000 25pix*25pix real texture blocks as input, and store them in Numpy data format;

[0023] Step 2.12: The input 10,000 25*25 real texture blocks are compressed through the encoder and converted into two 30-dimensional vectors μ and σ. 2 , respectively as the mean and variance of the distribution of the latent variable z;

[0024] Step 2.13, by randomly sampling ε and the vectors μ and σ obtained in the previous step 2 By performing addition and multiplication operations, we can obtain the hidden variable z corresponding to each real texture sample, which is the low-dimensional representation of the real calligraphy texture sample.

[0025] Step 2.14, the structure of the decoding network is opposite to that of the encoding network. It takes the latent variable z output by the encoder as input, and through three reverse neural networks, supplements the secondary information of the sample missing in the latent variable z and converts it into a new sample with the same dimension as the real calligraphy texture sample;

[0026] Step 2.2, the dual-modal generation module includes a TVAE-SEQ module and a TVAE-PIX module; the TVAE-PIX module and the TVAE-SEQ module encode the spatial shape information of the glyph image and the sequence information of the vector glyph respectively, and obtain the sequence and image fusion features of the vector glyph through the linear projection layer; specifically:

[0027] Step 2.21: Convert each glyph information in the glyph image after step 2.1 into a vector form G i , then each vector glyph G is obtained i , and sequence them As the input of the sequence-to-sequence encoder, YesN c The output of the Transformer sequence encoder is represented as d E Indicates the embedding dimension, set d E =256;

[0028] Compute global sequence-wise style features in (j=0, 1, ..., N c ) by linear projection from N r Aggregate from the jth token of the reference glyphs;

[0029] Set the input of the sequence decoder to The original modal markup is replaced by the fused feature f; an MLP layer is attached on top of the Transformer decoder to predict the command type and coordinates of the target glyph, denoted as and

[0030] Step 2.22: TVAE-PIX module: It consists of a CNN consisting of six convolutional layers and two fully connected layers. The convolutional layers use ReLU as the activation function, and the fully connected layers use the tanh function. The CNN outputs f img , and get the new latent vector parameters μ1 and By sampling random noise ε from a standard normal distribution, and then using the new latent vector parameters μ1 and variance To reparameterize, finally get the updated random latent vector z1;

[0031] Step 2.23: Obtain vector sequence data using the encoder in the TVAE-SEQ module and TVAE-PIX module to obtain the corresponding glyph bitmap data img The bimodal data of the input characters are concatenated and then formed into a new fused feature vector f through a multi-layer perceptron. The fused feature f' after the multi-layer perceptron MLP and concatenation operation is used as the input in the subsequent steps. The feature f' is input into the image decoder and the target glyph image I is generated through a deconvolutional neural network. t .

[0032] In step 3, the loss function L of the texture generation module and the bimodal generation module is included in the training process of the bimodal fusion model based on texture variational autoencoder. Tex and L DMF The loss function Lloss ; By optimizing the loss function, the model iterates to minimize the difference between the vector sequence in the decoder and the input vector glyph, as shown in the following formula:

[0033] L loss =L DMF +L Tex .

[0034] In step 3, specifically:

[0035] Step 3.1, loss function L of texture generation module Tex It consists of two parts, including the reconstruction loss function and the KL divergence part, as shown in the following formula:

[0036]

[0037] Among them, x represents the original sample, represents the synthetic sample, and n represents the number of samples.

[0038] KL divergence is used to ensure that the decoding network learns the latent space variables that satisfy the prior distribution, as shown below:

[0039]

[0040] The Adam stochastic gradient descent algorithm is used to obtain the optimal network parameters. The basic learning rate is set to 0.0001, the learning rate decay coefficient is set to 0.5, the training loss is displayed every 100 iterations, and the model is kept every 10,000 iterations. The maximum number of iterations is set to 500,000.

[0041] Step 3.2, TVAE-SEQ module: When predicting the command type and coordinates of the target glyph in the Transformer decoder, the initial prediction loss between the initial generated glyph and its corresponding GroundTruth is defined as follows:

[0042]

[0043] in, represents the cross entropy loss, all coordinates (p) are quantized to optimize w cmd represents the loss weight for command type prediction, and all unused parameters in the second term are masked;

[0044] Minimize Command C i-1 The predicted endpoint of the command C i The L2 distance between the starting points of is defined as the relaxed representation of the merge loss, as shown in the following formula:

[0045]

[0046] in, and Represent the starting point and the ending point respectively, N c Indicates the command number;

[0047] TVAE-PIX module: Input feature f into the image decoder to generate the target glyph image I t Then use L1 loss and perceptual loss to calculate image reconstruction loss, L1 loss, perceptual loss and image reconstruction loss L img They are shown as follows:

[0048]

[0049] Where n is the number of samples, y i is the true value of the i-th sample, is the predicted value of the i-th sample;

[0050]

[0051] Where x is the input image, y is the target image, and F i (x) and F i (y) respectively represent the feature representation of the i-th layer in the pre-trained neural network;

[0052]

[0053] in, is the predicted value of the target glyph;

[0054] The Adam stochastic gradient descent algorithm is used to obtain the optimal network parameters. The initial learning rate is set to 0.0001, and the learning rate is gradually reduced at a decay rate of 0.9999 at each step until the minimum learning rate of 0.00001 is reached;

[0055] Step 3.3, the remaining loss functions;

[0056] L KL represents the KL loss normalized on the latent code f, L Bezier is the loss function used to constrain the auxiliary points, so from each drawing command C t,j More points are sampled in the experiment, where the Bezier curve based on the control point p is defined as follows:

[0057]

[0058] Where m represents the order of the curve, represents the binomial coefficient, and r is the position parameter; therefore, the alignment loss L of the Bezier curve is calculated by the following formula Bezier :

[0059]

[0060] Among them, r p ∈{0.25, 0.5, 0.75} represents the parameters of the sampling auxiliary points, B t,j (r) represents the point of the Bezier curve drawing command;

[0061] By minimizing I loss To train the entire bimodal generation module, as shown below;

[0062]

[0063] The beneficial effects of the present invention are as follows: the TVAE-PIX module and the TVAE-SEQ module in the brush calligraphy vector font generation model integrating bimodal information of the present invention are responsible for encoding the sequence mode and spatial shape information of the vector glyphs, respectively. Unlike the previous vector graphics generation model, the model effectively integrates the bimodal information of the vector glyphs, and generates textures by combining variational autoencoders to render the generated vector fonts to present the artistic features of textures in calligraphy art, thereby generating vector brush calligraphy fonts. In addition, after analyzing and comparing the results of the Chinese font generation method in the prior art, the proposed TVAE-DMF model can better capture the glyph features and sequence features of the font, generate vector format calligraphy fonts with arbitrarily adjustable resolution, and after rendering the texture samples generated by the texture generation model, a brush calligraphy vector font can be generated to ensure that the font macro structure is complete and the details are more realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a flow chart of preprocessing of vector Chinese font data of the present invention;

[0065] Figure 2 It is a structural diagram of the TextureVAE-DMF network model of the present invention;

[0066] Figure 3 It is a structural diagram of the variational autoencoder of the present invention;

[0067] Figure 4 is a diagram of the reparameterization transformation process of the present invention;

[0068] Figure 5 is a diagram of the training process of the texture generation module of the present invention;

[0069] Figure 6 This is the visualization result of the ablation experiment;

[0070] Figure 7 These are the renderings of vector brush calligraphy fonts generated by different models;

[0071] Figure 8 This is a vector brush calligraphy font texture rendering effect diagram (I);

[0072] Fig. 9 This is the vector brush calligraphy font texture rendering effect picture (2). DETAILED DESCRIPTION

[0073] The present invention is described in detail below in conjunction with specific implementation modes and accompanying drawings.

[0074] Example 1

[0075] The present invention is based on a bimodal fusion method of brush calligraphy vector fonts based on texture variational autoencoders. Combining variational autoencoders and bimodal fusion, a bimodal fusion network model (TextureVAE-DMF) based on texture variational autoencoders is proposed. The model adopts the architecture of variational autoencoders to encode the texture information unique to brush calligraphy characters. In addition, the model adopts a dual-branch architecture, and uses CNN and Transformer to output image and sequence features respectively through image encoders and sequence encoders, and optimizes the spatial structure and sequence information of the font through bimodal information. Finally, the generated texture samples are rendered with the generated high-quality vector brush calligraphy fonts to obtain vector brush calligraphy fonts with texture features. Through ablation experiments and comparative experiments, the generation effect of TextureVAE-DMF is investigated, which proves that the method of the present invention has excellent generation effects in detail fidelity and structural integrity of brush calligraphy vector fonts.

[0076] Example 2

[0077] The present invention is based on the brush calligraphy vector font dual-modal fusion method of texture variational autoencoder, which is specifically implemented according to the following steps:

[0078] Step 1: construct a vector Chinese font dataset and preprocess the Chinese fonts in the dataset; Figure 1 As shown, specifically:

[0079] The vector Chinese font dataset consists of 2343 Chinese fonts in the Chinese Standard Simplified Character Set (GB2312-80);

[0080] Step 1.1, obtain the TTF format file of the Chinese font, use the FontForge tool combined with the Python script to read and traverse the TTF format file of the font from the character set file;

[0081] Step 1.2, convert the font library in TTF format into SFD (Spline Font Database) format using a multi-process parallel processing script;

[0082] Step 1.3, extracting the glyph data of each character from the SFD format file, which includes the outline and shape of the data, and converting the extracted glyph data into a glyph image form;

[0083] Step 1.4: In order to improve the generalization ability of the model, the generated glyph image is enhanced by rotating, scaling, and shearing;

[0084] Step 1.5, performing a relaxation representation on the character sequence in the glyph image processed in step 1.4 and calculating auxiliary Bezier points, so as to more accurately describe and process the curves and contours of the glyph;

[0085] Step 1.6, constrain the bitmap format of the glyph image obtained in step 1.5 to a 128*128 viewing frame; based on the 128*128 viewing frame, implement a data enhancement strategy through random factors to enhance the generalization of the data set; divide all glyph image data sets into a training set and a test set according to a ratio of 7:3;

[0086] Step 2: Construct a bimodal fusion network model (TextureVAE-DMF) based on texture variational autoencoder; Figure 2 As shown, the TextureVAE-DMF network model includes a texture generation module and a bimodal generation module;

[0087] The texture generation module consists of an encoding network Encoder and a decoding network Decoder; the structure of the encoding network (Encoder) contains three fully connected layers, each of which is followed by a ReLU activation function.

[0088] The bimodal generation module includes the TVAE-SEQ module and the TVAE-PIX module. The encoder and decoder in the TVAE-PIX module are composed of CNN and deconvolutional neural networks, respectively, to output image features and generate target glyph images. The TVAE-SEQ module is a sequence-to-sequence encoder-decoder composed of a four-layer and six-layer Transformer encoder block.

[0089] The specific process is as follows:

[0090] Step 2.1, use the encoding network and decoding network in the texture generation module to perform dimension conversion on the glyph images in the training set; specifically:

[0091] Step 2.11, randomly crop the glyph images in the training set in step 1 to generate 10,000 25pix*25pix real texture blocks as input, and store them in Numpy data format;

[0092] Step 2.12: The input 10,000 25*25 real texture blocks are compressed through the encoder and converted into two 30-dimensional vectors μ and σ. 2 , respectively as the mean and variance of the distribution of the latent variable z;

[0093] Step 2.13, by randomly sampling ε and multiplying it with the mean and variance obtained in the previous step, the latent variable z corresponding to each real texture sample can be obtained, that is, the low-dimensional representation of the real calligraphy texture sample;

[0094] The role of the encoding network is to reduce the dimensionality of high-dimensional data, which can be understood as compressing the data and extracting the main information of the texture sample from it.

[0095] Step 2.14, the structure of the decoding network (Decoder) is exactly the opposite of the encoding network. It takes the latent variable z output by the encoder as input, and through three reverse neural networks, supplements the secondary information of the sample missing in the latent variable z and converts it into a new sample with the same dimension as the real calligraphy texture sample;

[0096] like Figure 5 As shown in the figure, the training process of the texture generation module is shown, in which the structure of the encoding network (Encoder) contains three fully connected layers, each layer is followed by a ReLU activation function, and the 10,000 25*25 real texture blocks of the input are compressed by the encoder and converted into two 30-dimensional vectors μ and σ, which are used as the mean and variance of the distribution of the latent variable z respectively. By randomly sampling ε and then performing addition and multiplication operations, the latent variable z corresponding to each real texture sample can be obtained, that is, the low-dimensional representation of the real calligraphy texture sample.

[0097] Step 2.2, the dual-modal generation module includes a TVAE-SEQ module and a TVAE-PIX module; the TVAE-PIX module and the TVAE-SEQ module encode the spatial shape information of the glyph image and the sequence information of the vector glyph respectively, and obtain the sequence and image fusion features of the vector glyph through the linear projection layer; specifically:

[0098] Step 2.21: The TVAE-SEQ module is a variational autoencoder composed of an encoder and decoder;

[0099] Convert each glyph information in the glyph image after step 2.1 into a vector form G i , then each vector glyph G is obtained i , and sequence them As the input of the sequence-to-sequence encoder, YesN c The embedding of drawing commands, e i,0 Represents an auxiliary learnable "tag" for performing bimodal fusion; the output of the Transformer sequence encoder is represented as d E Indicates the embedding dimension, set d E =256;

[0100] Compute global sequence-wise style features in (j=0, 1, ..., N c ) by linear projection from N r It is aggregated from the j-th token of the reference glyphs.

[0101] Set the input of the sequence decoder to The original modal markup is replaced by the fused feature f. An MLP layer is attached on top of the Transformer decoder to predict the command type and coordinates of the target glyph, denoted as and

[0102] Step 2.22: TVAE-PIX module: It consists of a CNN consisting of six convolutional layers and two fully connected layers. The convolutional layers use ReLU as the activation function, and the fully connected layers use the tanh function. Its network parameters are shown in Table 1:

[0103] Table 1 TVAE-PIX module network parameter settings

[0104]

[0105]

[0106] Output through CNN At the same time, we get the new latent vector parameters μ1 and The size of the input bitmap font is 128×128×1, the size of the convolution kernel is 2×2, and the number of convolution kernels in each layer is (4, 4, 8, 8, 8, 8,). The last layer of the encoder uses a fully connected layer to convert the input into a 128-dimensional latent feature vector to capture the spatial information of the input glyph bitmap data;

[0107] By sampling random noise ε from a standard normal distribution, and then using the new latent vector parameters μ1 and variance To reparameterize, finally get the updated random latent vector z1;

[0108] Figure 3 This is the structure diagram of the variational autoencoder, the encoder Encoder and the decoder Decoder. Its network structure is similar to that of AE, but its encoder does not directly output a hidden variable, but calculates the low-dimensional mean μ and variance σ of each input data x 2 It is used to map high-dimensional input to its potential representation. The decoder takes the potential vector as input to create a high-dimensional output, where p(z) represents the prior distribution and p(z) is the model distribution. The specific calculation process is as follows:

[0109]

[0110] Where: z is the hidden variable of the encoder; p d (z) is the original data distribution; q(z|x) is the distribution function of the encoder; q(z) is the posterior distribution of the implicit function aggregation.

[0111] like Figure 4 As shown in the figure, it is a reparameterization process, which splits the sampling process into two parts: sampling noise from a fixed distribution, and then transforming the noise to obtain the final potential representation, where μ is the mean vector of the encoder output, σ is the variance vector of the output, ⊙ represents element-by-element multiplication, and ε is a random vector sampled from a standard normal distribution. This method allows the latent variables to be sampled during training while maintaining the differentiability of the network, thereby updating the parameters through back propagation. The introduction of reparameterization enables the variational autoencoder to be trained efficiently and can be applied to various complex latent distributions.

[0112] Step 2.23: Obtain vector sequence data using the encoder in the TVAE-SEQ module and TVAE-PIX module to obtain the corresponding glyph bitmap data img The bimodal data of the input characters is concatenated and then passed through a multi-layer perceptron to form a new fused feature vector f. The fused feature f' after the MLP and concatenation operation is used as the input in the subsequent steps. MLP stands for multi-layer perceptron, which consists of multiple fully connected layers, each of which includes a linear projection layer and a nonlinear activation function. MLP can learn the complex relationship between features and generate higher-order feature representations. The feature f' is input to the image decoder and passed through a deconvolutional neural network to generate the target glyph image I t ;

[0113] Step 3: Construct a loss function and train the bimodal fusion network model (TextureVAE-DMF);

[0114] The loss function L of the texture generation module and the bimodal generation module in the training process of the bimodal fusion model based on texture variational autoencoder is Tex and L DMF The loss function L loss By optimizing the loss function, the model continuously iterates to minimize the difference between the vector sequence in the decoder and the input vector glyph. The specific function can be expressed as shown in the following formula:

[0115] L loss =L DMF +L Tex

[0116] Specifically:

[0117] Step 3.1, loss function L of texture generation module Tex It consists of two parts, including the reconstruction loss function and the KL divergence part, where the reconstruction loss is L rec It is used to measure the difference between the output generated by the model reconstruction and the real data. In this invention, it is used to calculate the similarity between the synthetic sample and the real sample. Its main function is to reduce the distance between the generated reconstructed image and the real image, making them closer. The reconstruction loss function uses the mean square error (MSE) loss, as shown in the following formula:

[0118]

[0119] Among them, x represents the original sample, represents the synthetic sample, and n represents the number of samples.

[0120] KL divergence is used to ensure that the decoding network learns the latent space variables that satisfy the prior distribution. The formula is as follows:

[0121]

[0122] Among them, μ and σ 2 are the mean and variance of the latent variable z generated by the encoder network, and N is the dimension of the latent space. KL loss encourages the model to learn a suitable latent vector by minimizing the difference between the latent space distribution and the standard normal distribution.

[0123] The Adam stochastic gradient descent algorithm is used to obtain the optimal network parameters. The basic learning rate is set to 0.0001, the learning rate decay coefficient is set to 0.5, the training loss is displayed every 100 iterations, and the model is maintained every 10,000 iterations. The maximum number of iterations is set to 500,000, and the learning strategy adopts a fixed strategy. The weight decay parameter is set to 0.0005, and the value of the control coefficient β is set to 0.9. TexThe weight values ​​in the section are set to 1.0, 1.0 from left to right.

[0124] Step 3.2, TVAE-SEQ module: When predicting the command type and coordinates of the target glyph in the Transformer decoder, the initial prediction loss between the initial generated glyph and its corresponding GroundTruth is defined as follows:

[0125]

[0126] in, represents the cross entropy loss, all coordinates (p) are quantized to optimize w cmd represents the loss weight for command type prediction. All unused parameters in the second term are masked.

[0127] Then, ideally, the drawing commands LineFromTo and CurveFromTo are abbreviated as l and c. Regardless of the original synthesized sequence or the optimized sequence, the starting point of the current drawing command should coincide with the end point of the previous drawing command. Therefore, try to minimize the command C i-1 The predicted endpoint of the command C i The L2 distance between the starting points of is defined as the relaxed representation of the merge loss, as shown in the following formula:

[0128]

[0129] in, and Represent the starting point and the ending point respectively, N c Indicates the command number.

[0130] TVAE-PIX module: Input feature f into the image decoder to generate the target glyph image I t Then use L1 loss and perceptual loss to calculate image reconstruction loss, L1 loss, perceptual loss and image reconstruction loss L img They are shown as follows:

[0131]

[0132] Where n is the number of samples, y i is the true value of the i-th sample, is the predicted value of the ith sample.

[0133]

[0134] Where x is the input image, y is the target image, and F i (x) and F i(y) represent the feature representation of the i-th layer in the pre-trained neural network, and N is the number of feature layers.

[0135]

[0136] Among them, I t is the true value of the target glyph, is the predicted value of the target glyph.

[0137] The Adam stochastic gradient descent algorithm is used to obtain the optimal network parameters. The initial learning rate is set to 0.0001, and the learning rate is gradually reduced at a decay rate of 0.9999 at each step until the minimum learning rate of 0.00001 is reached. DMF The weight values ​​of different losses in the section are set to 1.0, 1.0, 10, 1.0, and 0.01 from left to right.

[0138] Step 3.3, the remaining loss functions;

[0139] L KL represents the KL loss normalized on the latent code f, L Bezier is the loss function used to constrain the auxiliary points. If only the control point p is used to supervise the alignment of the Bezier curve and the straight line, the effect is not good enough. Therefore, from each drawing command C t,j More points are sampled in the experiment, where the Bezier curve based on the control point p is defined as follows:

[0140]

[0141] Where m represents the order of the curve, represents the binomial coefficient, and r is the position parameter; therefore, the alignment loss L of the Bezier curve is calculated by the following formula Bezier :

[0142]

[0143] Among them, r p ∈{0.25, 0.5, 0.75} represents the parameters of the sampling auxiliary points, B t,j (r) represents the point of the Bezier curve drawing command. LineFromTo can be regarded as a special case of CurveFromTo, where all control points are evenly distributed on a straight line.

[0144] By minimizing L loss To train the entire bimodal generation module, as shown in the following formula, the coefficient α is used to control the relative weight of the texture generation module in the entire optimization function to enhance the generalization of the model so that it can still produce good results in the task of generating vector characters without texture characteristics;

[0145]

[0146] Step 4: Input the glyph images of the test set into the bimodal fusion network model for testing, and finally output the brush calligraphy vector font.

[0147] Example 3

[0148] Through ablation experiments and comparative experiments with mainstream font generation models, the generation effect of TextureVAE-DMF was examined, proving that the method of the present invention has excellent generation effects in terms of detail fidelity and structural integrity of brush calligraphy vector fonts.

[0149] Ablation experiment of the bimodal generation module: (1) using the baseline model represented by SVG; (2) replacing the Transformer codec based on (1) and adding the proposed relaxed representation format; (3) adding the Bezier curve alignment loss L based on (2) bezier ; (4) Use a dual-modal generation module; The texture variational autoencoder module has five sets of outputs in the interval (0, 1), and the values ​​are 0.1, 0.3, 0.4, 0.5, 0.7, 0.9, and 1 respectively. The corresponding outputs of the decoding network Decoder. From the results, it can be seen that with the change of the control coefficient, the output results of the encoding network show basically stable changes in light and dark, and there is also a certain diversity in the results of the same control coefficient, which meets the requirements of the model for the task of generating diverse texture samples, so that it can have subtle changes within the controllable range while generating texture samples with the expected light and dark.

[0150] The model of the present invention is compared with the mainstream Chinese font generation methods AGIS-Net, MX-Font, and WriteLikeYou to verify the effectiveness of the model. The comprehensive results show that the TVAE-DMF model proposed in the present invention combines the advantages of variational autoencoders and bimodal fusion, and can learn the local detail features of brush calligraphy fonts in the generated results. Through the display of visualization results and the comparison of evaluation indicators, the superiority and effectiveness of the model in generating simple brush calligraphy vector characters are proved from qualitative and quantitative levels.

[0151] like Figure 6 As shown, the baseline model of the bimodal generation model refers to the basic structure of DeepVecFont, which helps to improve the generation effect. The relaxation representation proposed in the present invention ensures a direct and relatively smooth intuitive display of adjacent drawing commands in the result, and compares the superiority of the results of the proposed model in the vector font generation task to verify the effectiveness of the proposed module.

[0152] like Figure 7As shown, the effectiveness and superiority of the model in the generation results compared with the mainstream Chinese font generation method in terms of the style of the generated results and the level of visible edge contours are demonstrated. Secondly, the present invention mainly solves the vectorization generation task of brush calligraphy fonts. The model focuses on solving the texture loss problem of brush calligraphy fonts in the generation task. The texture variational autoencoder is used to generate brush character texture samples, and the generated vector character contours are rendered to achieve the generation of vector characters with texture art features. Therefore, the main focus in the result analysis is on generating visually satisfactory brush calligraphy vector characters, such as Figure 8 and Fig. 9 Figure 2 shows a brush calligraphy character rendered after texture generation using a texture variational autoencoder.

Claims

1. A dual-modal fusion method for brush calligraphy vector fonts based on texture variational autoencoder, characterized in that: Follow the steps below to implement it: Step 1, construct a vector Chinese font dataset, pre-process the Chinese fonts in the dataset, and divide it into a training set and a test set; Step 2, constructing a bimodal fusion network model based on texture variational autoencoder; the bimodal fusion network model includes a texture generation module and a bimodal generation module; The bimodal fusion network model is used to process the glyph images in the training set; Step 3: construct a loss function and train the bimodal fusion network model; Step 4: Input the glyph images of the test set into the bimodal fusion network model for testing, and finally output the brush calligraphy vector font.

2. The method for bimodal fusion of brush calligraphy vector fonts based on texture variational autoencoder as claimed in claim 1, characterized in that: In the step 1, specifically: Step 1.1, obtain the TTF format file of the Chinese font, use the FontForge tool combined with the Python script to read and traverse the TTF format file of the font from the character set file; Step 1.2, convert the font library in TTF format into SFD format using a multi-process parallel processing script; Step 1.3, extracting the glyph data of each character from the SFD format file, which includes the outline and shape of the data, and converting the extracted glyph data into a glyph image form; Step 1.4, data enhancement is performed on the generated glyph image by rotation, scaling, and shearing; Step 1.5, performing a relaxation representation on the character sequence in the glyph image processed in step 1.4 and calculating auxiliary Bezier points; Step 1.6, constrain the bitmap format of the glyph image obtained in step 1.5 to a 128*128 viewing frame; based on the 128*128 viewing frame, implement a data enhancement strategy through random factors; and divide all glyph image data sets into training sets and test sets according to a 7:3 ratio.

3. The method for bimodal fusion of brush calligraphy vector fonts based on texture variational autoencoder as claimed in claim 1, characterized in that: In the step 2, the texture generation module is composed of an encoding network Encoder and a decoding network Decoder; the structure of the encoding network includes three fully connected layers, each layer is followed by a ReLU activation function; the bimodal generation module includes a TVAE-SEQ module and a TVAE-PIX module; wherein the encoder and decoder in the TVAE-PIX module are respectively composed of a CNN and a deconvolutional neural network to output image features and generate target glyph images, and the TVAE-SEQ module is a sequence-to-sequence codec, which is a codec composed of a four-layer and a six-layer Transformer encoder block.

4. The method for bimodal fusion of brush calligraphy vector fonts based on texture variational autoencoder as claimed in claim 3, characterized in that: In the step 2, specifically: Step 2.1, use the encoding network and decoding network in the texture generation module to perform dimension conversion on the glyph images in the training set; specifically: Step 2.11, randomly crop the glyph images in the training set in step 1 to generate 10,000 25pix*25pix real texture blocks as input, and store them in Numpy data format; Step 2.12: The input 10,000 25*25 real texture blocks are compressed through the encoder and converted into two 30-dimensional vectors μ and σ. 2 , respectively as the mean and variance of the distribution of the latent variable z; Step 2.13, by randomly sampling ε and the vectors μ and σ obtained in the previous step 2 By performing addition and multiplication operations, we can obtain the hidden variable z corresponding to each real texture sample, which is the low-dimensional representation of the real calligraphy texture sample. Step 2.14, the structure of the decoding network is opposite to that of the encoding network. It takes the latent variable z output by the encoder as input, and through three reverse neural networks, supplements the secondary information of the sample missing in the latent variable z and converts it into a new sample with the same dimension as the real calligraphy texture sample; Step 2.2, the dual-modal generation module includes a TVAE-SEQ module and a TVAE-PIX module; the TVAE-PIX module and the TVAE-SEQ module encode the spatial shape information of the glyph image and the sequence information of the vector glyph respectively, and obtain the sequence and image fusion features of the vector glyph through the linear projection layer; Specifically: Step 2.21: Convert each glyph information in the glyph image after step 2.1 into a vector form G i , then each vector glyph G is obtained i , and sequence them As the input of the sequence-to-sequence encoder, YesN c The output of the Transformer sequence encoder is represented as d E represents the embedding dimension; Compute global sequence-wise style features in, (j=0, 1, ..., N c ) by linear projection from N r Aggregate from the th token of th reference glyphs; Set the input of the sequence decoder to The original modal markup is replaced by the fused feature f; an MLP layer is attached on top of the Transformer decoder to predict the command type and coordinates of the target glyph, denoted as and Step 2.22: TVAE-PIX module: It consists of a CNN consisting of six convolutional layers and two fully connected layers. The convolutional layers use ReLU as the activation function, and the fully connected layers use the tanh function. The CNN outputs f img , and get the new latent vector parameters μ1 and By sampling random noise ε from a standard normal distribution, and then using the new latent vector parameters μ1 and variance To reparameterize, finally get the updated random latent vector z1; Step 2.23: Obtain vector sequence data using the encoder in the TVAE-SEQ module and TVAE-PIX module to obtain the corresponding glyph bitmap data img The bimodal data of the input characters are concatenated and then formed into a new fused feature vector f through a multi-layer perceptron. The fused feature f' after the multi-layer perceptron MLP and concatenation operation is used as the input in the subsequent steps. The feature f' is input into the image decoder and the target glyph image I is generated through a deconvolutional neural network. t .

5. The method for dual-modal fusion of brush calligraphy vector fonts based on texture variational autoencoder as claimed in claim 4, characterized in that: In step 3, the loss function L of the texture generation module and the bimodal generation module is included in the training process of the bimodal fusion model based on texture variational autoencoder. Tex and L DMF The loss function L loss ; By optimizing the loss function, the model iterates to minimize the difference between the vector sequence in the decoder and the input vector glyph, as shown in the following formula: L loss =L DMF +L Tex .

6. The method for dual-modal fusion of brush calligraphy vector fonts based on texture variational autoencoder as claimed in claim 5, characterized in that: In the step 3, specifically: Step 3.1, loss function L of texture generation module Tex It consists of two parts, including the reconstruction loss function and the KL divergence part, as shown in the following formula: Among them, x represents the original sample, represents the synthetic sample, and n represents the number of samples; KL divergence is used to ensure that the decoding network learns the latent space variables that satisfy the prior distribution, as shown below: The Adam stochastic gradient descent algorithm is used to obtain the optimal network parameters. The basic learning rate is set to 0.0001, the learning rate decay coefficient is set to 0.5, the training loss is displayed every 100 iterations, and the model is kept every 10,000 iterations. The maximum number of iterations is set to 500,000. Step 3.2, TVAE-SEQ module: When predicting the command type and coordinates of the target glyph in the Transformer decoder, the initial prediction loss between the initial generated glyph and its corresponding GroundTruth is defined as follows: Where l represents the cross entropy loss, and all coordinates (p) are quantized to optimize l, w cmd represents the loss weight for command type prediction, and all unused parameters in the second term are masked; Minimize Command C i-1 The predicted endpoint of the command C i The L2 distance between the starting points of is defined as the relaxed representation of the merge loss, as shown in the following formula: in, and Represent the starting point and the ending point respectively, N c Indicates the command number; TVAE-PIX module: Input feature f into the image decoder to generate the target glyph image I t Then use L1 loss and perceptual loss to calculate image reconstruction loss, L1 loss, perceptual loss and image reconstruction loss L img They are shown as follows: Where n is the number of samples, y i is the true value of the i-th sample, is the predicted value of the i-th sample; Where x is the input image, y is the target image, and F i (x) and F i (y) respectively represent the feature representation of the i-th layer in the pre-trained neural network; in, is the predicted value of the target glyph; The Adam stochastic gradient descent algorithm is used to obtain the optimal network parameters. The initial learning rate is set to 0.0001, and the learning rate is gradually reduced at a decay rate of 0.9999 at each step until the minimum learning rate of 0.00001 is reached; Step 3.3, the remaining loss functions; L KL represents the KL loss normalized on the latent code f, L Bezier is the loss function used to constrain the auxiliary points, so from each drawing command C t,j More points are sampled in the experiment, where the Bezier curve based on the control point p is defined as follows: Where m represents the order of the curve, represents the binomial coefficient, and r is the position parameter; therefore, the alignment loss L of the Bezier curve is calculated by the following formula Bezier : Among them, r p ∈{0.25, 0.5, 0.75} represents the parameters of the sampling auxiliary points, B t,j (r) represents the point of the Bezier curve drawing command; By minimizing L loss To train the entire bimodal generation module, as shown below;