Font generation method based on autoregressive network with vector quantization and IDS fusion modeling

Abstract

The present invention proposes a font generation method based on an autoregressive network with vector quantization and IDS fusion modeling. A Chinese character set is selected to generate corresponding glyph images. The target glyph images are quantized using a fine-tuned VQGAN model to obtain a glyph quantization sequence. The IDS sequence of the selected Chinese character set is then completed. An IDS mutation module is used to generate difficult samples, and an encoder is used to generate corresponding IDS features and IDS contrast features for them. The style glyph quantization sequence is converted into vector features using a fine-tuned VQGAN model and then re-encoded to obtain corresponding style features and style contrast features. The target glyph quantization sequence, corresponding style features, and IDS features are input into an autoregressive IDS prediction network for training. The style glyph quantization sequence and the target glyph's IDS sequence are input into the final model, and the VQGAN decoder is used to decode the final model to obtain the desired target glyph image. The present invention can circumvent the limitations of content-style decoupling by modeling the glyph quantization sequence.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a font generation method based on an autoregressive network fused with vector quantization and IDS modeling. Background Art

[0002] Fonts are used to elegantly display text in computer systems and hold immense value in commercial and artistic applications. Manually designing fonts for languages ​​like Arabic numerals and Latin letters is feasible due to their limited character set and simple structure. However, for ideographic scripts with tens of thousands of characters, such as Chinese, Japanese, and Korean, manually designing a font is extremely complex and time-consuming. This requires font generation technology, which can create images of the remaining characters based on a few provided glyphs of the same style. Font generation is a challenging task. While it falls within the same generational domain, it differs from other generation tasks, such as face generation. While human faces appear visually diverse, their image structure is relatively uniform. For example, the eyes are always positioned above the nose in the arrangement of facial features. Furthermore, minor errors in highly detailed faces are generally inconspicuous. However, fonts are completely different. Even minor errors in the generated fonts can significantly alter the overall glyph. For example, a single stroke added to a Chinese character can result in a completely different character with a completely different meaning. This places very stringent requirements on the model's feature learning capabilities. The main difficulties of this task lie in two aspects: glyph reconstruction, that is, the generated glyphs cannot have structural errors, ambiguity, and other defects that affect reading; style fusion, that is, the result must perfectly replicate the style of the reference character.

[0003] With the rapid advancement of graphics card computing power and model architecture, font generation and its related fields of style transfer and image editing have achieved significant progress. Image manipulation typically relies on content-style decoupling, separating an image's content structure from its style and texture. However, content and style are not independent factors, and in practice, they are often difficult to completely separate, which in turn affects subsequent editing and integration steps. For example, font generation tasks often suffer from the fact that the generated style is influenced by the input content, glyphs, and images. Summary of the Invention

[0004] The purpose of the present invention is to propose a font generation method based on vector quantization and IDS fusion modeling autoregressive network, which circumvents the limitation of content-style decoupling by modeling glyph quantization sequence.

[0005] To achieve the above object, the technical solution of the present invention is: a font generation method based on vector quantization and IDS fusion modeling autoregressive network, specifically comprising the following steps:

[0006] Step S1: Construct a data set and perform preprocessing; collect font files from the Internet, select a Chinese character set to generate corresponding glyph images, load the pre-trained weights of the VQGAN model for fine-tuning, and then use the fine-tuned VQGAN model to quantize the glyph images to obtain the quantized sequence corresponding to the glyphs; in each training iteration of the model, randomly select a glyph image as the generation target, and its corresponding quantized sequence is the target glyph quantized sequence; then randomly select a few glyph images with the same style as the selected glyph image as the style glyph image, and its corresponding quantized sequence is the style glyph quantized sequence; on the other hand, complete the ideographic description sequence (IDS) of the selected Chinese character set, using 13 ideographic description characters (IDC) and 29 strokes to represent each Chinese character, and each glyph image has a corresponding IDS sequence;

[0007] Step S2: Use the IDS mutation module to generate difficult samples close to the IDS sequence in step S1, and use the IDS encoder to generate corresponding IDS features and IDS comparison features for them;

[0008] The style glyph quantization sequence is restored to a feature matrix using the fine-tuned VQGAN model, and then re-encoded to obtain the corresponding style features and style contrast features;

[0009] The target glyph quantization sequence from step S1, the corresponding style features from step S2, and the IDS features are fed into the autoregressive IDS prediction network for training, where the style is fused in the form of cross attention.

[0010] Step S3: The style contrast feature calculates the supervised contrast loss according to the font it belongs to; the autoregressive IDS prediction network output logits and the target glyph quantization sequence calculate the cross entropy loss, the prediction network output logits are weighted encoded, and the quantization encoder enc with frozen parameters is used when encoding logits. vq , the encoding result is compared with the IDS feature to calculate the first IDS contrast loss; the quantized encoder enc using unfrozen parameters vq The target glyph quantization sequence is encoded, and the result is compared with the IDS contrast feature to obtain the second IDS contrast loss;

[0011] Step S4: Perform iterative training according to the specified training parameters, update the model parameters by optimizing the combined loss, and screen and record the optimal model weights based on the comprehensive validation set indicators; input the style glyph quantization sequence and the target glyph IDS into the trained model, predict the target glyph quantization sequence in an autoregressive manner, and then use the VQGAN decoder to decode it to obtain the desired target glyph image.

[0012] Furthermore, step S1 specifically includes the following steps:

[0013] Step S11: Select the Chinese character set charset for training ch , and collect public font files, use each font file to render charset ch All Chinese characters in the form of pictures are obtained from the glyph dataset dataset ch ;

[0014] Step S12: Use dataset ch Fine-tune the weights of the VQGAN model; load the public pre-trained weights of the VQGAN model and use the glyph dataset processed in step S11 ch As input, continue training for 10 epochs to adapt the VQGAN model to the encoding and decoding of glyph images;

[0015] Step S13: Load the VQGAN model weights fine-tuned in step S12, and apply the glyph dataset processed in step S11 to the dataset. ch Perform vector quantization; randomly select from the dataset in each training iteration of the model ch Take a glyph image x img As the generated target, it is quantized. The specific quantization process is as follows: first, it is input into the VQGAN encoder enc vqgan , and get the output feature map Where size h , size w , size c Represent the height, width and number of channels of the output feature map respectively; then calculate Each vector of With codebook Each vector z k The Manhattan distance between them, and select the z closest to k to replace In this way Quantized to z q , the specific calculation method is as follows:

[0016]

[0017] The code book It is a vector table learned during VQGAN encoder training, which stores N z learnable vectors, q(·) represents the The element-by-element quantization operation is performed; in the quantization process, each z k Use it in Replace position k in the image and flatten it into a vector to get the glyph image x.img The corresponding quantized sequence x indices , as the target glyph quantization sequence, the specific process is as follows:

[0018]

[0019] Where flatten(·) means flattening the matrix into a vector, N z Represents the codebook The number of recorded vectors;

[0020] From the dataset ch Randomly pick an even number of cards and x img The image c belonging to the same style is used as the style font image, and it is quantized into the matrix cz according to the above quantization process. q , further replaced and flattened into a quantized sequence c indices , as a quantized sequence of style glyphs;

[0021] Step S14: Chinese character set charset ch Each Chinese character in the charset matches the corresponding IDS sequence; first obtain the open source ideographic character description sequence mapping table, each row of which gives a Chinese character and its corresponding IDS sequence, so according to the charset ch Just take out the corresponding part; each IDS sequence is composed of a combination of 13 ideographic description characters and 98 Chinese character components. 29 strokes are used to replace the 98 Chinese character components, making the IDS representation more delicate.

[0022] Furthermore, step S2 specifically includes the following steps:

[0023] Step S21: Build IDS mutation module gen ids , using the Chinese character set charset obtained in step S14 ch The IDS sequence of each Chinese character is used as input, and N is generated based on the original IDS sequence through operations including insertion, deletion, and modification. ids A similar but different variant IDS is used as a difficult sample for subsequent calculation of contrast loss; create an IDS queue ids , the first occurrence of the IDS sequence is calculated and the result is stored in the queue ids , and when the IDS sequence is encountered again, it will be directly removed from the queue ids Get N ids -1 mutation IDS, only gen ids Generate a mutation IDS and integrate it as the newly generated result, and then update the queue with the newly generated result ids ;

[0024] Step S22: Use IDS encoder enc ids queue ids Generate corresponding code and IDS comparison features; due to the N of each IDS sequence obtained in step S21 ids The lengths of the variant IDSs are different. First, use the special symbol [pad] to pad them to the same length, and then use another special symbol [cls] to insert it at the beginning of each variant IDS. ids Mutated IDSs form a matrix As shown below:

[0025]

[0026] Among them L ids Indicates the fixed length of each variant IDS after padding [pad], Each row represents a variant IDS; in order to extract features, an IDS encoder enc is designed ids , the IDS encoder enc ids It consists of an embedding layer and 6 bidirectional Transformer blocks, followed by Enter enc ids , convert each element into a corresponding vector according to a fixed mapping relationship to obtain the embedding feature embed ids ; Embed in the Transformer block ids Perform multi-layer self-attention calculations and finally take the first vector of each row as the IDS contrast feature cl ids , subsequent L ids -1 vector together as the final IDS feature x ids ;

[0027] Step S23: The quantized sequence x obtained in step S13 is indices As the model generation target, we first use the fine-tuned VQGAN to restore the style glyph quantization sequence to the feature matrix cz q ; Design style encoder enc style , the style encoder enc style The main body of the model is a residual network, which consists of 8 ordinary convolution blocks and 8 deformable convolution blocks. A residual connection is used between every two blocks to form a residual block. The style encoder enc style Encode cz again q Get the corresponding style feature x style , further style Perform average pooling operation to obtain style contrast feature cl style ;

[0028] Step S24: Design the autoregressive IDS prediction network netids , which consists of 6 coding modules block s With 6 feature fusion modules block c Alternating composition, i.e. Among them, b l The output of the encoding module is recorded as And the b l The output of the feature fusion module is recorded as x in step S13 indices and x in step S22 ids As the first coding module Input, the following 5 encoding modules The input is the output of the previous feature fusion module Will and x in step S23 style As a feature fusion module The input of , and the cross attention mechanism is used to fuse the two features. The specific calculation method is as follows:

[0029]

[0030]

[0031]

[0032]

[0033] Where proj1, proj2, and proj3 are different linear layers, transpose(·) represents the transposition of the matrix, and softmax(·) represents the softmax function; It is b l The output of the encoding module The features after linear layer projection, Is the style feature x style After linear layer projection and transposition, the features Is the style feature x style After another linear layer projection feature, size c is the number of channels of the output feature map mentioned in step S13, and its value is and The number of channels is the same; That is net ids The final output result logits is used for subsequent loss calculation.

[0034] Furthermore, step S3 specifically includes the following steps:

[0035] Step S3 specifically includes the following steps:

[0036] Step S31: Style contrast feature cl style Calculate style contrast loss based on the font cs , in order to help the style encoder encode style features more effectively, which is calculated as follows:

[0037]

[0038]

[0039]

[0040]

[0041] in All of them calculate Loss cs The intermediate result, τ cs represents the temperature coefficient of style contrast loss, max(·) represents the maximum value operation, mask cs It is the mask used to extract the positive sample pairs. express The csi-th element of ;

[0042] Step S32: weighted encoding of logits; first use softmax to process the logits from step S24 into each element in the sequence belonging to the codebook Probability of different vectors proba l , the specific calculation method is as follows:

[0043]

[0044]

[0045] in Indicates the probability that the xith vector in logits belongs to the xjth category, logits (xi)(xj) Indicates that the xi-th vector corresponds to the output of the xj-th category, logits (xi)(xk) Indicates that the xith vector corresponds to the output of the xkth category; then proba l is the weight, All vectors of are weighted summed to obtain the feature representation corresponding to logtis; Indicates freezing the quantization encoder enc vq The weight is used to encode, mean(·) represents the average operation, and finally the predicted sequence comparison feature cl is obtained xp ;

[0046] Step S33: cl xp Comparison with IDS features ids Calculate the first IDS contrast loss Loss co , the loss is calculated by adding logtis to the corresponding IDS contrast feature cl ids Closer, to constrain the autoregressive IDS prediction network net ids The predicted glyphs are correct and have no structural errors. The calculation method is as follows:

[0047]

[0048]

[0049]

[0050] where τ co represents the temperature coefficient of style contrast loss, It is cl xp With cl ids The result of the multiplication calculation, is The intermediate result after subtracting its own maximum value; label ids Represents logits and x ids The corresponding relationship between them is used to filter out positive sample pairs. Represents logits and coi-th x ids The degree of matching between vectors, log(·) represents the logarithmic operation;

[0051] Step S34: On the other hand, the quantization encoder enc vq x indices Encoding, the result is then compared with the IDS feature cl ids Calculate the second IDS contrast loss Loss ct ; for x indices , find its corresponding in the code book The vector inside, following step S32, uses the quantization encoder enc vq Encoding is performed, and the parameters are not frozen at this time. The encoding result is the same as cl ids Calculate the second IDS contrast loss Loss ct ;Loss ct According to x indices Information to guide the IDS encoder enc ids The extracted features cl ids Contains the correct structural information.

[0052] Furthermore, step S4 specifically includes the following steps:

[0053] Step S41: Calculate the overall network loss function Loss, the calculation formula is as follows

[0054] Loss=Loss sp +Loss cs +λ co Loss co +Loss ct

[0055] The overall optimization objective consists of four parts, where λ co It's Loss co The weight of λ co Starting from 0, it increases linearly to 1 according to the training progress to stabilize the model training; Loss sp It is the sequence prediction loss in the form of cross entropy, and the autoregressive IDS prediction network net ids Output logits and quantized sequence x indices As input, it is responsible for overall constraining the model prediction to be close to the true series, and is calculated as follows:

[0056] proba x =log(softmax(logits (spi) ) (spj) )

[0057]

[0058] where logits (spi) represents the spi-th vector of logits, softmax(·) (spj) It represents the spjth element of the vector after softmax processing, proba x is the intermediate result of the calculation, Represents x indices The spi-th element of ;

[0059] Step S42: Perform iterative training according to the specified parameters, and continuously update the gradient according to the overall loss Loss calculated in step S41 to perform iterative training;

[0060] Step S43: During training, a warmup strategy is used to allow learning to start from small to large and then maintain a constant value. The optimal and newer model is saved at a certain iteration interval based on the validation set indicators and training time, and the optimal model is continuously saved. When the model reaches the set maximum number of iterations, the training is terminated and the test data set is used to calculate the indicators of the optimal model weights that appear in the entire training process.

[0061] Step S44: Load the optimal model weights during inference. The goal is to generate the target glyph image. To do this, you need to input the IDS of the style glyph and the target glyph. Then use the IDS encoder and style encoder to encode the two into the IDS feature x of the target glyph respectively. ids and style features x style , then enter net ids , and predict logits in an autoregressive manner, which is calculated as follows:

[0062] logits (pi) =net ids (p pi )

[0063]

[0064]

[0065] where p pi is the model input in the pi-th iteration, net ids (·) indicates that the autoregressive IDS prediction network is used for one iteration, logits (pi) is the logits output by the pi-th iteration model, logits (pi)(-1) Then take out the last vector of logits in the pi-th iteration, Represents the next element to be predicted, and append(·) is used to splice the next element to the end of the previous vector to form the input p of the next iteration pi+1 , where p p0 It's x ids , each time generating one element of the glyph quantization sequence; repeat the above process until the iteration reaches the predetermined number of times;

[0066] Take the final p above pi , and remove the front L ids belongs to x ids The elements of the final predicted sequence p are obtained, that is, the predicted target glyph quantization sequence, in the codebook The corresponding vector is taken out to form the corresponding feature map, and then the VQGAN decoder is used to decode it to convert it into the final predicted glyph image.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] 1. The font generation method constructed by the present invention, which is based on the autoregressive network with vector quantization and IDS sequence fusion modeling, can accurately and harmoniously integrate the font style of the reference glyph into the generated result, and will not be affected by the style of the content glyph in the traditional method, thereby achieving a delicate and perfect font generation effect.

[0069] 2. The present invention revolutionizes the use of ideographic character description sequences as glyph semantic input, abandoning the input form of content glyph images. It can freely and flexibly synthesize some non-existent Chinese characters and generate glyph images that are more in line with the input style.

[0070] 3. To achieve clean and efficient style fusion, this invention comprehensively considers multiple input style glyph images and leverages a supervised contrast loss to further help the model distinguish fine-grained style differences. Most existing methods only accept a fixed number of input images and employ simple vector concatenation or adaptive instance normalization, commonly used in other fields, without the flexible and efficient style fusion capabilities of this invention.

[0071] 4. Unlike most previous methods that use generative adversarial networks, the input and output of the prediction network of the present invention are both quantized sequences of VQGAN, rather than rasterized images. By limiting the solution space to the vector space of fine-tuned VQGAN, problems such as blurring and artifacts common in images generated by traditional methods are avoided. At the same time, due to the soft supervision of VQGAN, problems such as missing strokes and disordered structure in the generated glyphs are also alleviated.

[0072] 5. Generally speaking, for contrast loss, the number of sample pairs and the proportion of difficult samples are the two factors that determine the effectiveness. Since the ideographic description sequence corresponds to the glyph and is easy to modify, small additions, deletions, and modifications can easily construct a large number of difficult samples. These samples can then be used to calculate the contrast loss with the prediction network output, allowing the model to grasp the structure of the original glyphs and guide the network to generate legal and compliant quantized sequences. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a schematic diagram of the principle of the present invention. DETAILED DESCRIPTION

[0074] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0075] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0076] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0077] like Figure 1 As shown, this embodiment provides a font generation method based on vector quantization and IDS sequence fusion modeling autoregressive network, which specifically includes the following steps:

[0078] Step S1: Construct a data set and perform preprocessing; collect font files from the Internet, select a Chinese character set to generate corresponding glyph images, load the pre-trained weights of the VQGAN model for fine-tuning, and then use the fine-tuned VQGAN model to quantize the glyph images to obtain the quantized sequence corresponding to the glyphs; in each training iteration of the model, randomly select a glyph image as the generation target, and its corresponding quantized sequence is the target glyph quantized sequence; then randomly select a few glyph images with the same style as the selected glyph image as the style glyph image, and its corresponding quantized sequence is the style glyph quantized sequence; on the other hand, complete the ideographic description sequence (IDS) of the selected Chinese character set, using 13 ideographic description characters (IDC) and 29 strokes to represent each Chinese character, and each glyph image has a corresponding IDS sequence;

[0079] Step S2: Use the IDS mutation module to generate difficult samples that are close to the IDS sequence in step S1, and use the IDS encoder to generate corresponding encoding and IDS comparison features for them. The corresponding encoding is the IDS feature;

[0080] The style glyph quantization sequence is restored to a feature matrix using the fine-tuned VQGAN model, and then re-encoded to obtain the corresponding style features and style contrast features;

[0081] The target glyph quantization sequence from step S1, the corresponding style features from step S2, and the IDS features are fed into the autoregressive IDS prediction network for training, where the style is fused in the form of cross attention.

[0082] Step S3: The style contrast feature calculates the supervised contrast loss according to the font to which it belongs; the autoregressive IDS prediction network outputs logits and the target glyph quantization sequence to calculate the cross entropy loss, the logits are normalized and then the codebook is weighted summed, the result is combined with the IDS contrast feature to calculate the first IDS contrast loss, and the quantization encoder enc is fixed when encoding logits. vq, so as to use IDS to correct the model. Considering that the IDS encoder parameters are still unstable in the early stage of model training, the weight is set to gradually increase the proportion of this loss in the next three epochs to stabilize the model training; on the other hand, enc vq The target glyph quantization sequence is encoded, and the result is compared with the IDS feature to obtain the second IDS contrast loss;

[0083] Step S4: Perform iterative training according to the specified training parameters, update the model parameters by optimizing the combined loss, and screen and record the optimal model weights based on the comprehensive validation set indicators; input the style glyph quantization sequence and the target glyph IDS into the trained model, predict the target glyph quantization sequence in an autoregressive manner, and then use the VQGAN decoder to decode it to obtain the desired target glyph image.

[0084] In this embodiment, step S1 specifically includes the following steps:

[0085] Step S11: Select the Chinese character set charset for training ch , and collect public font files, use each font file to render charset ch All Chinese characters in the form of pictures are obtained from the glyph dataset dataset ch ;

[0086] Step S12: Use dataset ch Fine-tune the VQGAN model. Load the public pre-trained weights of the VQGAN model and use the glyph dataset processed in step S11. ch As input, continue training for 10 epochs to adapt the VQGAN model to the encoding and decoding of glyph images.

[0087] Step S13: Load the VQGAN model weights fine-tuned in step S12, and apply the glyph dataset processed in step S11 to the dataset. ch Perform vector quantization. Randomly extract the vector from the dataset in each training iteration of the model ch Take a glyph image x img As the generated target, it is quantized. The specific quantization process is as follows: first, it is input into the VQGAN encoder enc vqgan , get the output feature map Where size h , size w , size c Represent the height, width and number of channels of the output feature map respectively. Then calculate Each vector of With codebook Each vector z k The Manhattan distance between them, and select the z closest to k to replace In this way Quantized to z q , the specific calculation method is as follows:

[0088]

[0089] The code book It is a vector table learned during VQGAN encoder training, which stores N z learnable vectors, q(·) represents the The element-by-element quantization operation is performed. In the quantization process, each z k Use it in By replacing the position k in , and then flattening it into a vector, we can get the glyph image x img The corresponding quantized sequence x indices The specific process is as follows:

[0090]

[0091] Where flatten(·) means flattening the matrix into a vector, N z Represents the codebook The number of recorded vectors;

[0092] From the dataset ch Randomly pick an even number of cards and x img The image c belonging to the same style is used as the style font image, and it is quantized into the matrix cz according to the above quantization process. q , further replaced and flattened into a quantized sequence c indices , as a quantized sequence of style glyphs;

[0093] Step S14: Chinese character set charset ch Each Chinese character in matches the corresponding IDS sequence. First, obtain the open source ideographic description sequence (IDS) mapping table, each row of which gives a Chinese character and its corresponding IDS sequence, so according to charset ch Each IDS sequence is composed of a combination of 13 ideographic description characters (IDCs) and 98 Chinese character components. The existing IDS mapping table was further supplemented as needed, replacing the 98 Chinese character components with 29 strokes to make the IDS representation more detailed.

[0094] In this embodiment, step S2 specifically includes the following steps:

[0095] Step S21: Build IDS mutation module gen ids , taking the IDS sequence of each Chinese character obtained in step S14 as input, through operations such as insertion, deletion, and modification, based on the original IDS sequence, generates N ids Other IDSs that are similar but different from it will be used as difficult samples for subsequent calculation of contrast loss. Considering that this operation is time-consuming and cannot be processed in parallel, an IDS queue is created. ids , the first occurrence of the IDS sequence is calculated and the result is stored in the queue ids , and when the IDS sequence is encountered again, it will be directly removed from the queue ids Get N ids -1 mutation IDS, only gen ids Generate a mutation IDS and integrate it as the newly generated result, and then update the queue with the newly generated result ids That's it, taking into account both efficiency and diversity.

[0096] Step S22: Use IDS encoder enc ids queue ids Generate the corresponding code and IDS comparison features. ids The lengths of the variant IDSs are different. First, use the special symbol [pad] to pad them to the same length, and then use another special symbol [cls] to insert it at the beginning of each variant IDS. ids Mutated IDSs form a matrix As shown below:

[0097]

[0098] Among them L ids Indicates the fixed length of each variant IDS after padding [pad], Each row represents a variant IDS; in order to extract features, an IDS encoder enc is designed ids , the IDS encoder enc ids It consists of an embedding layer and 6 bidirectional Transformer blocks, followed by Enter enc ids , convert each element into a corresponding vector according to a fixed mapping relationship to obtain the embedding feature embed ids ; Embed in the Transformer block ids Perform multi-layer self-attention calculations and finally take the first vector of each row as the IDS contrast feature cl ids , subsequent L ids -1 vector together as the final IDS feature x ids;

[0099] Step S23: The quantized sequence x obtained in step S13 is indices As the model generation target, we first use the fine-tuned VQGAN to restore the style glyph quantization sequence to the feature matrix cz q ; Design style encoder enc style , the style encoder enc style The main body of the model is a residual network, which consists of 8 ordinary convolution blocks and 8 deformable convolution blocks. A residual connection is used between every two blocks to form a residual block. The style encoder enc style Encode cz again q Get the corresponding style feature x style , further style Perform average pooling operation to obtain style contrast feature cl style ;

[0100] It should be noted that the "target glyph quantization sequence" is necessary, while the "style glyph quantization sequence" is strictly speaking not necessary; since the quantization sequence and the feature matrix can be converted into each other, and the quantization sequence is more compact and concise; therefore, in step S13, for the convenience of storage we preprocess the glyph image into the form of a quantization sequence; and during actual training, it is taken out and restored, which can effectively reduce the hard disk space occupied by the data and improve the system's IO efficiency.

[0101] Step S24: Design the autoregressive IDS prediction network net ids , which consists of 6 coding modules block s With 6 feature fusion modules block c Alternating composition, i.e. Among them, b l The output of the encoding module is recorded as And the b l The output of the feature fusion module is recorded as x in step S13 indices and x in step S22 ids As the first coding module Input, the following 5 encoding modules The input is the output of the previous feature fusion module Will and x in step S23 style As a feature fusion module The input of , and the cross attention mechanism is used to fuse the two features. The specific calculation method is as follows:

[0102]

[0103]

[0104]

[0105]

[0106] Where proj1, proj2, and proj3 are different linear layers, transpose(·) represents the transposition of the matrix, and softmax(·) represents the softmax function; It is b l The output of the encoding module The features after linear layer projection, Is the style feature x style After linear layer projection and transposition, the features Is the style feature x style After another linear layer projection feature, size c is the number of channels of the output feature map mentioned in step S13, and its value is and The number of channels is the same; That is net ids The final output result logits is used for subsequent loss calculation.

[0107] In this embodiment, step S3 specifically includes the following steps:

[0108] Step S31: Style contrast feature cl style Calculate style contrast loss based on the font cs , in order to help the style encoder encode style features more effectively, which is calculated as follows:

[0109]

[0110]

[0111]

[0112]

[0113] in All of them calculate Loss cs The intermediate result of cs represents the temperature coefficient of style contrast loss, max(·) represents the maximum value operation, mask cs It is the mask used to extract the positive sample pairs. express The csith element of .

[0114] Step S32: weighted encoding of logits. First, use softmax to process the logits from step S24 into the codebook. Probability of different vectors proba l , the specific calculation method is as follows:

[0115]

[0116]

[0117] in Indicates the probability that the xith vector in logits belongs to the xjth category, logits (xi)(xj) Indicates that the xi-th vector corresponds to the output of the xj-th category, logits (xi)(xk) Indicates that the xith vector corresponds to the output of the xkth category. Then, we use proba l is the weight, All vectors of are weighted summed to obtain the feature representation corresponding to logtis. Indicates freezing the quantization encoder enc vq The weight is used to encode, mean(·) represents the average operation, and finally the predicted sequence comparison feature cl is obtained xp ;

[0118] Step S33: cl xp Comparison with IDS features ids Calculate the first IDS contrast loss Loss co , the loss is calculated by adding logtis to the corresponding IDS contrast feature cl ids Pull closer to constrain the net ids The predicted glyphs are correct and have no structural errors. The calculation method is as follows:

[0119]

[0120]

[0121]

[0122] where τ co represents the temperature coefficient of style contrast loss, It is cl xp With cl ids The result of the multiplication calculation, is The intermediate result after subtracting its own maximum value; label ids Represents logits and x idsThe corresponding relationship between them is used to filter out positive sample pairs. Represents logits and coi-th x ids The degree of matching between vectors, log(·) represents the logarithmic operation;

[0123] Step S34: On the other hand, the quantization encoder enc vq x indices Encoding, the result is then compared with the IDS feature cl ids Calculate the second IDS contrast loss Loss ct In addition, for x indices , find its corresponding in the code book The vector inside, following step S32, uses the quantization encoder enc vq Encode, but do not freeze the parameters at this time, and then the result is the same as cl ids Calculate the second IDS contrast loss Loss ct Loss ct According to x indices Information to guide the IDS encoder enc ids The extracted features cl ids Contains the correct structural information.

[0124] Among them, for x indices The reason why the parameters are not frozen during encoding is that x indices Compared to S32's logtis, it is the groundtruth in the dataset, not the result generated by the model, and can be considered "correct", so the parameters are unfrozen for update. vq It does not affect x indices The calculation process of the encoding result is the same as that of cl in S32 xp The calculation process of the second IDS contrast loss is ct The calculation formula is the same as that of S33, except that the input cl xp Replaced with x indices The encoding result.

[0125] In this embodiment, step S4 specifically includes the following steps:

[0126] Step S41: Calculate the overall network loss function Loss, the calculation formula is as follows

[0127] Loss=Loss sp +Loss cs +λ co Loss co +Loss ct

[0128] The overall optimization objective consists of four parts, where λ co It's Loss co Since the IDS encoder parameters are not stable at the beginning of model training, we will co Starting from 0, it increases linearly to 1 according to the training progress to stabilize the model training. sp It is the sequence prediction loss in the form of cross entropy, and the autoregressive IDS prediction network net ids Output logits and quantized sequence x indices As input, it is responsible for overall constraining the model prediction to be close to the true series, and is calculated as follows:

[0129] proba x =log(softmax(logits (spi) ) (spj) )

[0130]

[0131] where logits (spi) represents the spi-th vector of logits, softmax(·) (spj) It represents the spjth element of the vector after softmax processing, proba x is the intermediate result of the calculation, Represents x indices The spi-th element of .

[0132] Step S42: Perform iterative training according to the specified parameters, and continuously update the gradient according to the overall loss Loss calculated in step S41 to perform iterative training;

[0133] Step S43: Use the warmup strategy during training, allowing learning to start from small to large, then maintain a constant, and save the optimal and newer model at a certain iteration interval based on the validation set indicators and training time, and continue to save the optimal model (model saving and indicators). When the model reaches the set maximum number of iterations, end the training and use the test data set to calculate the indicators of the optimal model weights that appeared in the entire training process;

[0134] Step S44: Load the optimal model weights during inference. The goal is to generate the target glyph image. To do this, you need to input the IDS of the style glyph and the target glyph. Then use the IDS encoder and style encoder mentioned in steps S22 and S23 to encode the two into the IDS feature x of the target glyph. ids and style features x style , then enter net ids , and predict logits in an autoregressive way, the calculation formula is as follows:

[0135] logits (pi) =net ids (p pi )

[0136]

[0137]

[0138] where p pi is the model input in the pi-th iteration, net ids (·) indicates that the autoregressive IDS prediction network is used for one iteration, logits (pi) is the logits output by the pi-th iteration model, logits (pi)(-1) Then take out the last vector of logits in the pi-th iteration, Represents the next element to be predicted, and append(·) is used to splice the next element to the end of the previous vector to form the input p of the next iteration pi+1 . Where p p0 It's x ids , each time generating one element of the glyph quantization sequence; repeat the above process until the iteration reaches the predetermined number of times.

[0139] Take the final p above pi , and remove the front L ids belongs to x ids The elements of get the final predicted sequence p, that is, the predicted target glyph quantization sequence. The corresponding vector is taken out to form the corresponding feature map, and then the VQGAN decoder is used to decode it to convert it into the final predicted glyph image.

[0140] In particular, this embodiment revolutionizes the use of ideographic character description sequences as glyph semantic input, abandons the input form of content glyph images, can freely and flexibly synthesize some non-existent Chinese characters, and the generated results are more in line with the input style glyph images. In order to achieve a clean and neat style fusion effect, the present invention can comprehensively consider multiple input style glyph images, and use supervised contrast loss to further help the model distinguish fine-grained style differences. Most existing methods can only receive a fixed number of input images, and the style fusion means are simple vector splicing or adaptive instance normalization that is mostly used in other fields. They do not have the flexible and efficient style fusion capabilities of the present invention. Unlike most previous methods that use generative adversarial networks, the input and output of the prediction network of the present invention are both quantized sequences of VQGAN, rather than rasterized images. By limiting the solution space to the vector space of fine-tuned VQGAN, problems such as blurring and artifacts commonly found in images generated by traditional methods are avoided. At the same time, due to the soft supervision of VQGAN, problems such as missing strokes and disordered structures in the generated glyphs are also alleviated. Generally speaking, for contrast loss, the number of sample pairs and the proportion of difficult samples are the two factors that determine the effectiveness of the results. Since the ideographic description sequence has a corresponding relationship with the glyph and is easy to modify, small additions, deletions, and modifications to it can easily construct a large number of difficult samples, which can be used to calculate the contrast loss with the prediction network output. The model can grasp the structure of the original glyph and guide the network to generate legal and compliant quantization sequences. The font generation method constructed by the present invention, based on a vector quantization and IDS sequence fusion modeling autoregressive network, can accurately and harmoniously integrate the font style of the reference glyph into the generated result, and is not affected by the style of the content glyph in traditional methods, thereby achieving a delicate and perfect font generation effect.

[0141] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions and effects do not exceed the scope of the technical solution of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. A font generation method based on an autoregressive network with vector quantization and IDS fusion modeling, characterized by: The following steps are involved: Step S1: construct the data set and perform preprocessing; Font files are collected from the Internet, and a Chinese character set is selected to generate corresponding glyph images. The pre-trained weights of the VQGAN model are loaded and fine-tuned. The glyph images are then quantized using the fine-tuned VQGAN model to obtain the corresponding quantized sequences of the glyphs. In each training iteration of the model, a glyph image is randomly selected as the generation target, and its corresponding quantized sequence is the target glyph quantized sequence. A small number of glyph images with the same style as the selected glyph image are randomly selected as style glyph images, and their corresponding quantized sequences are the style glyph quantized sequences. In addition, the IDS sequence of the selected Chinese character set is completed, using 13 ideographic characters and 29 strokes to represent each Chinese character. Each glyph image has a corresponding IDS sequence. Step S2: Based on the IDS sequence in step S1, the IDS mutation module is used to generate difficult samples, and the IDS encoder is used to generate corresponding IDS features and IDS comparison features for the difficult samples; The style glyph quantization sequence is restored to a feature matrix using the fine-tuned VQGAN model, and then re-encoded to obtain the corresponding style features and style contrast features; The target glyph quantization sequence from step S1, the corresponding style features from step S2, and the IDS features are fed into the autoregressive IDS prediction network for training, where the style is fused in the form of cross attention. Step S3: The style contrast feature calculates the supervised contrast loss according to the font it belongs to; the autoregressive IDS prediction network output logits and the target glyph quantization sequence calculate the cross entropy loss, the prediction network output logits are weighted encoded, and the quantization encoder enc with frozen parameters is used when encoding logits. vq , the encoding result is compared with the IDS feature to calculate the first IDS contrast loss; the quantized encoder enc using unfrozen parameters vq The target glyph quantization sequence is encoded, and the result is compared with the IDS contrast feature to obtain the second IDS contrast loss; Step S4: Perform iterative training according to the specified training parameters, update the model parameters by optimizing the combined loss, and screen and record the optimal model weights based on the comprehensive validation set indicators; input the style glyph quantization sequence and the target glyph IDS into the trained model, predict the target glyph quantization sequence in an autoregressive manner, and then use the VQGAN decoder to decode it to obtain the desired target glyph image.

2. The font generation method based on vector quantization and IDS fusion modeling autoregressive network according to claim 1 is characterized in that: Step S1 specifically includes the following steps: Step S11: Select the Chinese character set charset for training ch , and collect public font files, use each font file to render charset ch All Chinese characters in the form of pictures are obtained from the glyph dataset dataset ch ; Step S12: Use dataset ch Fine-tune the weights of the VQGAN model; Step S13: Load the VQGAN model weights fine-tuned in step S12, and apply the glyph dataset processed in step S11 to the dataset. ch Perform vector quantization; randomly select from the dataset in each training iteration of the model ch Take a glyph image x img As the generated target, it is quantized. The specific quantization process is as follows: first, it is input into the VQGAN encoder enc vqgan , and get the output feature map Where size h ,size w ,size c Represent the height, width and number of channels of the output feature map respectively; then calculate Each vector of With codebook Each vector z k The Manhattan distance between them, and select the z closest to k to replace In this way Quantized to z q , the specific calculation method is as follows: The code book It is a vector table learned during VQGAN encoder training, which stores N z learnable vectors, q(·) represents the The element-by-element quantization operation is performed; in the quantization process, each z k Use it in Replace position k in the image and flatten it into a vector to get the glyph image x. img The corresponding quantized sequence x indices , as the target glyph quantization sequence, the specific process is as follows: Where flatten(·) means flattening the matrix into a vector, N z Represents the codebook The number of recorded vectors; From the dataset ch Randomly pick an even number of cards and x img The image c belonging to the same style is used as the style font image, and it is quantized into the matrix cz according to the above quantization process. q , further replaced and flattened into a quantized sequence c indices , as a quantized sequence of style glyphs; Step S14: Chinese character set charset ch Each Chinese character in the charset matches the corresponding IDS sequence; first obtain the open source ideographic character description sequence mapping table, each row of which gives a Chinese character and its corresponding IDS sequence, so according to the charset ch Just take out the corresponding part; each IDS sequence is composed of a combination of 13 ideographic description characters and 98 Chinese character components. 29 strokes are used to replace the 98 Chinese character components, making the IDS representation more delicate.

3. The font generation method based on vector quantization and IDS fusion modeling autoregressive network according to claim 2 is characterized in that: The use of dataset ch Fine-tune the weights of the VQGAN model as follows: load the public pre-trained weights of the VQGAN model, use the glyph dataset processed in step S11 ch As input, continue training for 10 epochs to adapt the VQGAN model to the encoding and decoding of glyph images.

4. The font generation method based on vector quantization and IDS fusion modeling autoregressive network according to claim 2 is characterized in that: Step S2 specifically includes the following steps: Step S21: Build IDS mutation module gen ids , using the Chinese character set charset obtained in step S14 ch The IDS sequence of each Chinese character is used as input, and N is generated based on the original IDS sequence through operations including insertion, deletion, and modification. ids A similar but different variant IDS is used as a difficult sample for subsequent calculation of contrast loss; create an IDS queue ids , the first occurrence of the IDS sequence is calculated and the result is stored in the queue ids , and when the IDS sequence is encountered again, it will be directly removed from the queue ids Get N ids -1 mutation IDS, only gen ids Generate a mutation IDS and integrate it as the newly generated result, and then update the queue with the newly generated result ids ; Step S22: Use IDS encoder enc ids queue ids Generate corresponding code and IDS comparison features; due to the N of each IDS sequence obtained in step S21 ids The lengths of the variant IDSs are different. First, use the special symbol [pad] to pad them to the same length, and then use another special symbol [cls] to insert it at the beginning of each variant IDS. ids Mutated IDSs form a matrix As shown below: Among them L ids Indicates the fixed length of each variant IDS after padding [pad], Each row represents a variant IDS; in order to extract features, an IDS encoder enc is designed ids , the IDS encoder enc ids It consists of an embedding layer and 6 bidirectional Transformer blocks, followed by Enter enc ids , convert each element into a corresponding vector according to a fixed mapping relationship to obtain the embedding feature embed ids ; Embed in the Transformer block ids Perform multi-layer self-attention calculations and finally take the first vector of each row as the IDS contrast feature cl ids , subsequent L ids -1 vector together as the final IDS feature x ids ; Step S23: The quantized sequence x obtained in step S13 is indices As the model generation target, we first use the fine-tuned VQGAN to restore the style glyph quantization sequence to the feature matrix cz q ; Design style encoder enc style , the style encoder enc style The main body of the model is a residual network, which consists of 8 ordinary convolution blocks and 8 deformable convolution blocks. A residual connection is used between every two blocks to form a residual block. The style encoder enc style Encode cz again q Get the corresponding style feature x style , further style Perform average pooling operation to obtain style contrast feature cl style ; Step S24: Design the autoregressive IDS prediction network net ids , which consists of 6 coding modules block s With 6 feature fusion modules block c Alternating composition, i.e. Among them, b l The output of the encoding module is recorded as And the b l The output of the feature fusion module is recorded as x in step S13 indices and x in step S22 ids As the first coding module Input, the following 5 encoding modules The input is the output of the previous feature fusion module Will and x in step S23 style As a feature fusion module The two features are fused using the cross attention mechanism. for net ids The final output result logits is used for subsequent loss calculation.

5. The font generation method based on vector quantization and IDS fusion modeling autoregressive network according to claim 4 is characterized in that: Feature fusion module The specific calculation method for feature fusion using the cross attention mechanism is as follows: Where proj1, proj2, and proj3 are different linear layers, transpose(·) represents the transposition of the matrix, and softmax(·) represents the softmax function; It is b l The output of the encoding module The features after linear layer projection, Is the style feature x style After linear layer projection and transposition, the features Is the style feature x style After another linear layer projection feature, size c is the number of channels of the output feature map mentioned in step S13, and its value is and The number of channels is the same.

6. The font generation method based on vector quantization and IDS fusion modeling autoregressive network according to claim 4 is characterized in that: Step S3 specifically includes the following steps: Step S31: Style contrast feature cl style Calculate style contrast loss based on the font cs , to help the style encoder encode style features more effectively; Step S32: weighted encoding of logits; first use softmax to process the logits from step S24 into each element in the sequence belonging to the codebook Probability of different vectors proba l , the specific calculation method is as follows: in Indicates the probability that the xith vector in logits belongs to the xjth category, logits (xi)(xj) Indicates that the xi-th vector corresponds to the output of the xj-th category, logits (xi)(xk) Indicates that the xith vector corresponds to the output of the xkth category; then proba l is the weight, All vectors of are weighted summed to obtain the feature representation corresponding to logtis; Indicates freezing the quantization encoder enc vq The weight is used to encode, mean(·) represents the average operation, and finally the predicted sequence comparison feature cl is obtained xp ; Step S33: cl xp Comparison with IDS features ids Calculate the first IDS contrast loss Loss co , the loss is calculated by adding logtis to the corresponding IDS contrast feature cl ids Closer, to constrain the autoregressive IDS prediction network net ids The predicted glyphs are correct and have no structural errors. The calculation method is as follows: where τ co represents the temperature coefficient of style contrast loss, It is cl xp With cl ids The result of the multiplication calculation, is The intermediate result after subtracting its own maximum value; label ids Represents logits and x ids The corresponding relationship between them is used to filter out positive sample pairs. Represents logits and coi-th x ids The degree of matching between vectors, log(·) represents the logarithmic operation; Step S34: On the other hand, the quantization encoder enc vq x indices Encoding, the result is then compared with the IDS feature cl ids Calculate the second IDS contrast loss Loss ct ; for x indices , find its corresponding in the code book The vector inside, following step S32, uses the quantization encoder enc vq Encoding is performed, and the parameters are not frozen at this time. The encoding result is the same as cl ids Calculate the second IDS contrast loss Loss ct ;Loss ct According to x indices Information to guide the IDS encoder enc ids The extracted features cl ids Contains the correct structural information.

7. The font generation method based on vector quantization and IDS fusion modeling autoregressive network according to claim 6 is characterized in that: The style contrast loss Loss cs is calculated as follows: in All of them calculate Loss cs The intermediate result, τ cs represents the temperature coefficient of style contrast loss, max(·) represents the maximum value operation, mask cs It is the mask used to extract the positive sample pairs. express The csith element of .

8. The font generation method based on vector quantization and IDS fusion modeling autoregressive network according to claim 7 is characterized in that: Step S4 specifically includes the following steps: Step S41: Calculate the overall network loss function Loss, the calculation formula is as follows Loss=Loss sp +Loss cs +λ co Loss co +Loss ct The overall optimization objective consists of four parts, where λ co It's Loss co The weight of λ co Starting from 0, it increases linearly to 1 according to the training progress to stabilize the model training; Loss sp It is the sequence prediction loss in the form of cross entropy, and the autoregressive IDS prediction network net ids Output logits and quantized sequence x indices As input; Step S42: Perform iterative training according to the specified parameters, and continuously update the gradient according to the overall loss Loss calculated in step S41 to perform iterative training; Step S43: During training, a warmup strategy is used to allow learning to start from small to large and then maintain a constant value. The optimal and newer model is saved at a certain iteration interval based on the validation set indicators and training time, and the optimal model is continuously saved. When the model reaches the set maximum number of iterations, the training is terminated and the test data set is used to calculate the indicators of the optimal model weights that appear in the entire training process. Step S44: Load the optimal model weights during inference. The goal is to generate the target glyph image. To do this, you need to input the IDS of the style glyph and the target glyph. Then, use the IDS encoder and style encoder to encode the two into the IDS feature x of the target glyph. ids and style features x style , then enter net ids , and predict logits in an autoregressive manner, and obtain the quantized sequence of the target glyph, in the encoding book The corresponding vector is taken out to form the corresponding feature map, and then the VQGAN decoder is used to decode it to convert it into the final predicted glyph image.

9. The font generation method based on vector quantization and IDS fusion modeling autoregressive network according to claim 8, characterized in that: Sequence prediction loss in the form of cross entropy sp , which is calculated as follows: proba x =log(softmax(logits (spi) ) (spj) ) where logits (spi) represents the spi-th vector of logits, softmax(·) (spj) It represents the spjth element of the vector after softmax processing, proba x is the intermediate result of the calculation, Represents x indices The spi-th element of .

10. The font generation method based on vector quantization and IDS fusion modeling autoregressive network according to claim 8, characterized in that: The quantized sequence of the target glyph is predicted in an autoregressive manner. The calculation formula is as follows: logits (pi) =net ids (p pi ) where p pi is the model input in the pi-th iteration, net ids (·) indicates that the autoregressive IDS prediction network is used for one iteration, logits (pi) is the logits output by the pi-th iteration model, logits (pi)(-1) Then take out the last vector of logits in the pi-th iteration, Represents the next element to be predicted, and append(·) is used to splice the next element to the end of the previous vector to form the input p of the next iteration pi+1 , where p p0 It's x ids , each time it generates one element of the glyph quantization sequence; Repeat the above process until the number of iterations reaches the predetermined number; Take the final p above pi , and remove the front L ids belongs to x ids The elements of get the final predicted sequence p, that is, the predicted target glyph quantization sequence.

Images