A method for generating offline handwriting samples based on multi-feature fusion

Abstract

The present invention discloses an offline handwriting sample generation method based on multi-feature fusion. The method constructs a handwriting sample generation model based on multi-feature fusion. The method inputs the word text to be generated and a reference author style image or author ID into the trained handwriting sample generation model based on multi-feature fusion to generate a handwriting sample image. The method extracts local and global style features and content features from the fixed-size author style image to obtain a global style code, a local style code, a global text content code, and a local text content code. The global style code and the global text content code are concatenated to obtain a style content code. The style content code, local style code, and local text content code are subjected to feature fusion at different scales. The feature map after feature fusion is upsampled to generate a handwriting sample image. This method overcomes the problems of offline handwriting sample generation such as a single style, a lack of author style prominence, and poor generation quality.

Description

Technical Field

[0001] The present invention belongs to the field of offline handwriting sample generation, and relates to an offline handwriting sample generation method based on multi-feature fusion. Background Art

[0002] Most offline handwriting sample generation methods are based on GANs, such as GANwriting, ScrabbleGAN, Handwriting Transformers, SmartPatch, and HiGAN. These methods consider both content and style, so the GAN framework generally has two inputs: text content and style. However, in previous work, global features (thickness, depth, tilt, roundness, and curvature of the handwriting) and local features (the personalized writing of specific letters) have not been well integrated and balanced. In languages ​​such as English, French, and German, which are composed of a limited number of characters arranged horizontally, cursive writing is very common. This requires that sample generation methods should not lose sight of global features while focusing on local features. This is extremely important in offline handwriting sample generation. Therefore, existing methods suffer from the problems of single generated style, lack of author style, and poor generation quality. Summary of the Invention

[0003] The purpose of the embodiments of the present invention is to provide an offline handwriting sample generation method based on multi-feature fusion to solve the problems of the existing offline handwriting sample generation methods such as single generation style, lack of author style prominence and poor generation quality.

[0004] The technical solution adopted in the embodiment of the present invention is: a method for generating offline handwriting samples based on multi-feature fusion, comprising the following steps:

[0005] Step S1: construct a handwriting sample generation model based on multi-feature fusion;

[0006] Step S2: training a handwriting sample generation model based on multi-feature fusion;

[0007] Step S3: Input the word text to be generated and the reference author style image or author ID into the trained multi-feature fusion-based handwriting sample generation model to generate a handwriting sample image.

[0008] Furthermore, the specific implementation process of step S3 is as follows:

[0009] Step S31: Input the word text to be generated and the reference author style image or author ID. If the author ID is input, call the author style image through the author ID, and pre-process the author style image directly input or called through the author ID to fix the size of the author style image;

[0010] Step S32: extract local style features, global style features, local content features and global content features from the fixed-size author style image to obtain the global style code F S , local style encoding F Sn , Global text content encoding F C and local text content encoding α n and β n , n={1,2,3};

[0011] Step S33: Encode the global style F S With global text content encoding F C After splicing, we get the style content code F CS , encode the style content F CS , local style encoding F Sn and local text content encoding α n and β n Perform feature fusion at different scales;

[0012] Step S34: up-sample the feature map obtained in step S33 by fusing features of different scales to generate a handwritten sample image.

[0013] Furthermore, in step S32, the fixed-size author style image is input into the writing style encoder VGG19, and the writing style encoder VGG19 outputs the global style code F S ;

[0014] The outputs of the ReLU5-1, ReLU4-1, and ReLU3-1 layers of VGG19 are used as the local style encoding F S1 、F S2 、F S3 , extract the local style code F Sn , n={1,2,3};

[0015] The fixed-size author style image is input into the text encoder, which contains two parallel multi-layer perceptrons. The two parallel multi-layer perceptrons output two global text content codes F C and f C , global text content encoding F C With the global style code F S The same size as the global text content encoding f C It is a two-dimensional feature matrix that encodes the text content f C Divide equally into 3 pairs of one-dimensional parameters α n and β n .

[0016] Furthermore, step S33 encodes the style content FCS , local style encoding F Sn and local text content encoding α n and β n Input image generator for feature fusion. The specific implementation process of feature fusion is as follows:

[0017] (1) Encode the style content F CS After upsampling and local style encoding F S1 The fused feature matrix Z is obtained by adding them together, and the fused feature matrix Z is sent to the AdaIN normalization function to be fused with the local text content codes α1 and β1, and the feature code F that combines the global style features, local style features and text content features is output;

[0018] (2) Use the feature code F as the new style content code F CS , follow step (1) to encode the style content F CS , local style encoding F S2 And local text content encoding α2 and β2 for feature fusion;

[0019] (3) Take the feature code F output from step (2) as the new style content code F CS , follow step (1) to encode the style content F CS , local style encoding F S3 And local text content encoding α3 and β3 are used for feature fusion, and a feature map that fuses features of different scales is output.

[0020] Furthermore, the style content is encoded F CS After upsampling and local style encoding F Sn The specific operations of adding and fusion of features are as follows:

[0021] First, encode the style content F CS After upsampling and style encoding F Sn Perform broadcast addition;

[0022] Then, the broadcast addition result X is fed into the multi-scale channel attention mechanism, which consists of two parts: global channel attention and local channel attention.

[0023] The local channel attention L(X) can be expressed as:

[0024] L(X)=BN(PWConv2(δ(BN(PWConv1(X)))));

[0025] Among them, δ is the RELU activation function, PWConv1 and PWConv2 are two different point convolution layers, and BN is the batch normalization operation;

[0026] The global channel attention G(X) can be expressed as:

[0027] G(X)=BN(PWConv2(δ(BN(PWConv1(GAP(X))))));

[0028] Among them, GAP represents the global average pooling operation;

[0029] The results of global channel attention and local channel attention are broadcast and added and then sent to the Sigmoid activation function. The final output result Y is mapped to [0,1]. The output result Y is the weight of feature fusion. Using the weight Y of feature fusion, the feature matrix F is calculated according to the following formula. CS With style code F Sn Weighted sum:

[0030]

[0031] Among them, the fusion feature matrix Z is the weighted summation result of the features, Indicates broadcast addition.

[0032] Furthermore, the AdaIN normalization function can be expressed as:

[0033]

[0034] Among them, μ(Z) represents the variance of the fused feature matrix Z, and σ(Z) represents the mean of the fused feature matrix Z.

[0035] Furthermore, in step S2, when training the offline handwriting sample generation model based on multi-feature fusion, the generated handwriting sample image is input to the discriminator D, the author classifier W and the word recognizer R respectively to determine the discrimination loss L D , writing style loss L W and text content loss L R , and back-propagates the error to the image generator of the offline handwriting sample generation model based on multi-feature fusion to update the parameters of the image generator.

[0036] Furthermore, the discriminant loss L D The formula is:

[0037]

[0038] Where E is the expectation, χ is the training set, and x is the sample collected in the training set χ; A collection of samples generated by a handwriting sample generation model based on multi-feature fusion. For the sample set The samples collected in , D(x) is the true predicted probability distribution made by the discriminator D for sample x, is the discriminator D for the sample The probability distribution of the true predictions made.

[0039] Furthermore, the writing style loss L W The formula is:

[0040]

[0041] Where E is the expectation, χ is the training set, and x is the sample collected in the training set χ; A collection of samples generated by a handwriting sample generation model based on multi-feature fusion. For the sample set Samples collected in is the probability that the sample x predicted by the author classifier W is the handwritten sample of the kth author. During training, the author classifier W is optimized only with real samples collected from the training set χ. At this time, x~χ. When used to guide the training of the image generator, the parameters of the author classifier W are fixed. Only generated using image generators Samples in the set; w k is the true probability distribution of the kth author, and |Z| is the number of authors.

[0042] Furthermore, the text content loss L R The formula is:

[0043]

[0044] Where E is the expectation, χ is the training set, and x is the sample collected in the training set χ; The set of samples generated for the image generator, For the sample set Samples collected in is the probability distribution of the word recognizer R identifying the i-th character in sample x as the j-th character in the character set A, t i,j yes The corresponding true probability; l represents the maximum length of a word in the current training set, and |A| represents the total number of characters in the character set A.

[0045] The beneficial effects of the embodiments of the present invention are: in-depth research on the offline handwriting sample generation process, combining the method of feature fusion of different scales in the generative adversarial network GANs, i.e., the image generator, overcomes the problems of single style of offline handwriting sample generation, lack of prominent author style, and poor generation quality. It can be used as a data augmentation method for handwritten text recognition, or directly used to expand handwritten text recognition datasets, handwriting imitation and other fields, which helps to promote the development of offline handwriting sample generation research, greatly reduces the difficulty of constructing handwritten text datasets, provides favorable conditions for handwritten text recognition, and has very important academic and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 This is a flowchart of the offline handwriting sample generation method based on multi-feature fusion of this embodiment.

[0048] Figure 2 3 is a structural diagram of the offline handwriting sample generation model based on multi-feature fusion in this embodiment.

[0049] Figure 3 Schematic diagram of the structure of the feature fusion module of this embodiment.

[0050] Figure 4 Schematic diagram of the structure of the discriminator of this embodiment.

[0051] Figure 5 This is the experimental result of the offline handwriting sample generation method based on multi-feature fusion in this embodiment. DETAILED DESCRIPTION

[0052] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0053] This embodiment provides a method for generating offline handwriting samples based on multi-feature fusion, including:

[0054] Step S1: Construct a handwriting sample generation model based on multi-feature fusion. The structure of the handwriting sample generation model based on multi-feature fusion is shown in Figure 2 ;

[0055] Step S2: training a handwriting sample generation model based on multi-feature fusion;

[0056] Step S3: Input the word text to be generated and the reference author style picture or author ID into the trained multi-feature fusion-based handwriting sample generation model to generate a handwriting sample image corresponding to the word text, such as Figure 1 As shown, the specific process is shown in steps S31 to S34.

[0057] Figure 2 This is a schematic diagram of the offline handwriting sample generation model for multi-feature fusion in this embodiment. The offline handwriting sample generation model for multi-feature fusion includes a writing style encoder, a text content encoder, and an image generator. The writing style encoder consists of a VGG19. The author's style image is fed into the writing style encoder, and the final output is the style code F S , the outputs of the ReLU5-1, ReLU4-1, and ReLU3-1 layers in VGG19 are the local style code F S1 、F S2 、F S3 , get the local style code F Sn , n = {1,2,3}. The text encoder is composed of two multi-layer perceptrons. When a word text is fed into the text encoder, the word text is first converted into a one-hot vector. The end of the one-hot vector is padded with "#" to convert it into a fixed-length character code. The character code is mapped into the text content code F through a multi-layer perceptron. C , and the text content is encoded in F C With style code F S The same size, the global style encoding F S With global text content encoding F C After splicing, we get the style content code F CS ; At the same time, the character code is mapped into a two-dimensional feature matrix through another multi-layer perceptron, which is the text content code f C , encode the text content f C Divide equally into 3 pairs of one-dimensional parameters α n and β n , n={1,2,3}. And encode the style content F CS , local style encoding F Sn and 3 pairs of one-dimensional parameters α n and β n, n = {1, 2, 3} are injected into the feature fusion module of the image generator. The image generator includes three feature fusion modules and four bilinear interpolation as upsampling layers. The three feature fusion modules are evenly interspersed between the four upsampling layers. The output of the first three upsampling layers is used as an input to the adjacent feature fusion modules in turn. The output of the last upsampling layer is the generated handwritten sample image, which is output after the tanh activation function. Style content encoding F CS Entering the first upsampling layer, the output of the ReLU5-1, ReLU4-1, and ReLU3-1 layers in the writing style encoder VGG19 and three pairs of one-dimensional parameters α n and β n (n={1,2,3}), which are received by the three feature fusion modules respectively. The handwritten sample images generated by the image generator are fed into the discriminator D, the author classifier W, and the word recognizer R respectively, to obtain the discriminant loss, writing style loss, and text content loss.

[0058] Step S31: Input the word text to be generated and the reference author style image or author ID. If the author ID is entered, the author style image is called by the author ID. The author style image, either directly input or called by the author ID, is preprocessed and fixed in size. Specifically, during training, a document is fed in that corresponds to the image name, image text content, image corresponding author, and author ID. This document is provided in the IAM dataset.

[0059] When generating offline handwriting samples for the words to be generated, if the reference author style picture is an existing author style picture that has been preprocessed and stored, there is no need to preprocess it again and it can be directly called from the stored database. If it is an author style picture that has not been preprocessed and stored, it is preprocessed, the size of the author style picture is fixed, and it is used and stored in the corresponding database.

[0060] If the size of the input author style image is too large or too small, a corresponding scaling operation is performed to fix the height of the author style image to 64 and the length to 216. The specific size of the author style image is determined by the structure of the neural network model used. This embodiment uses VGG19, so the height of the author style image needs to be fixed to 64 and the length needs to be fixed to 216. If the author ID is entered, the author style image of the author is called. In order to ensure both training speed and effect, each author refers to 50 different author style images during training. If the current author style images are less than 50, the existing author style images are copied to make up the 50 author style images. However, after the training is completed, the generation process can generate a handwritten sample of the author's writing style by referring to only one author style image.

[0061] Step S32: extract local style features, global style features, local content features and global content features from the fixed-size author style image to obtain the global style code F S , local style encoding F Sn , Global text content encoding F C and local text content encoding α n and β n , n={1,2,3}.

[0062] The fixed-size author style image is input into three convolutional neural networks of different depths to extract the local style code F S1 、F S2 、F S3 The writing style encoder of this embodiment can select VGG19, and the output of VGG19 is the global style code F S , the outputs of the ReLU5-1, ReLU4-1, and ReLU3-1 layers in VGG19 are the local style code F S1 、F S2 、F S3 .

[0063] The fixed-size author style image is input into the text encoder, which contains two parallel multi-layer perceptrons. The two parallel multi-layer perceptrons output two global text content codes F C and f C , global text content encoding F C With the global style code F S The same size as the global text content encoding f C It is a two-dimensional feature matrix that encodes the text content f C Divide equally into 3 pairs of one-dimensional parameters α n and β n (n={1,2,3}).

[0064] Encode the text content C Divide equally into 3 pairs of one-dimensional parameters α n and β n The specific approach is: encode the global text content f C Split by row to obtain two one-dimensional feature matrices α and β, and then divide the one-dimensional feature matrices α and β into n equal parts to obtain n pairs of one-dimensional parameters α n and β n .

[0065] Step S33: Encode the global style F S With global text content encoding F C After splicing, we get the style content code F CS, Since the VGG19 neural network structure used in this embodiment is based on the size of the author's style image, the output of VGG19 is 512*8*27, so the style content encoding F CS It is a 512-dimensional feature vector that encodes the style content F CS , local style encoding F Sn and local text content encoding α n and β n Perform feature fusion. The specific implementation process of feature fusion is as follows:

[0066] (1) Encode the style content F CS After upsampling and local style encoding F S1 The fused feature matrix Z is obtained by adding them together and then fed into the AdaIN normalization function to perform feature fusion with the local text content codes α1 and β1 to avoid losing the text content during style fusion. The output is the feature code F that combines the global style features, local style features, and text content features.

[0067] (2) Use the feature code F as the new style content code F CS , follow step (1) to encode the style content F CS , local style encoding F S2 And local text content encoding α2 and β2 for feature fusion;

[0068] (3) Take the feature code F output from step (2) as the new style content code F CS , follow step (1) to encode the style content F CS , local style encoding F S3 And the local text content encoding α3 and β3 are used for feature fusion. Through three feature fusions, the fusion of features of different scales is finally completed, and the feature map of fusion of features of different scales is output.

[0069] Encode the style content F CS After upsampling and local style encoding F Sn The specific operations of adding and fusion of features are as follows:

[0070] First, encode the style content F CS After upsampling and style encoding F Sn Perform broadcast addition;

[0071] Then, the broadcast addition result X is fed into the multi-scale channel attention mechanism, which consists of two parts: global channel attention and local channel attention.

[0072] The local channel attention L(X) can be expressed as:

[0073] L(X)=BN(PWConv2(δ(BN(PWConv1(X)))));

[0074] Among them, δ is the RELU activation function, PWConv1 and PWConv2 are two different point convolution layers, and BN is the batch normalization operation (Batch Normalization);

[0075] The global channel attention G(X) can be expressed as:

[0076] G(X)=BN(PWConv2(δ(BN(PWConv1(GAP(X))))));

[0077] Among them, GAP represents the global average pooling operation;

[0078] The results of global channel attention and local channel attention are broadcast and added and then fed into the Sigmoid activation function. The final output result Y is mapped to [0,1], as shown in Figure 3 As shown, the output result Y is the weight of feature fusion. Using the weight Y of feature fusion, the feature matrix F CS With style code F Sn Weighted summation, the formula can be expressed as:

[0079]

[0080] Among them, the fusion feature matrix Z is the weighted summation result of the features, Indicates broadcast addition.

[0081] The image generator needs to collect semantic information at different levels. Therefore, the image generator must be able to accept and fuse features of different scales, and it must ensure that the required content features are not lost during the fusion of style features. However, there is a great deal of semantic inconsistency among multiple fused features, which results in the fusion weights of different layers having a great impact on the quality of the generated image. In order to eliminate this effect, this embodiment adopts an attention-based method to fuse features. This method can obtain the feature matrix F CS With style code F Sn The added weight coefficients are implemented using the Multi-Scale Channel Attention Model (MS-CAM).

[0082] When the fused feature matrix Z is fed into the AdaIN normalization function and the feature is fused with the local text content encoding α1 and β1, the AdaIN normalization function can be expressed as:

[0083]

[0084] Among them, μ(Z) represents the variance of the fused feature matrix Z, and σ(Z) represents the mean of the fused feature matrix Z.

[0085] Step S34: up-sample the feature map obtained in step S33 that fuses features of different scales to generate a handwriting sample image, which is then output after passing through a tanh activation function.

[0086] The style code F S With text content encoding F C After splicing, we get the style content code F CS , and then sent to the image generator. The feature matrix doubles in size every time it passes through the upsampling layer in the image generator, and the matrix size is halved every time the author style picture passes through the pooling layer in the writing style encoder. Therefore, the style code F output by the ReLU5-1, ReLU4-1 and ReLU3-1 layers of the writing style encoder is S1 、F S2 、F S3 The three feature fusion modules of the image generator are connected in a one-to-one correspondence according to the matrix size. In this embodiment, the image generator composed of three feature fusion modules realizes the fusion of global style features, local style features and text content features.

[0087] The method for generating an offline handwriting sample based on multi-feature fusion in this embodiment further includes:

[0088] The generated handwritten sample images are input to the discriminator D, author classifier W and word recognizer R respectively to determine the discrimination loss L D , writing style loss L W and text content loss L R .

[0089] like Figure 2 As shown, this embodiment has three loss functions to control the generation of offline handwriting samples based on multi-feature fusion, namely the discriminator D, the author classifier W and the word recognizer R, thereby obtaining the discriminant loss L of the discriminator D D , the writing style loss L of the author classifier W W and the text content loss L of the word recognizer R R .

[0090] The discriminator D is used to determine whether the generated handwriting sample is real. It is a supervised image binary classification problem. It determines whether the generated image is real without considering the accuracy and stylization of the generation. The discriminator D consists of a convolutional layer, 6 residual blocks and an output layer for judging two categories. The specific structure is as follows Figure 4 As shown, the discriminant loss L D The formula can be expressed as:

[0091]

[0092] Where E is the expectation, χ is the training set, and x is the sample collected in the training set χ; The set of samples generated for the image generator, For the sample set The samples collected in , D(x) is the true predicted probability distribution made by the discriminator D for sample x, is the discriminator D for the sample The probability distribution of the true predictions made.

[0093] The author classifier W is used to determine whether the style of the generated handwriting sample is consistent with the style of the reference author style image. It is a supervised image multi-classification problem. It determines which author the generated handwriting sample style belongs to. The structure is similar to the discriminator D, except that the output layer determines the category and changes it to the number of authors. The writing style loss L W The formula can be expressed as:

[0094]

[0095] in, is the author prediction probability distribution made by the author classifier W for sample x, that is, the probability that the sample x predicted by the author classifier W is the handwritten sample of the kth author; when training, the author classifier W is optimized only with the real samples collected in the training set χ, at this time x ~ χ, when used to guide the training of the image generator, the parameters of the author classifier W are fixed, at this time Only generated using image generators The samples in the set are then back-propagated to the image generator, the parameters of the image generator are updated, and alternating iterative training of the generative adversarial network GANs, i.e., the image generator, is performed; w k is the true probability distribution of the kth author, w k It is in matrix form, where the element value at the kth position is 1 and the element values ​​at other positions are 0, and |Z| is the number of authors.

[0096] The word recognizer R is used to determine whether the generated handwriting sample is consistent with the input text content. The generated handwriting sample is recognized using the CRNN model and then compared with the input target text word. The text content loss L R The formula can be expressed as:

[0097]

[0098] in, is the probability distribution of the word recognizer R identifying the i-th character in sample x as the j-th character in the character set A. When generating English offline handwriting samples, the character set A contains the uppercase and lowercase forms of the 26 English letters. i,j yes The corresponding true probability, for example, for the word an, the word recognizer R recognizes the first position as a with a probability of 80% and the probability of recognizing it as b with 20%, then the corresponding And t i,j is certain, so t 1,1 =1,t 1,2 =0; l represents the maximum length of a word in the current training set, and |A| represents the total number of characters in character set A.

[0099] This example uses the IAM dataset for training and testing. This dataset was collected and released by the Computer Vision and Artificial Intelligence Research Group at the University of Bern in 2002. It consists of 9,862 lines of text, approximately 63,000 English words, and style images from 500 different authors. The dataset provides mutually exclusive author segmentation, with style images from 340 authors used for training and style images from the remaining authors used for testing. The word text includes all words originally in the dataset, and to demonstrate the reliability of the method, several hundred words not present in the dataset were also added.

[0100] This embodiment can generate the required handwriting samples based on the input word text and author style pictures. The specific experimental results are as follows: Figure 5 As shown, Figure 5 Each column in the example represents a handwriting sample from a different author. The first three rows are referenced author style images, and the last four rows are the input word text and the generated corresponding handwriting sample. Analysis of the experimental results shows that the handwriting samples generated by this embodiment have high word accuracy, a prominent author style, and good generation results.

[0101] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A method for generating offline handwriting samples based on multi-feature fusion, characterized in that: The following steps are involved: Step S1: construct a handwriting sample generation model based on multi-feature fusion; Step S2: training a handwriting sample generation model based on multi-feature fusion; Step S3: input the word text to be generated and the reference author style image or author ID into the trained multi-feature fusion-based handwriting sample generation model to generate a handwriting sample image; The specific implementation process of step S3 is as follows: Step S31: Input the word text to be generated and the reference author style image or author ID. If the author ID is input, call the author style image through the author ID, and pre-process the author style image directly input or called through the author ID to fix the size of the author style image; Step S32: extract local style features, global style features, local content features and global content features from the fixed-size author style image to obtain the global style code F S , local style encoding F Sn , Global text content encoding F C and local text content encoding α n and β n , n={1,2,3}; Step S33: Encode the global style F S With global text content encoding F C After splicing, we get the style content code F CS , encode the style content F CS , local style encoding F Sn and local text content encoding α n and β n Perform feature fusion at different scales; Step S34: upsampling the feature map obtained in step S33 by fusing features of different scales to generate a handwritten sample image; In step S32, the fixed-size author style image is input into the writing style encoder VGG19, and the writing style encoder VGG19 outputs the global style code F S ; The outputs of the ReLU5-1, ReLU4-1, and ReLU3-1 layers of VGG19 are used as the local style encoding F S1 、F S2 、F S3 , extract the local style code F Sn , n={1,2,3}; The fixed-size author style image is input into the text encoder, which contains two parallel multi-layer perceptrons. The two parallel multi-layer perceptrons output two global text content codes F C and f C , global text content encoding F C With the global style code F S The same size as the global text content encoding f C is a two-dimensional feature matrix that encodes the global text content f C Divide equally into 3 pairs of one-dimensional parameters α n and β n .

2. The method for generating offline handwriting samples based on multi-feature fusion according to claim 1, characterized in that: Step S33 encodes the style content F CS , local style encoding F Sn and local text content encoding α n and β n Input image generator for feature fusion. The specific implementation process of feature fusion is as follows: (1) Encode the style content F CS After upsampling and local style encoding F S1 The fused feature matrix Z is obtained by adding them together, and the fused feature matrix Z is sent to the AdaIN normalization function to be fused with the local text content codes α1 and β1, and the feature code F that combines the global style features, local style features and text content features is output; (2) Use the feature code F as the new style content code F CS , follow step (1) to encode the style content F CS , local style encoding F S2 And local text content encoding α2 and β2 for feature fusion; (3) Take the feature code F output from step (2) as the new style content code F CS , follow step (1) to encode the style content F CS , local style encoding F S3 And local text content encoding α3 and β3 are used for feature fusion, and a feature map that fuses features of different scales is output.

3. The method for generating offline handwriting samples based on multi-feature fusion according to claim 2, characterized in that: Encode the style content F CS After upsampling and local style encoding F Sn The specific operations of adding and fusion of features are as follows: First, encode the style content F CS After upsampling and style encoding F Sn Perform broadcast addition; Then, the broadcast addition result X is fed into the multi-scale channel attention mechanism, which consists of two parts: global channel attention and local channel attention. The local channel attention L(X) can be expressed as: L(X)=BN(PWConv2(δ(BN(PWConv1(X))))); Among them, δ is the RELU activation function, PWConv1 and PWConv2 are two different point convolution layers, and BN is the batch normalization operation; The global channel attention G(X) can be expressed as: G(X)=BN(PWConv2(δ(BN(PWConv1(GAP(X)))))); Among them, GAP represents the global average pooling operation; The results of global channel attention and local channel attention are broadcast and added and then sent to the Sigmoid activation function. The final output result Y is mapped to [0,1]. The output result Y is the weight of feature fusion. Using the weight Y of feature fusion, the feature matrix F is calculated according to the following formula. CS With style code F Sn Weighted sum: Among them, the fusion feature matrix Z is the weighted summation result of the features, Indicates broadcast addition.

4. The method for generating offline handwriting samples based on multi-feature fusion according to claim 2, characterized in that: The AdaIN normalization function can be expressed as: Among them, μ(Z) represents the variance of the fused feature matrix Z, and σ(Z) represents the mean of the fused feature matrix Z.

5. The method for generating offline handwriting samples based on multi-feature fusion according to any one of claims 1 to 4, characterized in that: In step S2, when training the offline handwriting sample generation model based on multi-feature fusion, the generated handwriting sample image is input to the discriminator D, the author classifier W and the word recognizer R respectively to determine the discrimination loss L D , writing style loss L W and text content loss L R , and back-propagates the error to the image generator of the offline handwriting sample generation model based on multi-feature fusion to update the parameters of the image generator.

6. The method for generating offline handwriting samples based on multi-feature fusion according to claim 5, characterized in that: Discriminative loss L D The formula is: Where E is the expectation, χ is the training set, and x is the sample collected in the training set χ; A collection of samples generated by a handwriting sample generation model based on multi-feature fusion. For the sample set The samples collected in , D(x) is the true predicted probability distribution made by the discriminator D for sample x, is the discriminator D for the sample The probability distribution of the true predictions made.

7. The method for generating offline handwriting samples based on multi-feature fusion according to claim 5, characterized in that: Writing style loss L W The formula is: Where E is the expectation, χ is the training set, and x is the sample collected in the training set χ; A collection of samples generated by a handwriting sample generation model based on multi-feature fusion. For the sample set Samples collected in is the probability that the sample x predicted by the author classifier W is the handwritten sample of the kth author. During training, the author classifier W is optimized only with real samples collected from the training set χ. At this time, x~χ. When used to guide the training of the image generator, the parameters of the author classifier W are fixed. Only generated using image generators Samples in the set; w k is the true probability distribution of the kth author, and |Z| is the number of authors.

8. The method for generating offline handwriting samples based on multi-feature fusion according to claim 5, characterized in that: Text content loss L R The formula is: Where E is the expectation, χ is the training set, and x is the sample collected in the training set χ; The set of samples generated for the image generator, For the sample set Samples collected in is the probability distribution of the word recognizer R identifying the i-th character in sample x as the j-th character in the character set A, t i,j yes The corresponding true probability; l represents the maximum length of a word in the current training set, and |A| represents the total number of characters in the character set A.

Images