Handwritten text image generation method, device, electronic device and storage medium

Abstract

The present invention provides a method, device, electronic device and storage medium for generating a handwritten text image, wherein the method comprises: determining a content text and a reference handwritten text image; based on a handwritten text image generation model, applying the positional relationship between adjacent characters in the reference handwritten text image, respectively extracting features from the content text and the reference handwritten text image, and applying the extracted features to generate a target handwritten text image. Compared with the character-by-character generation method, the handwritten text image generation method, device, electronic device and storage medium provided by the embodiments of the present invention fully consider the structural correlation between two adjacent characters in the reference handwritten text image, thereby being able to obtain a target handwritten text image of higher quality, and improving the visual rationality of handwritten text image generation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device, electronic device and storage medium for generating a handwritten text image. Background Art

[0002] In the field of handwritten text generation, rendering handwritten text images with diverse handwriting styles has attracted more and more attention. Generated handwritten text images can not only reduce the burden on font designers, but also increase the data for optical character recognition tasks and reduce data annotation costs.

[0003] At present, the target handwritten text image with the source text content is mainly obtained by character-by-character font style transfer. However, for text lines containing multiple characters or artistic characters with coupled characters, the character-by-character font style transfer method often loses some features, resulting in low quality of the generated text image and affecting the performance improvement of the optical character recognition task.

[0004] Therefore, how to generate high-quality handwritten text images is an urgent problem to be solved. Summary of the invention

[0005] The present invention provides a handwritten text image generation method, device, electronic device and storage medium, which are used to solve the defect of low quality of text images generated in the prior art.

[0006] The present invention provides a method for generating a handwritten text image, comprising:

[0007] Determine the content text and reference handwritten text image;

[0008] Based on the handwritten text image generation model, applying the positional relationship between adjacent characters in the reference handwritten text image, respectively extracting features from the content text and the reference handwritten text image, and applying the extracted features to generate a target handwritten text image;

[0009] The handwritten text image generation model is trained based on sample content text, sample reference handwritten text images and label handwritten text images.

[0010] According to the handwritten text image generation method provided by the present invention, the handwritten text image generation model is based on the position relationship between adjacent characters in the reference handwritten text image, and the content text and the reference handwritten text image are respectively extracted. The extracted features are used to generate a target handwritten text image, including:

[0011] Based on the first recognizer in the handwritten text image generation model, the content and position of the characters in the reference handwritten text image are recognized to obtain the character positions and blank character positions of the reference handwritten text image;

[0012] Based on the content feature adjustment layer in the handwritten text image generation model, the feature map of the blank character position of the reference handwritten text image is applied to adjust the original content features of the content text to obtain the content features of the content text;

[0013] Based on the style feature adjustment layer in the handwritten text image generation model, the feature map of the character position of the reference handwritten text image is applied to adjust the original style features of the reference handwritten text image to obtain the style features of the reference handwritten text image;

[0014] Based on the decoder in the handwritten text image generation model, the content features and style features are applied to generate the target handwritten text image.

[0015] According to the handwritten text image generation method provided by the present invention, the content feature adjustment layer in the handwritten text image generation model is based on the blank character position feature of the reference handwritten text image to adjust the original content feature of the content text to obtain the content feature of the content text, including:

[0016] Based on the spacing feature extraction layer in the content feature adjustment layer, performing spacing feature extraction on the feature map of the blank character position to obtain spacing features between adjacent characters in the reference handwritten text image;

[0017] Based on the spacing feature adjustment layer in the content feature adjustment layer, the spacing feature is applied to adjust the spacing between adjacent characters in the original content feature to obtain the content feature of the content text.

[0018] According to the handwritten text image generation method provided by the present invention, the content feature adjustment layer in the handwritten text image generation model is based on the feature map of the blank character position of the reference handwritten text image to adjust the original content features of the content text to obtain the content features of the content text, and the content features of the content text are obtained, which also includes:

[0019] Determining a component index and a structure index of each character in the content text based on a component structure dictionary in the handwritten text image generation model;

[0020] Based on the content encoder in the handwritten text image generation model, the component index and structure index of each character in the content text are applied to perform content encoding on the content text to obtain the original content features of the content text.

[0021] According to the handwritten text image generation method provided by the present invention, the content encoder based on the handwritten text image generation model applies the component index and structure index of each character in the content text to perform content encoding on the content text to obtain the original content features of the content text, including:

[0022] Based on the character embedding layer in the content encoder, applying the component index and the structure index of each character in the content text, content encoding is performed on each character of the content text to obtain the character embedding feature of each character;

[0023] Based on the character splicing layer in the content encoder, the character embedding features of the characters are spliced ​​to obtain the original content features of the content text.

[0024] According to the handwritten text image generation method provided by the present invention, the handwritten text image generation model is trained based on the following steps:

[0025] Inputting the sample content text and the sample reference handwritten text image into the initial model to obtain a generated handwritten text image output by the initial model;

[0026] Based on at least one of the differences in pixel features and writing styles between the generated handwritten text image and the label handwritten text image, the differences in text content between the generated handwritten text image and the content text, and the discriminator's discrimination result for the generated handwritten text image, the initial model is trained to obtain the handwritten text image generation model.

[0027] The present invention also provides a handwritten text image generation device, comprising:

[0028] a text and image determination unit, for determining a content text and a reference handwritten text image;

[0029] A target image generation unit, configured to extract features from the content text and the reference handwritten text image based on a handwritten text image generation model and using the positional relationship between adjacent characters in the reference handwritten text image, and to generate a target handwritten text image using the extracted features;

[0030] The handwritten text image generation model is trained based on sample content text, sample reference handwritten text images and label handwritten text images.

[0031] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned methods for generating a handwritten text image is implemented.

[0032] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the handwritten text image generation method as described in any one of the above is implemented.

[0033] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for generating a handwritten text image as described above is implemented.

[0034] The handwritten text image generation method, device, electronic device and storage medium provided by the present invention are based on a handwritten text image generation model, and use the positional relationship between adjacent characters in a reference handwritten text image to extract features from the content text and the reference handwritten text image, and use the extracted features to generate a target handwritten text image. Compared with the character-by-character generation method, the structural correlation between two adjacent characters in the reference handwritten text image is fully considered, so that a high-quality target handwritten text image can be obtained, and the visual rationality of the handwritten text image generation is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 It is one of the flow charts of the handwritten text image generation method provided by the present invention;

[0037] Figure 2 is a schematic diagram of a component structure dictionary provided by the present invention;

[0038] Figure 3 It is a flow chart of the handwritten text image generation model training method provided by the present invention;

[0039] Figure 4 This is the second flow chart of the method for generating a handwritten text image provided by the present invention;

[0040] Figure 5 It is a structural schematic diagram of the handwritten text image generation device provided by the present invention;

[0041] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] In the prior art, there are mainly the following technologies for generating handwritten text images:

[0044] (1) zi2zi has started the work of font style transfer driven by generative models. In Zi2zi, the source font obtains content embedding in the latent space through the content encoder, and the source font is concatenated with the untrainable Gaussian noise representing the target font and passes through the decoder network to generate the target font with the corresponding content. However, zi2zi can only stylize the target font with a large amount of training data, and the transfer of new font styles in real scenarios is what we are interested in.

[0045] (2) EMD and CalliGAN are dedicated to the transfer of new font styles that have not been trained with generative models. Specifically, EMD inherits the classic U-Net structure in zi2zi as the source font content encoder and decoder, and realizes the transfer of new font styles by adding a target font style encoder and a mixer network in parallel with the content encoder. The mixer network replaces the simple vector concatenation operation and learns the interaction between style embedding and content embedding. CalliGAN increases the training samples in disguise by decomposing the radicals of Chinese characters. On the basis of EMD, a radical sequence encoding network is added, and the obtained radical sequence encoding is concatenated with the content embedding and style embedding through the decoder to obtain the target font with the corresponding content. In the training stage, the above models are optimized using reconstruction loss and adversarial loss. In the inference stage, EMD and CalliGAN can directly input the source font and target font to obtain the target font with the content of the source font.

[0046] Although the above method has achieved the best performance, for handwritten text lines containing multiple characters or artistic words with characters coupled with each other, the existing character-by-character generation method often loses some features and cannot generate high-quality handwritten text images.

[0047] Based on this, an embodiment of the present invention provides a method for generating a handwritten text image. The method can be applied to a scenario of generating a handwritten text image with any text length and target writing style.

[0048] Figure 1This is one of the flow charts of the handwritten text image generation method provided by the present invention. The method can be executed by any device that executes the handwritten text image generation method. The device can be implemented by software and / or hardware. In this embodiment, the device can be integrated in an electronic device. Figure 1 As shown, the handwritten text image generation method provided by the embodiment of the present invention includes:

[0049] Step 110: Determine the content text and the reference handwritten text image.

[0050] Specifically, the content text may be a single character or a text line containing multiple characters, such as a word, a sentence or a paragraph.

[0051] The content text may be a text line obtained by Unicode variable-length character encoding (Unicode Transformation Format, UTF-8), or may be content text obtained by using optical character recognition (Optical Character Recognition, OCR) for an image containing characters, which is not specifically limited in the embodiment of the present invention.

[0052] Furthermore, the content text may be in Chinese.

[0053] The reference handwritten text image is an image containing handwritten text, which can be the handwritten text acquired through an image acquisition device such as a scanner, a mobile phone, or a camera.

[0054] The reference handwritten text image may be a handwritten text image of any writer, for example, it may be a handwritten text image of Zhang San or a handwritten text image of Li Si. The text content included in the reference handwritten text image may have nothing to do with the text content included in the content text.

[0055] Step 120 , based on the handwritten text image generation model, using the positional relationship between adjacent characters in the reference handwritten text image, respectively extracting features from the content text and the reference handwritten text image, and using the extracted features to generate a target handwritten text image.

[0056] Among them, the handwritten text image generation model is trained based on sample content text, sample reference handwritten text images and labeled handwritten text images.

[0057] Specifically, the target handwritten text image is a handwritten image that the user expects to obtain. The target handwritten text image has the same text content as the content text, and has the same or similar writing style as the reference handwritten text image.

[0058] The writing style of the reference handwritten text image is not only reflected in the writing style characteristics of individual characters such as font type, font size, font neatness or sloppiness, stroke characteristics, and pen tip trend, but also in the structural correlation and interaction mode between each character.

[0059] However, in the prior art, the structural correlation between two adjacent characters is often lost through the character-by-character generation method, resulting in low quality of the generated handwritten image.

[0060] Therefore, the method provided in an embodiment of the present invention is based on a handwritten text image generation model, applies the positional relationship between adjacent characters in a reference handwritten text image, performs feature extraction on the content text and the reference handwritten text image respectively, and applies the extracted features to generate a target handwritten text image.

[0061] Here, referring to the positional relationship between adjacent characters in the handwritten text image, the structural correlation between adjacent characters can be represented, and specifically may include the positional relationship between adjacent characters in various directions, such as the horizontal positional relationship and the vertical positional relationship.

[0062] The target handwritten text image thus obtained fully considers the structural correlation between two adjacent characters in the reference handwritten text image, thereby being able to obtain a target handwritten text image with higher quality.

[0063] Before executing step 120, a handwritten text image generation model can also be pre-trained. Specifically, the handwritten text image generation model can be trained in the following manner: collect a large number of sample reference handwritten text images, sample content texts, and corresponding labeled handwritten text images; extract relevant features from the sample content texts and sample reference handwritten text images, and generate a handwritten text image based on the extracted features, which is consistent with the sample content text in terms of content and similar or close to the reference handwritten text image in terms of writing style. Then, calculate the loss of the generated handwritten text image and the labeled handwritten text image, and perform supervised training on the initial model to obtain the handwritten text image generation model.

[0064] The handwritten text image generation method provided by the embodiment of the present invention is based on the handwritten text image generation model, applies the positional relationship between adjacent characters in the reference handwritten text image, respectively extracts features from the content text and the reference handwritten text image, and applies the extracted features to generate the target handwritten text image. Compared with the character-by-character generation method, the structural correlation between two adjacent characters in the reference handwritten text image is fully considered, so that a higher-quality target handwritten text image can be obtained, and the visual rationality of the handwritten text image generation is improved.

[0065] Based on the above embodiment, the handwritten text image generation model includes a first recognizer, a content feature adjustment layer, a style feature adjustment layer and a decoder. Accordingly, step 120 specifically includes:

[0066] Step 121 , based on the first recognizer in the handwritten text image generation model, recognize the content and position of the characters in the reference handwritten text image to obtain the character positions and blank character positions of the reference handwritten text image.

[0067] Specifically, the first recognizer may be a pre-trained CRNN network, which may be mainly composed of seven convolutional layers and four pooling layers. Finally, a fully connected layer is used to output the character score of each character in the reference handwritten text image.

[0068] The first recognizer can be used to recognize the content and position of characters in the reference handwritten text image. The reference handwritten text image is input into the first recognizer to obtain the predicted characters at each position output by the first recognizer, and retain the blank characters. The blank characters here are the blank characters between adjacent characters. Thus, the character positions and blank character positions of the reference handwritten text image are obtained. The character positions and blank character positions can characterize the structural correlation between adjacent characters in the reference handwritten text image, and further reflect the writing style of the reference handwritten text image.

[0069] Step 122, based on the content feature adjustment layer in the handwritten text image generation model, the original content features of the content text are adjusted by applying the feature map of the blank character positions of the reference handwritten text image to obtain the content features of the content text.

[0070] Specifically, the original content features of the content text can represent the text content contained in the content text, such as "a 180-degree turn" or "the weather is fine today". Specifically, the original content features of the content text can be extracted by converting each character in the content text into a feature space through a content feature extraction network. Among them, the content feature extraction network can be a deep learning neural network, such as a fully convolutional neural network based on Vgg19, or a residual network Resnet, etc., which is not specifically limited in the embodiments of the present invention.

[0071] Considering that each character embedding in the original content feature has the same receptive field width, which is inconsistent with the fact of Chinese text writing, the feature map of the blank character position of the reference handwritten text image can be applied to adjust the original content feature to obtain the content feature of the content text.

[0072] The content features obtained after adjustment combine the blank character position features of the reference handwritten text image and fully consider the structural correlation between adjacent characters in the reference handwritten text image. The content features obtained in this way can not only represent the content of the content text, but also represent the structural correlation between adjacent characters in the reference handwritten text image.

[0073] Step 123, based on the style feature adjustment layer in the handwritten text image generation model, the feature map of the character position of the reference handwritten text image is applied to adjust the original style features of the reference handwritten text image to obtain the style features of the reference handwritten text image.

[0074] Specifically, the original style features of the reference handwritten text image can characterize the writing style of each character of the reference handwritten text image. Specifically, the image feature space of the reference handwritten text image can be converted into the style feature space through a style encoder to extract the original style features of the reference handwritten text image.

[0075] On the basis of obtaining the original style features, the original style features are adjusted in combination with the character positions output by the first recognizer to obtain the style features of the reference handwritten text image. Specifically, the character positions output by the first recognizer can be aligned with the output of the style encoder in width, thereby outputting the original style features corresponding to the character positions as the style features of the reference handwritten text image. The style features thus obtained can more truly reflect the style features of the writing style of the reference handwritten text image.

[0076] Step 124, based on the decoder in the handwritten text image generation model, apply the content features and style features to generate a target handwritten text image.

[0077] Specifically, on the basis of obtaining the content features and style features, the target handwritten text image can be generated according to the content features and style features. For example, the content features and style features can be fused to obtain fused features, and the target handwritten text image can be generated according to the fused features; the content features and style features can also be connected together along the channel and sent to the decoder to generate the target handwritten text image, so as to achieve the effect of handwriting imitation.

[0078] The method provided by the embodiment of the present invention obtains the character positions and blank character positions of the reference handwritten text image by recognizing the reference handwritten text image, and adjusts the original content features and the original style features based on the obtained content features and style features after the adjustment, and generates the target handwritten text image according to the content features and style features obtained after the adjustment. The method can improve the quality and visual rationality of the generated handwritten text image.

[0079] Based on any of the above embodiments, the content feature adjustment layer may specifically include a spacing feature extraction layer and a spacing feature adjustment layer. Accordingly, step 122 specifically includes:

[0080] Step 122-1, based on the spacing feature extraction layer in the content feature adjustment layer, extracting spacing features from the feature map of the blank character position, to obtain spacing features between adjacent characters in the reference handwritten text image;

[0081] Step 122 - 2 , based on the spacing feature adjustment layer in the content feature adjustment layer, apply the spacing feature to adjust the spacing between adjacent characters in the original content feature to obtain the content feature of the content text.

[0082] Specifically, based on the feature map of the blank character position of the reference handwritten text image, the spacing feature between adjacent characters in the reference handwritten text image can be extracted according to the feature map of the blank character position. Specifically, the feature map of the blank character position can be globally pooled to obtain the spacing feature. The spacing feature can reflect the character spacing between adjacent characters in the reference handwritten text image. Among them, the spacing feature can be controlled by the ratio of blank characters to all characters.

[0083] After obtaining the spacing feature, the spacing feature can be applied to adjust the spacing between adjacent characters in the original content feature. Specifically, the spacing feature can be inserted between the character embeddings of the original content feature to obtain the final content feature.

[0084] Based on any of the above embodiments, before step 122, the following steps may also be included:

[0085] Step 210, determining the component index and structure index of each character in the content text based on the component structure dictionary in the handwritten text image generation model;

[0086] Step 220 , based on the content encoder in the handwritten text image generation model, the component index and structure index of each character in the content text are applied to perform content encoding on the content text to obtain the original content features of the content text.

[0087] Specifically, the handwritten text image generation model also includes a component structure dictionary and a content encoder. Considering the problem of model parameter explosion that may be caused by thousands of complex structure Chinese characters when encoding each character in the content text, the embodiment of the present invention provides a more informative encoding method for Chinese characters through a component decomposition and reuse mechanism. By obtaining the content representation of Chinese characters at the component level, a new way is provided to expand Chinese characters with a small sample size.

[0088] It should be noted that although there are already content representation methods that decompose Chinese characters into radicals, some of them are not learnable and some are deeply coupled, which is not conducive to directly generating variable-length handwritten Chinese text line images. In contrast, the embodiments of the present invention are dedicated to generating handwritten Chinese text images of arbitrary length through variable-length content representation.

[0089] Figure 2 is a schematic diagram of the component structure dictionary provided by the present invention. As Figure 2 shown, the component structure dictionary contains the component index and structure index of each character. For example, the character "森" can be split into three "木". Here, the components are three "木". Looking up the index of "木" in the component structure dictionary, the index of the component "木" is found to be 33. Then, the component indexes of the character "森" are 33, 33, 33 respectively, and the structure index of "森" is found to be 1.

[0090] For another example, the character "想" can be split into components "木", "目", and "心". At the same time, looking up in the component structure dictionary, the component indexes are found to be 33, 48, and 66 respectively, and the structure index of "想" is found to be 4.

[0091] Based on the component index and structure index of each character, the content text can be content-encoded according to the component index and structure index of each character to obtain the original content features of the content text.

[0092] It should be noted that the component index and structure index here belong to two different feature spaces.

[0093] The method provided by the embodiment of the present invention can directly generate a variable-length content text by encoding Chinese characters through component and structure combination, so as to obtain a handwritten text image of any length. In addition, this method can also obtain a content text in which characters are decoupled from each other, further improving the generation efficiency and quality of the handwritten text image.

[0094] Based on any of the above embodiments, step 220 specifically includes:

[0095] Step 221, based on the character embedding layer in the content encoder, applying the component index and structure index of each character in the content text to perform content encoding on each character of the content text to obtain the character embedding features of each character;

[0096] Step 222, based on the character splicing layer in the content encoder, splice the character embedding features of each character to obtain the original content features of the content text.

[0097] Specifically, based on the component index and structure index of each character obtained, the obtained component index and structure index are respectively embedded into the latent space to obtain 256-D latent vectors. By combining the latent vectors of components and structures, the content representation of a single character in the content text, that is, the character embedding feature, can be obtained.

[0098] Splicing the character embedding features of each character in the content text in the width dimension can obtain the original content features of the content text.

[0099] Based on any of the above embodiments, Figure 3 : is a flow chart of the handwritten text image generation model training method provided by the present invention, such as Figure 3 As shown in the figure, the handwritten text image generation model is trained based on the following steps:

[0100] Step 310, inputting the sample content text and the sample reference handwritten text image into the initial model to obtain a generated handwritten text image output by the initial model;

[0101] Step 320, based on the difference in pixel features and writing style between the generated handwritten text image and the label handwritten text image, the difference in text content between the generated handwritten text image and the content text, and at least one of the discriminator's discrimination results for the generated handwritten text image, the initial model is trained to obtain a handwritten text image generation model.

[0102] Specifically, the sample content text and the sample reference handwritten text image may be first input into the initial model to obtain the generated handwritten text image output by the initial model.

[0103] In order to generate handwritten images with the same text content and the same or similar writing style, at least one of the following four types of loss functions can be used in the iterative optimization process to minimize the loss of generating handwritten text images.

[0104] The discriminator and decoder are used for adversarial training. The discriminator can be composed of a four-layer residual module. The final output layer uses a global pooling plus a fully connected layer to output the discrimination result for generating handwritten text images.

[0105] (1) Adversarial loss: In order to generate images that are closer to the real data distribution, the adversarial training mechanism is used to train the initial model. Hinge loss can be used as the optimization objective. The loss function can be expressed as follows:

[0106] L D =E(max(0,1-D(I s ))+E(max(0,1+D(G(Z s , Z c ))

[0107] L G =-E(D(G(Z s , Z c )))

[0108] In the formula, I s represents the reference handwritten text image, D represents the discriminator, G represents the decoder, and Z s Indicates style characteristics, Z c Represents content features, G(Zs , Z c ) represents the generation of handwritten text images, D(G(Z s , Z c )) represents the discriminator's discriminant result for generating handwritten text images.

[0109] (2) Reconstruction loss: Although realistic and sharp images can be obtained by only using adversarial loss, there are unreasonable omissions or additions. In order to encourage the initial model to generate text images that respect the real input, the reconstruction loss function is used to measure the difference in pixel features between the generated handwritten text image and the labeled handwritten text image. The loss function can be expressed as follows:

[0110] L rec =||I c，s -O c，s ||1

[0111] In the formula, I c，s represents the label handwritten text image, that is, the real image; O c，s It means generating a handwritten text image, that is, generating an image.

[0112] (3) Perceptual loss: This loss is calculated by the intermediate feature layer of the pre-trained second recognizer. Since the reconstruction loss calculates the average error of pixels, it is easy to cause local blur. In order to capture more feature details, the perceptual loss calculates the reconstruction loss of the feature layer and its gram matrix in the writing style after the activation function of each feature layer.

[0113]

[0114]

[0115]

[0116] Where, L feat represents the loss of the feature layer, φ i is the matrix mapping corresponding to the i-th layer feature, L style represents the gram matrix loss of the i-th layer feature, is the gram matrix of the i-th layer. i is the normalization factor of the i-th layer.

[0117] (4) Content consistency loss: To generate text images whose content is recognizable and not disturbed by the style of the image. In order to constrain the generated image O c，s To maintain consistency with the content text Ic, the connectionist temporal classification (CTC) loss function can be used:

[0118] LR =CTC(O c，s , I c )

[0119] In one embodiment, a pre-trained second recognizer may be used to measure the difference in text content between the generated handwritten text image and the content text, and to measure the difference in writing style between the generated handwritten text image and the label handwritten text image.

[0120] Furthermore, the final optimization goal of the model training method provided by the embodiment of the present invention can be written as:

[0121] L(G, D, R) = λ1L D / G +λ2L rec +λ3L feat +λ4L style +λ5L R

[0122] Among them, λ1λ2λ3λ4λ5 are adjustable hyperparameters that control the weights of each item in the above formula. It should be noted that the first recognizer and the second recognizer are both pre-trained networks and share weights.

[0123] The method provided by the embodiment of the present invention captures different interaction modes between characters in different handwritings through perceptual style loss and content consistency loss. In addition, adversarial loss is used to train the model to obtain more realistic and sharper images.

[0124] Based on any of the above embodiments, Figure 4 FIG. 2 is a flow chart of the method for generating a handwritten text image provided by the present invention. Figure 4 As shown, an embodiment of the present invention provides a method for generating a handwritten text image, comprising:

[0125] S1, determine the content text I c and the reference handwritten text image I s ;

[0126] S2, based on the handwritten text image generation model, the content text I c and the reference handwritten text image I s Perform feature extraction to obtain content feature Z c and style features Z s Then, the content feature Z is transformed along the channel c and style features Z s They are connected together and sent to the decoder G to generate the target handwritten text image, in this way the function of handwriting imitation can be achieved.

[0127] Among them, the handwritten text image generation model mainly includes the following five sub-networks: component structure dictionary CS, content encoder Ec , style encoder E s , decoder G and a first identifier R1.

[0128] Component structure dictionary CS: It is used to obtain the component index and structure index of each character based on the component structure dictionary CS given a content text.

[0129] Content Encoder E c : The component index and structure index are embedded into the latent space to obtain a 256-D latent vector. By combining the latent vectors of the component and structure, the character embedding feature of a single character can be obtained. The original content feature of the content text, Raw code, is the concatenation of the character embedding features of a single character in the width dimension.

[0130] Style Encoder E s :The network is mainly composed of four layers of residual modules. The residual module consists of residual mapping and identity mapping, where residual mapping contains two layers of 3×3 convolutional layers and identity mapping consists of one layer of 1×1 convolutional layer. In addition, ReLu and spectral normalization are used after each module. The average pooling operation is used to perform downsampling to align the output of the last layer of residual module with the output of the first recognizer in the width dimension.

[0131] First recognizer R1: The pre-trained CRNN network is selected as the recognizer. It is mainly composed of seven convolutional layers and four pooling layers. Finally, a fully connected layer is used to output the category score. The output width of CRNN is set to be equal to E s Output alignment, deriving style feature Z from the output of R1 s The style image passes through R1, outputs the predicted characters at each position, and retains the blank characters. The style feature Z is obtained by global pooling in the feature map of the character position. s , the feature map of the blank character position is globally pooled to obtain the spacing feature. The spacing feature is inserted between the single character embeddings of the original content feature, and the spacing is controlled by the ratio of blank characters to all characters to obtain the final content feature Z c .

[0132] The formula can be expressed as follows:

[0133] Rawcode=E c (CS(I c ))

[0134] Z c = Rawcode + R1(I s )

[0135] Z s =Es (I s )+R1(I s )

[0136] Decoder G: In this network, four layers of residual modules are introduced, and the structure can be consistent with the residual modules in the style encoder. In addition, nearest neighbor interpolation is used to perform upsampling operations. After the last layer of residual modules, a convolutional layer is used to output the final image.

[0137] The generation process can be written as: c，s =G(Z c , Z s )

[0138] However, the above formula does not guarantee that Z c and Z s The handwritten text image generated by the decoder successfully models the different structural relationships between characters in different writing styles. To this end, perceptual loss and CTC loss are introduced.

[0139] The original perceptual loss uses the pre-trained classification network VGG16, which is usually applied to the style transfer task of natural images. However, font style transfer not only includes texture, but also includes structural associations between characters and topological changes. In order to better transfer the image style containing sequence targets, the embodiment of the present invention introduces a second recognizer R2 to calculate the perceptual loss and CTC loss. Combining reconstruction loss and adversarial loss, a multi-task joint training method is adopted, and a handwritten Chinese text line image with specified content and style can be generated. Among them,

[0140] Discriminator D: Similar to the style encoder, this network also consists of four layers of residual modules. The final output layer uses global pooling plus a fully connected layer to output the discriminative score.

[0141] The second identifier R2 has the same structure as the first identifier R1 and shares the same weights.

[0142] The handwritten text image generation model proposed in the embodiment of the present invention can imitate the handwriting style of a handwritten Chinese text line to generate new content. Different from single-character font style transfer, this model can not only generate sequence characters of arbitrary length, but also model the spacing and interaction between adjacent characters of different handwriting styles. A more informative text line content representation is obtained by using component decomposition. At the same time, the reference handwritten text image is decomposed into a style representation and a spacing vector, which respectively regulate the font style of the generated text line image and the interaction between characters. In this way, the model can directly generate a target image with specified text line content and style.

[0143] The handwritten text image generation device provided by the present invention is described below. The handwritten text image generation device described below and the handwritten text image generation method described above can be referred to each other.

[0144] Figure 5 is a schematic diagram of the structure of a handwritten text image generating device provided by an embodiment of the present invention. Figure 5 As shown, the device comprises:

[0145] A text and image determination unit 510, configured to determine a content text and a reference handwritten text image;

[0146] A target image generation unit 520 is used to extract features from the content text and the reference handwritten text image based on a handwritten text image generation model and apply the positional relationship between adjacent characters in the reference handwritten text image, and generate a target handwritten text image using the extracted features;

[0147] The handwritten text image generation model is trained based on sample content text, sample reference handwritten text images and label handwritten text images.

[0148] The handwritten text image generation device provided by the embodiment of the present invention is based on the handwritten text image generation model, applies the positional relationship between adjacent characters in the reference handwritten text image, respectively extracts features from the content text and the reference handwritten text image, and generates the target handwritten text image using the extracted features. Compared with the character-by-character generation method, the structural correlation between two adjacent characters in the reference handwritten text image is fully considered, so that a higher-quality target handwritten text image can be obtained, and the visual rationality of the handwritten text image generation is improved.

[0149] Based on any of the above embodiments, the target image generating unit 520 is further configured to:

[0150] Based on the first recognizer in the handwritten text image generation model, the content and position of the characters in the reference handwritten text image are recognized to obtain the character positions and blank character positions of the reference handwritten text image;

[0151] Based on the content feature adjustment layer in the handwritten text image generation model, the feature map of the blank character position of the reference handwritten text image is applied to adjust the original content features of the content text to obtain the content features of the content text;

[0152] Based on the style feature adjustment layer in the handwritten text image generation model, the feature map of the character position of the reference handwritten text image is applied to adjust the original style features of the reference handwritten text image to obtain the style features of the reference handwritten text image;

[0153] Based on the decoder in the handwritten text image generation model, the content features and style features are applied to generate the target handwritten text image.

[0154] Based on any of the above embodiments, the target image generating unit 520 is further configured to:

[0155] Based on the spacing feature extraction layer in the content feature adjustment layer, performing spacing feature extraction on the feature map of the blank character position to obtain spacing features between adjacent characters in the reference handwritten text image;

[0156] Based on the spacing feature adjustment layer in the content feature adjustment layer, the spacing feature is applied to adjust the spacing between adjacent characters in the original content feature to obtain the content feature of the content text.

[0157] Based on any of the above embodiments, the text image generating device further includes an original content feature generating unit, which is used to:

[0158] Determining a component index and a structure index of each character in the content text based on a component structure dictionary in the handwritten text image generation model;

[0159] Based on the content encoder in the handwritten text image generation model, the component index and structure index of each character in the content text are applied to perform content encoding on the content text to obtain the original content features of the content text.

[0160] Based on any of the above embodiments, the original content feature generating unit is further used to:

[0161] Based on the character embedding layer in the content encoder, the component index and the structure index of each character in the content text are applied, and each character of the content text is content encoded to obtain the character embedding feature of each character;

[0162] Based on the character splicing layer in the content encoder, the character embedding features of the characters are spliced ​​to obtain the original content features of the content text.

[0163] Based on any of the above embodiments, the text image generation device further includes a model training unit, which is used to:

[0164] Inputting the sample content text and the sample reference handwritten text image into the initial model to obtain a generated handwritten text image output by the initial model;

[0165] Based on at least one of the differences in text content, writing style, and pixel features between the generated handwritten text image and the label handwritten text image, and a discriminator's discrimination result for the generated handwritten text image, the initial model is trained to obtain the handwritten text image generation model.

[0166] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the text image generation method, which includes: determining the content text and the reference handwritten text image; based on the handwritten text image generation model, applying the position relationship between adjacent characters in the reference handwritten text image, respectively extracting features from the content text and the reference handwritten text image, and applying the extracted features to generate the target handwritten text image; wherein the handwritten text image generation model is trained based on the sample content text, the sample reference handwritten text image and the label handwritten text image.

[0167] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0168] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the text image generation method provided by the above-mentioned methods, which includes: determining a content text and a reference handwritten text image; based on a handwritten text image generation model, applying the positional relationship between adjacent characters in the reference handwritten text image, respectively extracting features from the content text and the reference handwritten text image, and applying the extracted features to generate a target handwritten text image; wherein the handwritten text image generation model is trained based on sample content text, sample reference handwritten text images and labeled handwritten text images.

[0169] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the text image generation method provided by the above-mentioned methods, the method comprising: determining a content text and a reference handwritten text image; based on a handwritten text image generation model, applying the positional relationship between adjacent characters in the reference handwritten text image, respectively extracting features from the content text and the reference handwritten text image, and applying the extracted features to generate a target handwritten text image; wherein the handwritten text image generation model is trained based on sample content text, sample reference handwritten text images, and labeled handwritten text images.

[0170] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0171] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a handwritten text image, characterized in that: include: Determine the content text and reference handwritten text image; Based on the first recognizer in the handwritten text image generation model, the content and position of the characters in the reference handwritten text image are recognized to obtain the character positions and blank character positions of the reference handwritten text image; Based on the content feature adjustment layer in the handwritten text image generation model, the feature map of the blank character position of the reference handwritten text image is applied to adjust the original content features of the content text to obtain the content features of the content text; Based on the style feature adjustment layer in the handwritten text image generation model, the feature map of the character position of the reference handwritten text image is applied to adjust the original style features of the reference handwritten text image to obtain the style features of the reference handwritten text image; Based on the decoder in the handwritten text image generation model, applying the content features and style features to generate a target handwritten text image; Wherein, the handwritten text image generation model is obtained by training based on sample content text, sample reference handwritten text image and label handwritten text image; The content feature adjustment layer in the handwritten text image generation model is based on the feature map of the blank character position of the reference handwritten text image to adjust the original content features of the content text to obtain the content features of the content text, including: Based on the spacing feature extraction layer in the content feature adjustment layer, performing spacing feature extraction on the feature map of the blank character position to obtain spacing features between adjacent characters in the reference handwritten text image; Based on the spacing feature adjustment layer in the content feature adjustment layer, the spacing feature is applied to adjust the spacing between adjacent characters in the original content feature to obtain the content feature of the content text.

2. The handwritten text image generation method according to claim 1, characterized in that: The content feature adjustment layer in the handwritten text image generation model is based on the feature map of the blank character position of the reference handwritten text image to adjust the original content features of the content text to obtain the content features of the content text, and the above also includes: Determining a component index and a structure index of each character in the content text based on a component structure dictionary in the handwritten text image generation model; Based on the content encoder in the handwritten text image generation model, the component index and structure index of each character in the content text are applied to perform content encoding on the content text to obtain the original content features of the content text.

3. The handwritten text image generation method according to claim 2, characterized in that: The content encoder based on the handwritten text image generation model applies the component index and structure index of each character in the content text to perform content encoding on the content text to obtain the original content features of the content text, including: Based on the character embedding layer in the content encoder, applying the component index and the structure index of each character in the content text, content encoding is performed on each character of the content text to obtain the character embedding feature of each character; Based on the character splicing layer in the content encoder, the character embedding features of the characters are spliced ​​to obtain the original content features of the content text.

4. The method for generating a handwritten text image according to any one of claims 1 to 3, characterized in that: The handwritten text image generation model is trained based on the following steps: Inputting the sample content text and the sample reference handwritten text image into the initial model to obtain a generated handwritten text image output by the initial model; Based on at least one of the differences in pixel features and writing styles between the generated handwritten text image and the label handwritten text image, the differences in text content between the generated handwritten text image and the content text, and the discriminator's discrimination result for the generated handwritten text image, the initial model is trained to obtain the handwritten text image generation model.

5. A handwritten text image generation device, characterized in that: include: a text and image determination unit, for determining a content text and a reference handwritten text image; The target image generating unit is used for: Based on the first recognizer in the handwritten text image generation model, the content and position of the characters in the reference handwritten text image are recognized to obtain the character positions and blank character positions of the reference handwritten text image; Based on the content feature adjustment layer in the handwritten text image generation model, the feature map of the blank character position of the reference handwritten text image is applied to adjust the original content features of the content text to obtain the content features of the content text; Based on the style feature adjustment layer in the handwritten text image generation model, the feature map of the character position of the reference handwritten text image is applied to adjust the original style features of the reference handwritten text image to obtain the style features of the reference handwritten text image; Based on the decoder in the handwritten text image generation model, applying the content features and style features to generate a target handwritten text image; Wherein, the handwritten text image generation model is obtained by training based on sample content text, sample reference handwritten text image and label handwritten text image; The target image generating unit is specifically used for: Based on the spacing feature extraction layer in the content feature adjustment layer, performing spacing feature extraction on the feature map of the blank character position to obtain spacing features between adjacent characters in the reference handwritten text image; Based on the spacing feature adjustment layer in the content feature adjustment layer, the spacing feature is applied to adjust the spacing between adjacent characters in the original content feature to obtain the content feature of the content text.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the handwritten text image generation method according to any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating a handwritten text image according to any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating a handwritten text image according to any one of claims 1 to 4 is implemented.

Images