Handwritten text image generation method, device, equipment and storage medium

Abstract

The present invention provides a handwritten text image generation method, device, equipment and storage medium, comprising: obtaining text data of a handwritten text image to be generated; obtaining style data of the handwritten text image to be generated; inputting the text data and the style data into a preset handwritten text image model to obtain a handwritten text image; wherein the handwritten text image model is trained based on interference labels, training images and their corresponding text labels. The technical solution of the present invention can improve the quality of the synthesized handwritten text image.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device, electronic device and non-transitory computer-readable storage medium for generating a handwritten text image. Background Art

[0002] In handwritten text recognition (HTR) and handwriting verification technology, the correct analysis and digitization of handwritten text is seriously hindered because the style of handwritten text varies from person to person and the writing form is not standardized.

[0003] In order to cope with the irregular characteristics of handwritten text, handwritten text images of various styles can be generated for model training of handwritten text recognition, so that the trained model has higher recognition ability.

[0004] In current handwritten text image generation schemes, there is a common problem of low generated image quality. Summary of the invention

[0005] The present invention provides a handwritten text image generation method, device, electronic device and non-transitory computer-readable storage medium to solve the problem of low quality of generated handwritten text images in related technologies, so as to improve the quality of generated images of handwritten text images.

[0006] The present invention provides a method for generating a handwritten text image, comprising: obtaining text data of a handwritten text image to be generated; obtaining style data of the handwritten text image to be generated; inputting the text data and the style data into a preset handwritten text image model to obtain a handwritten text image; wherein the handwritten text image model is trained based on interference labels, training images and their corresponding text labels.

[0007] According to a handwritten text image generation method provided by the present invention, the handwritten text image model includes a style encoder, a content encoder, a generator and a discriminator, and the training method of the handwritten text image model includes: inputting the training image and the corresponding text label into the style encoder and the content encoder respectively to generate style variables and content variables; inputting the interference label into the content encoder to generate interference content variables; inputting the interference content variables and the style variables into the generator to generate a first text image; inputting the content variables and the style variables into the generator to generate a second text image; calculating the loss value based on the first text image, the second text image, the training image and the text label, and training the handwritten text image model based on the loss value.

[0008] According to a handwritten text image generation method provided by the present invention, the training image and the corresponding text label are respectively input into the content encoder and the style encoder to generate style variables and content variables, including: inputting the text label into the content encoder to obtain the character index of the text label based on the alphabet; generating the content variable according to the one-hot vector of the character index.

[0009] According to a handwritten text image generation method provided by the present invention, the loss function used when calculating the loss value is a weighted sum of at least two single loss functions, and the single loss function includes at least one of the following functions: a KL divergence function, an adversarial loss function, a reconstruction loss function, and a content loss function.

[0010] According to a handwritten text image generation method provided by the present invention, the KL divergence function obtains a loss value based on style variable parameters, and the style variable parameters are generated by the style encoder according to the training image and its corresponding text label; the adversarial loss function generates a loss value based on the style variable, the training image, the first text image and the second text image, and the discrimination result of the discriminator; the reconstruction loss function generates a loss value based on the second text image and the training image; and the content loss function is based on the loss value calculated based on the first text image and the interference label.

[0011] According to a handwritten text image generation method provided by the present invention, the training image and the corresponding text label are respectively input into the content encoder and the style encoder to generate style variables and content variables, including: using the style encoder in the initial image model to sample the training image and then output style variable parameters; inputting the training image into the style encoder to generate style variable parameters; and sampling a preset style variable set through Gaussian distribution according to the style variable parameters to obtain style variables.

[0012] According to a handwritten text image generation method provided by the present invention, the acquiring of style data of the handwritten text image to be generated includes: randomly sampling a preset style variable set through Gaussian distribution to generate the style data.

[0013] The present invention provides a handwritten text image generation device, comprising: a text acquisition unit, used to acquire text data of a handwritten text image to be generated; a style acquisition unit, used to acquire style data of the handwritten text image to be generated; and a generation unit, used to input the text data and the style data into a preset handwritten text image generation model to obtain a handwritten text image; wherein the handwritten text image model is trained based on interference labels, training images and their corresponding text labels.

[0014] According to a handwritten text image generation device provided by the present invention, the handwritten text image model includes a style encoder, a content encoder, a generator and a discriminator, and the device also includes a training unit, and the training unit includes: an encoding subunit, used to input the training image and the corresponding text label into the content encoder and the style encoder respectively to generate content variables and style variables, and input the interference label into the content encoder to generate interference content variables; a generating subunit, used to input the interference content variable and the style variable into the generator to generate a first text image, and input the content variable and the style variable into the generator to generate a second text image; a training subunit, used to calculate a loss value based on the first text image, the second text image, the training image and the text label, and train the handwritten text image model based on the loss value.

[0015] According to a handwritten text image generation device provided by the present invention, the encoding subunit is used to: input the text label to the content encoder to obtain the character index of the text label based on the alphabet; and generate the content variable according to the one-hot vector of the character index.

[0016] According to a handwritten text image generation device provided by the present invention, the loss function used when calculating the loss value is a weighted sum of at least two single loss functions, and the single loss function includes at least one of the following functions: KL divergence function, adversarial loss function, reconstruction loss function and content loss function.

[0017] According to a handwritten text image generation device provided by the present invention, the KL divergence function obtains a loss value based on style variable parameters, and the style variable parameters are generated by the style encoder according to the training image and its corresponding text label; the adversarial loss function generates a loss value based on the style variable, the training image, the first text image and the second text image, and the discrimination result of the discriminator; the reconstruction loss function generates a loss value based on the second text image and the training image; and the content loss function is a loss value calculated based on the first text image and the interference label.

[0018] According to a handwritten text image generation device provided by the present invention, the encoding subunit is also used to: use the style encoder in the initial image model to sample the training image and then output style variable parameters; input the training image to the style encoder to generate style variable parameters; and sample a preset style variable set through Gaussian distribution according to the style variable parameters to obtain style variables.

[0019] According to a handwritten text image generation device provided by the present invention, the style acquisition unit is further used to: generate the style data by randomly sampling a preset style variable set through Gaussian distribution.

[0020] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned methods for generating a handwritten text image are implemented.

[0021] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for generating a handwritten text image are implemented.

[0022] The handwritten text image generation method, device, electronic device and non-transitory computer-readable storage medium provided by the present invention can obtain a handwritten text image with higher quality by obtaining a handwritten text image generation model through training based on interference labels, training images and text labels, and processing the handwritten text image to be generated based on the handwritten text image generation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 It is one of the flow charts of the handwritten text image generation method provided by the present invention;

[0025] Figure 2 This is one of the flow charts of the training handwritten text image generation model provided by the present invention;

[0026] Figure 3 This is the second flow chart of the training handwritten text image generation model provided by the present invention;

[0027] Figure 4 This is the second flow chart of the method for generating a handwritten text image provided by the present invention;

[0028] Figure 5 is a schematic diagram of a handwritten text image provided by the present invention;

[0029] Figure 6 It is a structural schematic diagram of the handwritten text image generation device provided by the present invention;

[0030] Figure 7It is a structural schematic diagram of a training unit provided by the present invention;

[0031] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] The terms used in one or more embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present invention. The singular forms of "a", "said" and "the" used in one or more embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present invention refers to and includes any or all possible combinations of one or more associated listed items.

[0034] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present invention, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present invention, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0035] In the field of computer vision, rendering handwritten text images with diverse handwriting styles can not only promote visual design and artistic creation, but also augment data by generating handwritten text images, which will reduce data annotation costs and potentially improve the performance of handwritten text recognition.

[0036] In the related art, handwritten glyph synthesis algorithms in the field of handwritten glyph synthesis have achieved certain achievements, but most of them rely on online data, which requires additional equipment and is not suitable for processing existing pure image data. In addition, none of them considers improving the handwritten text recognition performance by generating new text images. In the technical solution of directly outputting handwritten text images based on offline data, the structural correlation between adjacent characters is not considered for style diversity and the visual rationality of the synthesized image. In addition, the recognizability of the generated content is not high, and the recognizability of the generated content is also crucial for downstream tasks such as handwritten text recognition.

[0037] The quality of the handwritten text images generated in the above methods is not ideal. To solve this problem, an embodiment of the present invention provides a handwritten text image generation solution.

[0038] The embodiment of the present invention adopts a decoupling mechanism to decouple the handwritten text image into a content vector and a content-independent style variable, which is conducive to providing refined control over the content variables and style variables, thereby improving the style diversity and content recognizability of the generated image.

[0039] Exemplary embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0040] like Figure 1 FIG. 1 is a flow chart of a method for generating a handwritten text image according to an embodiment of the present invention. The method provided by the embodiment of the present invention can be executed by any electronic device with computer processing capability, such as a terminal or a server. Figure 1 As shown, the handwritten text image generation method includes:

[0041] Step 102: Acquire text data for generating a handwritten text image.

[0042] Specifically, the text data may be a character string with preset content.

[0043] Step 104: Acquire style data of the handwritten text image to be generated.

[0044] Specifically, the style data may be randomly selected standard normal distribution N(0,1) or variable data in other distributions, or may be specified variable data. When the style data is specified variable data, a handwritten text image having a specified style of the specified variable data will be generated.

[0045] Step 106, inputting the text data and style data into a preset handwritten text image generation model to obtain a handwritten text image, wherein the handwritten text image model is trained based on interference labels, training images and their corresponding text labels.

[0046] Specifically, the handwritten text image model includes a content encoder and a generator. The content encoder is used to generate a text vector based on text data. The text vector and style data in variable form are input into the generator to obtain the above-mentioned handwritten text image.

[0047] In the technical solution of the embodiment of the present invention, the training image and its corresponding text label can be decoupled into style variables and content vectors, and further, the style variables can be mapped to a Gaussian distribution. In this way, the handwritten text image model of the embodiment of the present invention can generate new images with specified text content and diverse styles, and the handwritten text image model is used to process the handwritten text image to be generated, that is, a high-quality handwritten text image can be obtained. Furthermore, the obtained handwritten text image can be used for data augmentation to improve the handwritten text recognition performance.

[0048] like Figure 2 As shown, the training method of the handwritten text image generation model includes the following steps:

[0049] Step 202: input the training image and its corresponding text label into the content encoder and the style encoder respectively to generate content variables and style variables.

[0050] Step 204: input the interference tag into the content encoder to generate an interference content variable.

[0051] Step 206: Input the interference content variables and the style variables into a generator to generate a first text image.

[0052] Step 208: Input the content variables and style variables into the generator to generate a second text image.

[0053] Step 210, calculating a loss value based on the first text image, the second text image, the training image, and the text label, and training the handwritten text image model based on the loss value.

[0054] Specifically, the initial image model may be a neural network model including a fully convolutional encoder-decoder network.

[0055] The technical solution of the embodiment of the present invention proposes a decoupling mechanism, using two encoders E c and E s Separate the styles from the training images and their text labels. s Representation style encoder, E c The technical solution of the embodiment of the present invention can redefine the modeling of the structural association between adjacent characters as a style variable decoupling problem.

[0056] like Figure 3As shown, the network of the handwritten text image model includes: content encoder E c 302. Style Encoder E s 304, generator G 305, discriminator D 308, and recognizer R 309. The text label 301 is input into the content encoder, and the handwritten text image 303 to be generated is input into the style encoder. Two different text labels "machine" and "without" correspond to different output images 306 and 307 of the generator G, respectively.

[0057] Specifically, the input training image I and its corresponding text label t are first encoded into latent variables through encoding formulas. The encoding formulas (1) and (2) are as follows:

[0058] (1)

[0059] (2)

[0060] Among them, z c and z s denote content variables and style variables respectively. training_image is a training image.

[0061] At this point, the decoupled style variable z s The training images will be kept I In order to make the style encoder focus only on the style variables, the embodiment of the present invention introduces an interference label t dis So that z c dis and z s The generated text image has the distractor label t dis Content and training images I style.

[0062] The reconstructed text image has the content corresponding to the label t and the training image I The second text image corresponding to the text label t has the content of the text label t and the training image I style, interference label t dis The corresponding generated first text image has an interference label t dis Content and training images I The first text image can also be called an interference image. In this way, the content vector and the style variable can be successfully separated, so that a new image with specified content and style can be generated.

[0063] The encoding process of the content variable corresponding to the interference label can be expressed by formula (3), the generation formula of the second text image can be expressed by formula (4), and the generation formula of the first text image can be expressed by formula (5).

[0064] (3)

[0065] (4)

[0066] (5)

[0067] Among them, G is the generator, z c dis is the content variable corresponding to the interference label, is the second text image, is the first text image.

[0068] In the training phase of the handwritten text image generation model, the text label t is input into the content encoder E c To generate the content variable z c , input the training image into the style encoder E s , using Gaussian distribution to randomly sample the preset style variable set to obtain the style variable z s . Among them, the style variable set is a collection of pre-collected style variables.

[0069] Afterwards, the content variables and style variables are connected together along the channel and fed into the generator G to generate a new handwritten text image. Similarly, the interference label t dis Input content encoder E c , input the training image into the style encoder E s , we can get the content variables and style variables respectively. Then, we connect the content variables and style variables together along the channel and send them to the generator G to generate a new handwritten text image.

[0070] Below, the network of the handwritten text image model is described in detail:

[0071] Content Encoder E c The index of the characters of the input text tag can be obtained based on the alphabet, and then the content variable can be represented by a one-hot vector. In step 202, the text tag can be input to the content encoder to obtain the character index of the text tag based on the alphabet; and the first content vector is generated according to the one-hot vector of the character index. This step 204 can be performed by using the content encoder.

[0072] Style Encoder E sThe network is mainly composed of four layers of residual modules and two fully connected layers. The residual module consists of two parts: residual mapping and identity mapping. The residual mapping consists of two layers of 3×3 convolutional layers, and the identity mapping consists of one layer of 1×1 convolutional layer. Each layer of residual module is followed by an activation layer and a spectral normalization layer. Except for the last layer, each layer of residual module uses average pooling to perform downsampling operations. Finally, the output of the last layer of residual module is sent to two layers of fully connected layers to obtain the style variable parameters. and , for Gaussian sampling.

[0073] In step 202, the training image may be input into a style encoder to generate style variable parameters; and a preset style variable set may be sampled through a Gaussian distribution according to the style variable parameters to obtain style variables.

[0074] The network of the generator G mainly consists of three layers of residual modules and one convolutional layer. The structure of the residual module is consistent with that of the residual module in the style encoder. In addition, each layer of residual module uses nearest neighbor interpolation to perform upsampling operations. After the last layer of residual modules, a convolutional layer is used to output the final generated image.

[0075] The network of the discriminator D consists of four layers of residual modules and one fully connected layer. The fully connected layer as the output layer is used to output the discriminative signal, that is, the discrimination result.

[0076] The recognizer R selects the CRNN-based framework as the recognizer. Its network is mainly composed of seven convolutional layers, four pooling layers and one fully connected layer. The fully connected layer as the output layer is used to output category scores.

[0077] In step 210, a generator is used to process according to the first content variable, the second content variable and the style variable to output a target image; a recognizer is used to recognize the target image and the training image to obtain recognition data, so as to calculate a loss function based on the recognition data; a discriminator is used to discriminate the target image and the training image to obtain a discrimination result, so as to calculate a loss function based on the discrimination result.

[0078] In the embodiment of the present invention, the loss function used when calculating the loss value is a weighted sum of at least two single loss functions. For example, the single loss function may be a KL divergence function, an adversarial loss function, a reconstruction loss function, and a content consistency function.

[0079] Among them, the KL divergence function obtains the loss value based on the style variable parameters, and the style variable parameters are generated by the style encoder according to the training image and its corresponding text label; the adversarial loss function generates the loss value based on the style variable, the training image, the first text image and the second text image, and the discrimination result of the discriminator; the reconstruction loss function generates the loss value based on the second text image and the training image; the content loss function is based on the loss value calculated based on the first text image and the interference label.

[0080] Specifically, the KL divergence function L kl It is obtained based on the style variable parameters generated during the training process of the handwritten text image model. adv The reconstruction loss L is obtained based on the mathematical expectation of the discriminant output. rec The function is obtained based on the deviation between the training image and the interference image; the content consistency function L R The interference labels and their corresponding interference images are processed using the CTC algorithm function.

[0081] In the embodiment of the present invention, the loss function can be described as the following formula (6):

[0082] (6)

[0083] in, is the weight of each item in the above formula.

[0084] Specifically, the functions of each single loss function are as follows:

[0085] The KL divergence function is also called the Kullback Leibler loss. Since the ultimate goal of the current task is to collect style variables z from the preset style variable set, s Generating handwritten text images with diverse styles, during the training phase, the encoded style variables are constrained to a Gaussian distribution.

[0086] Assume that the prior of the latent variables, that is, the set of style variables, conforms to the central isotropic multivariate Gaussian distribution, which can be expressed as: Based on the KL divergence function, the handwritten text image generation model in the embodiment of the present invention can be optimized by resampling technology using the following formula (7):

[0087] (7)

[0088] in, and is the style variable parameter output by the style encoder. The latent variable is from The sampled .

[0089] In order to generate more realistic and sharp images, the adversarial training mechanism is used to train the handwritten text image generation model in the embodiment of the present invention. rec is a loss function used in adversarial training mechanism. The optimization objective of adversarial loss can be expressed as the following formula (8):

[0090] (8)

[0091] Among them, D() is the discriminant result output by the discriminator, E[ ] is the data expectation, ~P data Indicates that an image is taken from this distribution.

[0092] In order to enable the handwritten text image generation model in the embodiment of the present invention to generate visually satisfactory text images, a reconstruction loss function is introduced to measure the pixel-level deviation between the generated image and the original image. The reconstruction loss function can be expressed as the following formula (9):

[0093] (9)

[0094] Among them, L rec To reconstruct the loss function, is the reconstructed second text image, I is the original training image.

[0095] In order to decouple the style variable from the content information, the embodiment of the present invention additionally introduces a text tag t dis to generate a text image with recognizable content. This image is not affected by the image to which the style is provided. I In order to constrain the generated graph, the first text image To maintain consistency with its text label in terms of content, the embodiment of the present invention adopts the CTC (connectionist temporal classification) loss function, i.e., the content consistency loss L R As the loss function, it is related to the recognition data output by the recognizer. The consistency loss function can be expressed by the following formula (10):

[0096] (10)

[0097] like Figure 4 As shown, in the technical solution of the embodiment of the present invention, in the stage of applying the handwritten text image generation model for reasoning, the text label 401 is input into the content generator E cIn 403, content variables are generated. The style variables 402 and the content variables are input into the generator G 404 to generate a target image 405, which can be used in HTC 406, ie, handwritten text recognition.

[0098] Figure 5 The handwritten text image generated by the present invention is shown. Each row corresponds to the same style, and each column corresponds to the same content. Figure 5 It can be seen that the present invention can generate high-quality sequence handwritten texts with specified content and diverse styles.

[0099] The technical solution of the embodiment of the present invention can be experimented on two open source datasets, IAM and RIMES. Among them, the IAM dataset contains more than 100,000 handwritten labeled English word images by 657 authors, and the RIMES dataset contains about 60,000 French text images written by 1,300 authors. In specific experiments, the training set, validation set, and test set division that come with the open source dataset can be used.

[0100] 100,000 images are synthesized based on the IAM and RIMES datasets respectively, and then used as augmented data to train the handwritten text recognition model. Table 1 below shows the gain brought by the images generated based on the technical solution of the embodiment of the invention to the downstream handwritten text recognition task.

[0101] Table 1 Handwritten text recognition data

[0102]

[0103] Among them, the original data is the original image that has not been processed, the affine transformation is the image obtained by affine transformation of the original image, and the generated data is the image generated by applying the technical solution in the embodiment of the present invention. It can be found from the table that the application of the technical solution in the embodiment of the present invention can greatly improve the performance of handwritten text recognition.

[0104] According to the technical solution of the embodiment of the present invention, when generating style variables in step 104, the preset style variable set can be randomly sampled through Gaussian distribution to generate style data. In addition to random sampling, any text image can be selected and sampled through the style encoder E s Get its style variable z s , in this way, the function of handwriting imitation can be realized.

[0105] The handwritten text image generation model in the embodiment of the present invention can directly generate a handwritten text image based on the potential prior. Different from single-character font synthesis, the model proposed by the technical solution of the embodiment of the present invention can generate sequence characters of arbitrary length in an end-to-end manner, which pays more attention to the structural correlation between adjacent characters. Specifically, the technical solution of the embodiment of the present invention models the structural relationship between characters as style variables to avoid explicit modeling of the stroke layout. In the training stage, the decoupled style variables can be mapped to a Gaussian distribution, so that a large number of style variables can be obtained by sampling in the inference stage.

[0106] The handwritten text image generation method provided by the present invention obtains a handwritten text image generation model by training according to interference labels, training images and text labels, and processes the handwritten text image to be generated based on the handwritten text image generation model, so as to obtain a handwritten text image with higher quality.

[0107] The handwritten text image generation device provided by the present invention is described below. The handwritten text image generation device described below and the handwritten text image generation method described above can be referred to each other.

[0108] like Figure 6 As shown, a handwritten text image generation device according to an embodiment of the present invention includes:

[0109] The text acquisition unit 602 may be used to acquire text data for generating a handwritten text image.

[0110] The style acquisition unit 604 may be used to acquire style data of the handwritten text image to be generated.

[0111] The generation unit 606 can be used to input text data and style data into a preset handwritten text image generation model to obtain a handwritten text image; wherein the handwritten text image model is trained based on interference labels, training images and their corresponding text labels.

[0112] In an embodiment of the present invention, the handwritten text image model includes a style encoder, a content encoder, a generator, and a discriminator, such as Figure 7As shown, the handwritten text image generation device also includes a training unit, which includes: an encoding subunit 702, which can be used to input the training image and its corresponding text label into the style encoder and the content encoder respectively to generate style variables and content variables, and input the interference label into the content encoder to generate interference content variables; a generating subunit 704, which can be used to input the interference content variable and the style variable into the generator to generate a first text image, and input the content variable and the style variable into the generator to generate a second text image; a training subunit 706, which can be used to calculate the loss value based on the first text image, the second text image, the training image and the text label, and train the handwritten text image model based on the loss value.

[0113] In an embodiment of the present invention, the encoding subunit may be used to: input the text label into the style encoder to obtain the character index of the text label based on the alphabet; and generate the content variable according to the one-hot vector of the character index.

[0114] In an embodiment of the present invention, the loss function used when calculating the loss value is a weighted sum of at least two single loss functions, and the single loss function includes at least one of the following functions: a KL divergence function, an adversarial loss function, a reconstruction loss function, and a content loss function.

[0115] In an embodiment of the present invention, the KL divergence function obtains a loss value based on style variable parameters, and the style variable parameters are generated by a style encoder according to a training image and its corresponding text label; the adversarial loss function generates a loss value based on the style variable, the training image, the first text image and the second text image, and the discrimination result of the discriminator; the reconstruction loss function generates a loss value based on the second text image and the training image; and the content loss function generates a loss value based on the first text image and the interference label.

[0116] In an embodiment of the present invention, the single loss function also includes an adversarial loss function, and the adversarial loss function is obtained according to the mathematical expectation of the discrimination result output by the discriminator.

[0117] In an embodiment of the present invention, the single loss function also includes a reconstruction loss function and / or a content consistency function; the reconstruction loss function is obtained based on the deviation between the training image and the interference image; the content consistency function is obtained by processing the interference label and its corresponding interference image using the CTC algorithm function.

[0118] In an embodiment of the present invention, the encoding subunit can also be used to: use the style encoder in the initial image model to sample the training image and then output style variable parameters; input the training image into the style encoder to generate style variable parameters; sample a preset style variable set through Gaussian distribution according to the style variable parameters to obtain style variables.

[0119] In the embodiment of the present invention, the style acquisition unit may also be used to: generate style data by randomly sampling a preset style variable set through Gaussian distribution.

[0120] Since the various functional modules of the handwritten text image generation device of the exemplary embodiment of the present invention correspond to the steps of the exemplary embodiment of the handwritten text image generation method described above, for details not disclosed in the embodiment of the device of the present invention, please refer to the embodiment of the handwritten text image generation method described above.

[0121] The handwritten text image generation device provided by the present invention obtains a handwritten text image generation model by training according to interference labels, training images and text labels, and processes the handwritten text image to be generated based on the handwritten text image generation model, so as to obtain a handwritten text image with higher quality.

[0122] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 880, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 880. The processor 810 may call the logic instructions in the memory 830 to execute the handwritten text image generation method, which includes: obtaining text data of a handwritten text image to be generated; obtaining style data of the handwritten text image to be generated; inputting the text data and the style data into a preset handwritten text image model to obtain a handwritten text image; wherein the handwritten text image model is trained according to interference labels, training images and their corresponding text labels.

[0123] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0124] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the handwritten text image generation method provided by the above-mentioned methods, and the method includes: obtaining text data of a handwritten text image to be generated; obtaining style data of the handwritten text image to be generated; inputting the text data and the style data into a preset handwritten text image model to obtain a handwritten text image; wherein the handwritten text image model is trained based on interference labels, training images and their corresponding text labels.

[0125] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned handwritten text image generation method, the method comprising: obtaining text data of a handwritten text image to be generated; obtaining style data of the handwritten text image to be generated; inputting the text data and the style data into a preset handwritten text image model to obtain a handwritten text image; wherein the handwritten text image model is trained based on interference labels, training images and their corresponding text labels.

[0126] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a handwritten text image, characterized in that: include: Obtaining text data of a handwritten text image to be generated; Acquire style data of the handwritten text image to be generated; Inputting the text data and the style data into a preset handwritten text image model to obtain a handwritten text image; The handwritten text image model is trained based on interference labels, training images and their corresponding text labels; The handwritten text image model includes a style encoder, a content encoder, a generator and a discriminator; The training method of the handwritten text image model includes: Inputting the training image and the corresponding text label into the style encoder and the content encoder respectively to generate style variables and content variables; Inputting the interference tag into the content encoder to generate an interference content variable; Inputting the interference content variable and the style variable into the generator to generate a first text image; Inputting the content variable and the style variable into the generator to generate a second text image; Calculating a loss value based on the first text image, the second text image, the training image, and the text label, and training the handwritten text image model based on the loss value; The loss function used in calculating the loss value is a weighted sum of at least two single loss functions, wherein the single loss function includes at least one of the following functions: a KL divergence function, an adversarial loss function, a reconstruction loss function, and a content loss function; The KL divergence function obtains a loss value based on a style variable parameter, wherein the style variable parameter is generated by the style encoder according to the training image and its corresponding text label; The adversarial loss function generates a loss value based on the style variable, the training image, the first text image and the second text image, and the discrimination result of the discriminator; The reconstruction loss function generates a loss value based on the second text image and the training image; The content loss function is based on a loss value calculated by the first text image and the interference label; The loss function is described as follows: ； in, is the weight of each item in the above formula; L kl is the KL divergence function, L adv To combat the loss function, L rec is the reconstruction loss function, L R is the content consistency function; Among them, the KL divergence function is implemented by the following formula: ； in, and is the style variable parameter output by the style encoder, The adversarial loss function is implemented by the following formula: ； Among them, D() is the discriminant result output by the discriminator, E[ ] is the data expectation, ~P data Indicates taking an image from this distribution; The reconstruction loss function is implemented by the following formula: ； in, is the reconstructed second text image, I is the original training image; The consistency loss function is implemented by the following formula: ； Among them, t dis is the interference label, is the first text image.

2. The method according to claim 1, characterized in that Inputting the text label corresponding to the training image into the content encoder to generate a content variable includes: Input the text tag into the content encoder to obtain a character index of the text tag based on the alphabet; The content variable is generated according to the one-hot vector of the character index.

3. The method according to claim 1, characterized in that: Inputting the training image into the style encoder to generate style variables includes: Inputting the training image into the style encoder to generate style variable parameters; According to the style variable parameters, a preset style variable set is sampled through Gaussian distribution to obtain style variables.

4. The method according to claim 1, characterized in that The step of obtaining the style data of the handwritten text image to be generated includes: The style data is generated by randomly sampling a preset style variable set through Gaussian distribution.

5. A handwritten text image generation device, characterized in that: include: A text acquisition unit, used to acquire text data for generating a handwritten text image; A style acquisition unit, used to acquire style data of the handwritten text image to be generated; A generating unit, configured to input the text data and the style data into a preset handwritten text image model to obtain a handwritten text image; The handwritten text image model is trained based on interference labels, training images and their corresponding text labels; The handwritten text image model includes a style encoder, a content encoder, a generator and a discriminator; The training method of the handwritten text image model includes: Inputting the training image and the corresponding text label into the style encoder and the content encoder respectively to generate style variables and content variables; Inputting the interference tag into the content encoder to generate an interference content variable; Inputting the interference content variable and the style variable into the generator to generate a first text image; Inputting the content variable and the style variable into the generator to generate a second text image; Calculating a loss value based on the first text image, the second text image, the training image, and the text label, and training the handwritten text image model based on the loss value; The loss function used in calculating the loss value is a weighted sum of at least two single loss functions, wherein the single loss function includes at least one of the following functions: a KL divergence function, an adversarial loss function, a reconstruction loss function, and a content loss function; The KL divergence function obtains a loss value based on a style variable parameter, wherein the style variable parameter is generated by the style encoder according to the training image and its corresponding text label; The adversarial loss function generates a loss value based on the style variable, the training image, the first text image and the second text image, and the discrimination result of the discriminator; The reconstruction loss function generates a loss value based on the second text image and the training image; The content loss function is based on a loss value calculated by the first text image and the interference label; The loss function is described as follows: ； in, is the weight of each item in the above formula; L kl is the KL divergence function, L adv To combat the loss function, L rec is the reconstruction loss function, L R is the content consistency function; Among them, the KL divergence function is implemented by the following formula: ； in, and is the style variable parameter output by the style encoder, The adversarial loss function is implemented by the following formula: ； Among them, D() is the discriminant result output by the discriminator, E[ ] is the data expectation, ~P data Indicates taking an image from this distribution; The reconstruction loss function is implemented by the following formula: ； in, is the reconstructed second text image, I is the original training image; The consistency loss function is implemented by the following formula: ； Among them, t dis is the interference label, is the first text image.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images