Online Handwritten Text Synthesis Method, Apparatus, and Storage Medium

Abstract

A method, device and storage medium for online handwritten text synthesis, the method comprising: obtaining input text and inputting it into a trained handwritten text synthesis model based on an attention mechanism, wherein an encoder in the model outputs hidden state features; based on the attention mechanism of the model, the hidden state features are transformed into context features, wherein the context features can indicate the characters to which handwriting points to be decoded belong; the context features are input into a decoder in the model, wherein the decoder outputs online handwritten handwriting points of the input text. This application uses a trained handwritten text synthesis model based on an attention mechanism to synthesize online handwritten handwriting points for the input text. Since the context features output by the attention module indicate the characters to which handwriting points to be decoded belong, the decoder can more accurately locate which input character plays a decisive role when decoding the current handwriting point, thereby generating a more stable handwritten text line.

Description

Technical Field

[0001] The present application relates to the field of intelligent writing technology, and more specifically to an online handwritten text synthesis method, device and storage medium. Background Art

[0002] Reading and writing play crucial roles in human life, corresponding to inputting information from the world and outputting information to the world, respectively. Therefore, the question of how to empower machines with both reading (i.e., handwritten text recognition) and writing (i.e., handwritten text generation) technologies has attracted widespread attention.

[0003] Currently, the accuracy of handwritten text recognition has been greatly improved. However, when it comes to synthesizing handwritten data, it is difficult to define or predict the specific appearance representation of characters due to the writer's handwriting style, the mixture of disconnected and connected styles within the written content, the changes in glyph usage, character spacing and tilt, etc., making it challenging to synthesize realistic handwritten data.

[0004] Generally speaking, there are two ways to represent handwriting data: one is to view it as aligned pixels, such as a static image written on paper; the other is to represent it as a sequence of strokes, i.e., a writing trajectory. These two representations correspond to offline handwriting data and online handwriting data, respectively. Online handwriting data typically contains more information (such as time information) and can be converted into offline data. The key point is that in the actual writing process, people usually draw strokes one by one in a predefined order to write a character, rather than generating an image all at once. In actual online data application scenarios, the collection and annotation of online handwriting data is expensive, so the synthesis of online handwriting data is a challenging and promising task. Summary of the Invention

[0005] According to one aspect of the present application, a method for online handwritten text synthesis is provided, the method comprising: obtaining input text and inputting it into a trained handwritten text synthesis model based on an attention mechanism, wherein an encoder in the model outputs hidden state features; based on the attention mechanism of the model, the hidden state features are transformed into context features, wherein the context features can indicate the characters to which the handwriting points to be decoded belong; and the context features are input into a decoder in the model, wherein the decoder outputs the online handwritten handwriting points of the input text.

[0006] In one embodiment of the present application, writing style information is also obtained when the input text is obtained, and the decoder further outputs online handwriting points of the input text based on the writing style information.

[0007] In one embodiment of the present application, the encoder includes a bidirectional long short-term memory network, and the hidden state features output by the encoder include N hidden vectors, each of which is a concatenation of a forward state and a backward state, and N is greater than or equal to the number of characters contained in the input text.

[0008] In one embodiment of the present application, the decoder decodes once every time step, and outputs the position offset coordinates of a handwriting point and the handwriting status of the handwriting point each time it is decoded.

[0009] In one embodiment of the present application, the handwriting status of the handwriting point includes: a stroke unfinished state, a stroke completed state, and a complete text completed state.

[0010] In one embodiment of the present application, the decoder predicts the Gaussian mixture distribution of the relative position of the current handwriting point with respect to the previous handwriting point each time decoding is performed, and samples the Gaussian mixture distribution to obtain the position offset coordinates of the current handwriting point.

[0011] In one embodiment of the present application, the decoder concatenates the previous prediction output, the current context feature and the writing style information to obtain a concatenated feature each time decoding is performed, obtains a decoder hidden state feature based on the concatenated feature, obtains a linear mapping feature based on the decoder hidden state feature, and obtains the current prediction output based on the linear mapping feature.

[0012] In one embodiment of the present application, the decoder includes a bidirectional long short-term memory network.

[0013] In one embodiment of the present application, the handwritten text synthesis model based on the attention mechanism is obtained by training in the following manner: the first stage: rendering the handwritten text handwriting points in the training set into offline image data, training a sequence-to-sequence model based on the attention mechanism based on the offline image data, and obtaining an image text line recognition model based on the attention mechanism, so that the first stage obtains the attention position information on the original image corresponding to a single character; the second stage: training the handwritten text synthesis model, and during the training process, using the attention position information obtained in the first stage as a supervision signal to supervise the training of the handwritten text synthesis model, and obtaining the handwritten text synthesis model based on the attention mechanism.

[0014] In one embodiment of the present application, the handwritten text synthesis model is trained based on writer identification information, and the handwritten text handwriting points of the writer corresponding to the writer identification information are included in the training set of the first stage.

[0015] According to another aspect of the present application, an online handwritten text synthesis device is provided, which includes: an encoder module for obtaining input text and outputting hidden state features based on the input text; an attention module for transforming the hidden state features into context features based on an attention mechanism, wherein the context features can indicate the characters to which the handwriting points to be decoded belong; and a decoder module for outputting online handwritten handwriting points of the input text based on the context features.

[0016] According to another aspect of the present application, an online handwritten text synthesis device is provided, which includes a memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the processor executes the above-mentioned online handwritten text synthesis method.

[0017] According to another aspect of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is run, the computer program executes the above-mentioned online handwritten text synthesis method.

[0018] According to the online handwritten text synthesis method, device and storage medium of the embodiment of the present application, a trained handwritten text synthesis model based on the attention mechanism is used to synthesize online handwritten handwriting points (also called trajectory points) for the input text. Since the context features output by the attention module indicate the characters to which the handwriting points to be decoded belong, the decoder can more accurately locate which input character plays a decisive role when decoding the current handwriting point, thereby generating a more stable handwritten text line. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0020] Figure 1 Schematic diagram showing the challenging factors of synthesizing handwritten text.

[0021] Figure 2 A schematic diagram showing Chinese characters represented based on stroke sequences.

[0022] Figure 3 A schematic flowchart of an online handwritten text synthesis method according to an embodiment of the present application is shown.

[0023] Figure 4A schematic diagram showing the first stage of training a handwritten text synthesis model based on an attention mechanism in an online handwritten text synthesis method according to an embodiment of the present application.

[0024] Figure 5 A schematic diagram showing the second stage of training a handwritten text synthesis model based on an attention mechanism in an online handwritten text synthesis method according to an embodiment of the present application.

[0025] Figure 6 A schematic structural block diagram of an online handwritten text synthesis device according to an embodiment of the present application is shown.

[0026] Figure 7 A schematic structural block diagram of an online handwritten text synthesis device according to another embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present application more apparent, example embodiments according to the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application described in this application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of this application.

[0028] For handwriting data synthesis, it is difficult to define or predict character-specific appearance representations due to the writer's handwriting style, the mixture of disconnected and connected styles within the written content, the changes in the use of glyphs, character spacing and tilt, etc. Figure 1 As shown in , it is challenging to synthesize realistic handwriting data. There are usually two ways to represent handwriting data: one is to regard it as a static image of aligned pixels such as writing on paper; the other is to represent it as a sequence of strokes, i.e., writing trajectories, such as Figure 2 These two representations correspond to offline handwriting data and online handwriting data, respectively. In actual online data application scenarios, the collection and annotation of online handwriting data are expensive, so the synthesis of online handwriting data is a challenging and promising task.

[0029] With the rapid development of electronic engineering, people are increasingly using stylus-based devices such as smartphones, whiteboards, or tablets to write text in digital ink (online data), which is easy to process and manipulate. However, despite promising progress in handwriting recognition, handwritten text synthesis in online data remains relatively underutilized. The fundamental reason is that online data synthesis relies on handwriting point sequence generation models. Learning sequence generation models is a long-standing challenge in machine learning, initially within the field of dynamic Bayesian methods and later addressed by recurrent neural networks (RNNs). Although some RNN-based generative models have been designed for handwritten text, the text they synthesize still suffers from many issues compared to normal human handwriting. Generative adversarial networks (GANs) aim to generate realistic images of handwritten text. Researchers have applied them to optical character recognition, using recurrent layers of bidirectional long short-term memory (BiLSTM) networks to obtain word embedding representations of the corresponding characters and feeding them into the generator network. However, this method cannot directly synthesize online handwritten text. Although the generated images are realistic, the conversion from images to digital strokes requires an effective ink grab algorithm. Some researchers have applied variational autoencoders (VAEs) to RNNs and conducted experimental verification by generating speech and online data. However, the handwritten data synthesized by this solution is not realistic enough. Alex Grave proposed an RNN-based generator model to simulate handwritten data for online text trajectory point synthesis. However, the synthesis results are unstable and cannot synthesize handwritten text data of a specific style.

[0030] Based on at least one of the above problems, this application provides an online handwritten text synthesis solution. Figures 3 to 7 To describe by way of example.

[0031] Figure 3 FIG. 3 is a schematic flow chart of an online handwritten text synthesis method 300 according to an embodiment of the present application. Figure 3 As shown, the online handwritten text synthesis method 300 may include the following steps:

[0032] In step S310, the input text is input into a trained handwritten text synthesis model based on the attention mechanism, and the encoder in the model outputs the hidden state features.

[0033] In step S320, the model-based attention mechanism transforms the latent state features into context features, and the context features can indicate the character to which the handwriting point to be decoded belongs.

[0034] In step S330, the context features are input to a decoder in the model, and the decoder outputs the online handwriting points of the input text.

[0035] In an embodiment of the present application, a trained handwriting text synthesis model based on an attention mechanism is used to synthesize online handwriting handwriting points (also called track points) for the input text. The handwriting text synthesis model includes an encoder, an attention module, and a decoder. The encoder extracts features from the input text and outputs latent state features; the attention module transforms the latent state features into context features, indicating the category information of the handwriting points to be decoded (i.e., the characters to which the handwriting points to be decoded belong); the decoder decodes each handwriting point to be decoded based on the context features corresponding to each handwriting point to be decoded, obtains the relative position and handwriting state of each handwriting point to be decoded, and finally obtains the online handwriting handwriting points of the entire input text. Since the context features output by the attention module indicate the characters to which the handwriting points to be decoded belong, the decoder can more accurately locate which input character plays a decisive role when decoding the current handwriting point, thereby generating a more stable handwriting text line.

[0036] In a further embodiment of the present application, writing style information may also be obtained when acquiring the input text in step S310. Based on this, the decoder may also output online handwriting points of the input text based on the writing style information in step S330. In this embodiment, writing style information (e.g., feature information corresponding to a particular handwriter ID) is incorporated into the decoding end, thereby generating online handwriting data with a specific style.

[0037] The following combination Figure 4 and Figure 5 The following describes the training process of the handwritten text synthesis model based on the attention mechanism in the online handwritten text synthesis method according to the embodiment of the present application. Figure 4 A schematic diagram showing the first stage of training a handwritten text synthesis model based on an attention mechanism in an online handwritten text synthesis method according to an embodiment of the present application. Figure 5 A schematic diagram showing the second stage of training a handwritten text synthesis model based on an attention mechanism in an online handwritten text synthesis method according to an embodiment of the present application.

[0038] First, in the embodiments of the present application, the training data can come from the track point data collected on the electronic terminal device (such as a tablet office book, smart phone, whiteboard or tablet computer, etc.). Next, the training data can be preprocessed: the first step is to input the text content, and the label is a series of stroke points corresponding to the text in time order. Each stroke point consists of the (x, y) coordinates on the tablet device and the pen lift event. The pen coordinates are integer values limited by the screen resolution. When the pen is lifted from the screen, the state of the pen lift event is recorded as 1, otherwise it is 0. The handwriting sample is formally defined as where x tis a stroke point, and T is the number of stroke points. In actual model training, the input data stroke point at a given moment is defined as the relative position of the stroke coordinate point at the current moment and the stroke coordinate point at the previous moment. It consists of a real-valued pair x = (x1, x2) and a binary representation x3. If the current stroke ends, x3 = 1; otherwise, it is 0. The input data is also preprocessed, including calculating the global mean and variance (global_mean, global_var) of all stroke point offsets, expressed as follows: x = (x - global_mean) / global_var.

[0039] Next, the attention supervision signal training network can be constructed. Specifically, for the training data of the network, for a given online handwritten text training set, according to its coordinate points and stroke order, it is rendered into offline image data, and a sequence-to-sequence ED model is constructed based on the number of offline images. The attention-based image text line recognition model is trained to recognize the text content in the offline image. After the model training is stable, the attention position information of the single character of the recognition result in the offline original image is obtained, and the position information is matched with the original offline coordinate point, thereby obtaining the coordinate point corresponding to the single character on the original coordinate point, that is, the category information corresponding to each coordinate point in the online coordinate point, as shown in the following example. Figure 4 shown.

[0040] like Figure 4 As shown in the figure, the trajectory point data of the handwritten text "as though" is rendered into an offline image, which is input into the encoder of the image text line recognition model. The encoder of the image text line recognition model includes a convolutional neural network (CNN) and a bidirectional long short-term memory network (BiLSTM), and the encoder outputs the hidden state feature Hc. The decoder of the image text line recognition model includes a BiLSTM, whose output content is passed through the classifier softmax to output the text recognition result "as though". Then, the position information of each character of the handwritten text "as though" in the offline original image can be obtained, as shown in Figure 4 The attention position shown in the figure supervises its information and corresponds the position information with the original offline coordinate point, thereby obtaining the coordinate point corresponding to the single character at the original coordinate point, that is, the category information corresponding to each coordinate point in the online coordinate point. Figure 4 The process described is the first stage of the training process.

[0041] In the second stage, a generative network from text to trajectory points is constructed. The network includes an encoder, a decoder, and an attention module, where the encoder and decoder are both composed of a layer of bidirectional LSTM network. c={c1,c2,...,cn}, the encoder encodes it into the hidden state H c =ENC c (X c ), where Hc consists of N hidden vectors, each of which is the concatenation of the forward state and the backward state, and N is greater than or equal to the number of characters n contained in the input text. In the decoding stage, in order to make the input information of the decoding end most relevant to the category of the current decoding trajectory point, the solution of this application introduces an attention mechanism between the content encoder and the decoder. At the decoding time step t, the input content of the decoder (i.e., the context feature mentioned above) is c t =attention(H c ;h t-1 ), where h t-1 is the decoder hidden state of the previous time step, and attention(·) is an attention mechanism. t During the training phase, the category information of the single coordinate point obtained in the aforementioned first phase is introduced as a supervisory signal for supervised training.

[0042] In the above network, the decoder can include a bidirectional LSTM network. The decoder predicts the next point based on the previous prediction output and the current content information from the content encoder (i.e., the contextual features mentioned above). At the current decoding time t, during the training phase, the sample label p predicted at the previous moment t-1 Output with content c t Splice to get a t =[p t-1 ;c t ], used to get the current decoder hidden state h t =DEC(h t-1 ;a t ); Finally, h t Mapped to o through a linear layer t To predict the output handwriting point p t In general, the decoder can decode once every time step, and each decoding outputs the position offset coordinates of a handwriting point and the handwriting state of the handwriting point as the prediction output of this time, specifically as follows Figure 5 shown.

[0043] like Figure 5As shown, the input text "as though" is encoded into a one-hot vector, which is input into the encoder of the handwriting synthesis model. The BiLSTM of the encoder of the handwriting synthesis model obtains the input word embedding features and outputs the latent state features Hc. The decoder of the handwriting synthesis model includes a BiLSTM, whose output content passes through a Gaussian mixture model (GMM) and a classifier softmax, and outputs the handwriting text handwriting point "as though". In the process of training the handwriting synthesis model, the attention position information obtained in the first stage described above is used as a supervisory signal to supervise the training of the handwriting synthesis model. Here, the Gaussian mixture model (GMM) is used to model the position offset (dx; dy) of the handwriting point (for example, the decoder can use a GMM with an R bivariate normal distribution to model the position offset of the handwriting point). In this way, each time the decoder decodes, it can predict the Gaussian mixture distribution of the relative position of the current handwriting point relative to the previous handwriting point, and sample this Gaussian mixture distribution to obtain the position offset coordinates of the current handwriting point. Regarding the softmax classifier, in this embodiment of the present application, the decoder can use a three-category classifier (i.e., a softmax layer) to model point state categories (p1, p2, and p3). Based on this, the handwriting states of a handwriting point can include: unfinished stroke state (i.e., p1), completed stroke state (i.e., p2), and completed text state (i.e., p3).

[0044] also, Figure 5 In the example shown, the writer identification (ID) information is also shown. This information corresponds to the writer's writing style information. Adding this information to the handwriting text synthesis model training allows the model to implicitly learn the writer's writing style characteristics. In this embodiment, at the current decoding time t, in the training phase, the sample label p predicted at the previous moment t-1 Output with content c t And the ID information is written t Splice to get a t =[p t-1 ;c t ;d t ], used to get the current decoder hidden state h t =DEC(h t-1 ;a t ); Finally, h t Mapped to o through a linear layer t To predict the output handwriting point p t The above combination Figure 5 The process described is the second stage of the training process.

[0045] In general, the handwritten text synthesis model based on the attention mechanism of the present application is trained in the following manner: the first stage: the handwritten text handwriting points in the training set are rendered into offline image data, and a sequence-to-sequence model based on the attention mechanism is trained based on the offline image data to obtain an image text line recognition model based on the attention mechanism, so that the first stage obtains the attention position information on the original image corresponding to a single character; the second stage: the handwritten text synthesis model is trained. During the training process, the attention position information obtained in the first stage is used as a supervisory signal to supervise the handwritten text synthesis model, and the handwritten text synthesis model based on the attention mechanism is obtained. Among them, when training the handwritten text synthesis model, it can also be based on writer identification information. The handwritten text handwriting points of the writer corresponding to the writer identification information are included in the training set of the first stage. In other words, the training data contains samples written by multiple people, and each sample is marked by which writer wrote it, which is the writer identification information. In this way, the model implicitly learns the style characteristics of the writer's writing.

[0046] Based on the above training process, in actual use, the model is trained and converged to obtain the final handwritten text (trajectory point) synthesis model, and the test text sequence is passed through Figure 5 The encoder in

[15] obtains the encoded representation of the input text. During decoding, the model generates the relative positions of handwriting points by sampling from the Gaussian mixture distribution of the handwriting points predicted by the decoder. The handwriting state of the handwriting points is predicted by the decoder's state class, and the online handwriting trajectory points of the input text are generated. The style of the generated handwriting points is controlled by the writer's ID information in the input.

[0047] Based on the above description, the basic idea of the online handwritten text synthesis method according to the embodiment of the present application is to realize the generation of handwritten trajectory point sequences based on the autoregressive neural network RNN generation model. In order to improve the stability of the synthesized handwritten trajectory points in the algorithm, the present solution introduces an attention mechanism alignment strategy, which supervises and forces the alignment of attention when training the generation model to obtain a more stable generation result. Since the handwritten text trajectory point data only contains the trajectory point coordinates and text content, and the supervisory signal corresponding to the attention in training is the text information corresponding to the current trajectory point, but in the actual data there is no trajectory point annotation information corresponding to a single character, that is, the supervisory information of attention, in order to solve this problem, the present solution renders the trajectory points of the training set into offline image data training and trains a seq2seq image text line recognition model based on attention, and trains the image text line recognition model based on attention, so that the attention position information corresponding to a single character on the original image is obtained in this stage, which solves the need to label the corresponding character category for each trajectory point in the online handwritten data, saving the annotation cost. Furthermore, to enable the model to generate handwritten trajectory data in a variety of styles, this solution incorporates the writer's ID representation during training to guide the model's generation of data in a variety of styles. In this case, the training data includes samples of handwriting from multiple people, and each sample is labeled with the writer's ID information. This allows the model to implicitly learn the writer's stylistic characteristics. Therefore, the online handwritten text synthesis method according to the embodiment of this application can generate relatively stable handwritten text lines and can also generate online handwritten data with a specific style.

[0048] The above exemplary shows the online handwritten text synthesis method provided in accordance with one aspect of the present application. Figure 6 and Figure 7 The online handwritten text synthesis devices 600 and 700 provided according to another aspect of the present application are described. Both of them can implement the online handwritten text synthesis method 100 according to the embodiment of the present application described above. For the sake of brevity, some details can be found in the above description. Here, only the structural composition and main functions of the online handwritten text synthesis devices 600 and 700 are described.

[0049] Figure 6 FIG. 6 is a schematic structural block diagram of an online handwritten text synthesis device 600 according to an embodiment of the present application. Figure 6As shown, the online handwritten text synthesis device 600 includes an encoder module 610, an attention module 620, and a decoder module 630. The encoder module 610 is used to obtain input text and output latent state features based on the input text. The attention module 620 is used to transform the latent state features into context features based on the attention mechanism. The context features can indicate the character to which the handwriting points to be decoded belong. The decoder module 630 is used to output the online handwritten handwriting points of the input text based on the context features.

[0050] The online handwritten text synthesis device 600 according to the embodiment of the present application can be used to implement the online handwritten text synthesis method 100 according to the embodiment of the present application described above. Therefore, the online handwritten text synthesis device 600 according to the embodiment of the present application can realize the online handwritten text trajectory synthesis based on the attention mechanism alignment. Specifically, the online handwritten text synthesis device 600 according to the embodiment of the present application supervises and forces the alignment of attention when training the generation model to obtain a more stable generation result. In addition, considering that it is very important to be able to generate data of a specific style in actual usage scenarios, the writer's ID representation information can also be added during training. When generating online handwriting point data, the model can control the style information according to the handwriter's ID information.

[0051] Figure 7 FIG. 7 is a schematic structural block diagram of an online handwritten text synthesis device 700 according to another embodiment of the present application. Figure 7 As shown, the online handwritten text synthesis device 700 may include a memory 710 and a processor 720. The memory 710 stores a computer program executed by the processor 720. When the computer program is executed by the processor 720, the processor 720 executes the online handwritten text synthesis method 100 according to the embodiment of the present application described above. Those skilled in the art can understand the specific operation of the online handwritten text synthesis device 700 according to the embodiment of the present application in combination with the above content. For the sake of brevity, the specific details are not repeated here.

[0052] In addition, according to an embodiment of the present application, a storage medium is also provided, on which program instructions are stored. When the program instructions are executed by a computer or processor, the corresponding steps of the online handwritten text synthesis method of the embodiment of the present application are executed. The storage medium may include, for example, a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0053] Based on the above description, the online handwritten text synthesis method and device according to the embodiment of the present application provide an online handwritten text trajectory synthesis solution based on attention mechanism alignment. Specifically, for a given online handwritten text training set, it is rendered into offline image data according to its coordinate points and stroke order; in the first stage, a sequence-to-sequence ED model is constructed for the offline image data, and an attention-based image text line recognition model is trained so that the stage obtains the attention position information corresponding to a single character on the original image; in the second stage, a trajectory point generation model is trained, which is divided into an encoder, an attention module, and a decoder; the encoder input is the embedding representation of the text line characters, and a layer of BiLSTM is used to extract the sequence features of the input text characters; the decoder is modeled based on a Gaussian mixture model (GMM), obtains the distribution parameters of the relative positions of the trajectory coordinate points, and samples according to the predicted distribution function to obtain the relative positions of the trajectory points. During the two-stage model training, the attention learned in the first stage is used to supervise the attention of the generative model. This allows the model to more accurately identify the key character in the input when decoding the current handwriting point, thus generating more stable handwritten text lines. Furthermore, to ensure that the generated text lines have stylistic information, the proposed solution also incorporates the handwriter's ID feature information at the decoding end, enabling the generation of online handwritten data with a specific style.

[0054] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely illustrative and are not intended to limit the scope of the present application. Various changes and modifications may be made therein by those skilled in the art without departing from the scope and spirit of the present application. All such changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0055] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0056] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units described is merely a logical function division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another device, or ignoring or not performing some features.

[0057] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0058] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various inventive aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this approach of the present application should not be interpreted as reflecting the intention that the application claimed for protection requires more features than those explicitly recited in each claim. More precisely, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with fewer features than all the features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present application.

[0059] It will be understood by those skilled in the art that, except where mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus disclosed herein may be combined in any combination. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature providing the same, equivalent, or similar purpose.

[0060] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of this application and to form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

[0061] The various component embodiments of the present application can be implemented in hardware, or in a software module running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some modules according to the embodiments of the present application. The application can also be implemented as a part or all of a device program (e.g., a computer program and a computer program product) for performing the method described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0062] It should be noted that the above embodiments illustrate rather than limit the present application, and that a person skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbols placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0063] The above description is merely a specific embodiment or illustration of a specific embodiment of the present application, and the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. The scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An online handwritten text synthesis method, characterized in that: The method comprises: Taking the input text and inputting it into a trained handwritten text synthesis model based on the attention mechanism, the encoder in the model outputs the hidden state features; transforming the latent state features into context features based on the attention mechanism of the model, wherein the context features can indicate the character to which the handwriting point to be decoded belongs; Inputting the context features into a decoder in the model, and having the decoder output online handwriting trace points of the input text; The handwritten text synthesis model based on the attention mechanism is trained in the following way: The first stage: Render the handwritten text points in the training set into offline image data, train a sequence-to-sequence model based on the attention mechanism based on the offline image data, and obtain an image-text line recognition model based on the attention mechanism, so that the first stage can obtain the attention position information on the original image corresponding to a single character; The second stage: training the handwritten text synthesis model. During the training process, the handwritten text synthesis model is supervised and trained using the attention position information obtained in the first stage as a supervision signal to obtain the handwritten text synthesis model based on the attention mechanism.

2. The method according to claim 1, characterized in that When acquiring the input text, writing style information is also acquired, and the decoder further outputs online handwriting points of the input text based on the writing style information.

3. The method according to claim 1 or 2, characterized in that The encoder includes a bidirectional long short-term memory network, and the hidden state features output by the encoder include N hidden vectors, each of which is a concatenation of a forward state and a backward state, and N is greater than or equal to the number of characters contained in the input text.

4. The method according to claim 1 or 2, characterized in that The decoder decodes once every time step, and outputs the position offset coordinates of a handwriting point and the handwriting status of the handwriting point each time it is decoded.

5. The method according to claim 4, characterized in that The handwriting states of the handwriting points include: an unfinished stroke state, a completed stroke state, and a completed text state.

6. The method according to claim 4, characterized in that During each decoding, the decoder predicts the Gaussian mixture distribution of the relative position of the current handwriting point with respect to the previous handwriting point, and samples the Gaussian mixture distribution to obtain the position offset coordinates of the current handwriting point.

7. The method according to claim 4, characterized in that During each decoding, the decoder concatenates the previous prediction output, the current context feature, and the writing style information to obtain a concatenated feature, sends the concatenated feature to the decoder to obtain a decoder hidden state feature, obtains a linear mapping feature based on the decoder hidden state feature, and obtains the current prediction output based on the linear mapping feature.

8. The method according to claim 1 or 2, characterized in that The decoder includes a bidirectional long short-term memory network.

9. The method according to claim 1, characterized in that The handwritten text synthesis model is also trained based on the writer identification information, and the handwritten text handwriting points of the writer corresponding to the writer identification information are included in the training set of the first stage.

10. An online handwritten text synthesis device, characterized in that: The device comprises: An encoder module configured to take input text and output latent state features based on the input text; An attention module, configured to transform the latent state features into context features based on an attention mechanism, wherein the context features can indicate the character to which the handwriting point to be decoded belongs; a decoder module for outputting online handwriting trace points of the input text based on the contextual features; The input text is input into a trained handwritten text synthesis model based on an attention mechanism, and the encoder module outputs hidden state features; The handwritten text synthesis model based on the attention mechanism is trained in the following way: The first stage: Render the handwritten text points in the training set into offline image data, train a sequence-to-sequence model based on the attention mechanism based on the offline image data, and obtain an image-text line recognition model based on the attention mechanism, so that the first stage can obtain the attention position information on the original image corresponding to a single character; The second stage: training the handwritten text synthesis model. During the training process, the handwritten text synthesis model is supervised and trained using the attention position information obtained in the first stage as a supervision signal to obtain the handwritten text synthesis model based on the attention mechanism.

11. An online handwritten text synthesis device, characterized in that: The device includes a memory and a processor, wherein a computer program executed by the processor is stored in the memory, and when the computer program is executed by the processor, the processor executes the online handwritten text synthesis method according to any one of claims 1 to 9.

12. A storage medium, characterized in that: The storage medium stores a computer program, which, when running, executes the online handwritten text synthesis method according to any one of claims 1 to 9.

Images