Time sequence handwriting data generation method and device, equipment and medium

Abstract

The invention relates to a time sequence handwriting data generation method and device, equipment and a medium. The method comprises the following steps: a data acquisition step: acquiring input data; a data processing step: inputting the input data into the time sequence handwriting generation model for data processing; a result output step: outputting handwriting data according to the data processing result; wherein the time sequence handwriting generation model comprises an encoder and a decoder; the encoder is composed of a residual block and a BiLSTM structure. The time sequence handwriting generation model can be trained only through a small amount of data, key learning is carried out according to characteristics such as inflection points and topological structures of handwriting, and the generated data are more real and accord with the writing rule of people.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, equipment and medium for generating time-series handwriting data. Background Art

[0002] At present, the combination of touch screen devices and stylus has been widely used. In order to develop relevant systems based on stylus, sufficient handwriting data is required. However, when developing handwriting-related systems, we often encounter the problem of data sparsity. At the same time, handwriting data is often highly personalized. Data collected in one application scenario and from a group of users is difficult to migrate to another application scenario. In order to solve this situation, many models or algorithms that can be used to generate handwriting data have emerged, such as GAN networks and Diffusion models.

[0003] Among the existing machine learning methods, models such as GAN and Diffusion require a large amount of data for training, and the training requires extremely high computing power. For non-machine learning methods, the data generated by most traditional algorithms (flipping, scaling, rotation, etc.) do not vary enough and are too similar to the original data; and after adding Gaussian noise and resampling, the generated data is easy to produce large jitters in the handwriting, which does not conform to the actual writing rules of people.

[0004] To address the above problems, no effective solution has been proposed yet. Summary of the invention

[0005] Based on this, it is necessary to provide a method, device, equipment and medium for generating time-series handwriting data in response to the above technical problems. Compared with models such as Diffusion and Transformer, the time-series handwriting generation model used in the time-series handwriting data generation method of the present invention has very few parameters and only requires a small amount of data to complete the training. It focuses on learning the inflection points, topological structures and other features of the handwriting, so that the generated data is more realistic and in line with the writing rules of people.

[0006] According to a first aspect of the present invention, there is provided a method for generating time-series handwriting data, comprising:

[0007] Data acquisition step, obtaining input data;

[0008] A data processing step, inputting the input data into a time series handwriting generation model for data processing;

[0009] A result output step, outputting handwriting data according to the data processing result;

[0010] The components of the temporal handwriting generation model include: an encoder and a decoder;

[0011] The encoder consists of a residual block and a BiLSTM structure.

[0012] In some embodiments, the encoder extracts deep features of the input data through the convolutional network in the residual block, superimposes the input data with the convolutional network output through the residual block, and processes the timing information in the input data through BiLSTM, thereby converting the input data into a high-dimensional data distribution.

[0013] In some embodiments, the decoder includes: an inflection point module;

[0014] The inflection point module is used to extract the inflection points of the handwriting from the input data, connect the inflection points with straight lines in order to form a handwriting topological structure, and learn a weight matrix, wherein the weight values ​​in the weight matrix represent the degree to which the feature layer is affected by the topological structure in each dimension;

[0015] The feature layer in the inflection point module is used for data processing.

[0016] In some embodiments, the inflection point is calculated using a curvature algorithm; the dimension of the weight matrix is ​​N×D×I; the closer to the inflection point in the topological structure, the higher the weight value in the feature layer.

[0017] In some embodiments, the decoder further includes: a trend module, a period module, and a basic decoder module; the trend module is used to form a feature layer according to the trend information in the input data; the period module is used to form a feature layer according to the period information in the input data; the basic decoder module, as a residual branch, forms a feature layer according to the input data; the feature layers of each module in the decoder are processed by summing, and the summed feature layers are used for data processing.

[0018] In some embodiments, the training steps of the encoder and decoder include: inputting original data into the encoder to output latent space data, inputting latent space data into the decoder to output generated data, calculating the loss functions of the encoder and decoder through the original data and generated data, optimizing the model parameters of the encoder and decoder according to the loss functions of the encoder and decoder, and completing the training of the encoder and decoder.

[0019] In some embodiments, the training steps of the encoder and decoder include:

[0020] Treat the original data as a distribution of high-dimensional data The data set {x1,x2…x n} as from The samples obtained by sampling in the 1:T is a time series data of length T, x t is the data at time t, the encoder converts x1:T Mapping to latent space Get h 1:T , and then the decoder will h 1:T Mapping back to space get The relationship between encoder e and decoder h is:

[0021] h t =e(h t-1 ,x t )

[0022]

[0023] The loss function used to optimize the model parameters of the encoder and decoder is L ed , the calculation formula is as follows:

[0024]

[0025] In some embodiments, the time-series handwriting generation model is first pre-trained using a data set with multiple data sets, and then the model is trained and adjusted using a data set with a single data set.

[0026] According to a second aspect of the present invention, there is provided a time series handwriting data generating device, comprising:

[0027] A data acquisition module, used for acquiring input data;

[0028] A data processing module, used for inputting input data into the time-series handwriting generation model for data processing;

[0029] A result output module, outputting handwriting data according to the data processing result;

[0030] The components of the temporal handwriting generation model include: an encoder and a decoder;

[0031] The encoder consists of a residual block and a BiLSTM structure.

[0032] According to a third aspect of the present invention, there is provided a computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the computer program.

[0033] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method of any of the above embodiments are implemented.

[0034] Beneficial effects of the present invention:

[0035] 1. The temporal handwriting generation model of the present invention focuses on learning the inflection points, topological structures and other features of the handwriting, so that the generated data is more realistic and in line with human writing rules.

[0036] 2. The model in the present invention is a lightweight data generation model, which can generate time-series handwriting data without occupying too many resources, and the model training can be performed through the CPU.

[0037] 3. The model in the present invention can capture the complex time dependencies and patterns in time series, and through the variational autoencoder structure, effectively learn the potential representation of time series data and generate time series with similar distribution to the original data.

[0038] 4. The model of the present invention can process multivariate time series data, can process multiple features simultaneously and capture the interdependencies between them. Through variational reasoning and reparameterization techniques, the model of the present invention can effectively reduce overfitting during the learning process, which makes the model more robust when processing small data sets or high-noise data. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is the flow chart of LSTM-VAE training and its generated data;

[0040] Figure 2 This is the LSTM-VAE model structure diagram;

[0041] Figure 3 is a flow chart of some embodiments of the method for generating time-series handwriting data of the present invention;

[0042] Figure 4 It is a schematic diagram of the structure of some embodiments of the time series handwriting data generating device of the present invention;

[0043] Figure 5 It is a diagram of the internal structure of a computer device used to implement some embodiments of the present invention. DETAILED DESCRIPTION

[0044] Embodiments of the present invention will be described more fully below with reference to the accompanying drawings, in which embodiments of the present invention are shown. However, the present invention can be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

[0045] The terms used herein are used only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an", "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that when used herein, the term "comprising" specifies the presence of the claimed features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof.

[0046] Unless otherwise defined, the terms (including technical and scientific terms) used herein have the same meanings as commonly understood by those of ordinary skill in the art to which the present disclosure belongs. The terms used herein should be interpreted as having the meanings consistent with their meanings in the context of this specification and in the relevant field, and should not be interpreted in an idealized or overly formal sense, unless specifically defined as such herein.

[0047] Before describing the specific details of the present application, the meanings of the terms used in the present application will be briefly explained.

[0048] Loss function: Loss function, a function that maps the value of a random event or its related random variables to a non-negative real number to represent the "risk" or "loss" of the random event. In applications, the loss function is usually associated with the optimization problem as a learning criterion, that is, the model is solved and evaluated by minimizing the loss function.

[0049] Latent space data: The data obtained after the encoder maps the original data to the latent space.

[0050] The temporal handwriting generation model in the present invention is LSTM-VAE, which is mainly composed of two parts, an encoder and a decoder, and is used to generate a single simple handwriting. Figure 1 Flowchart for LSTM-VAE training and data generation. Figure 2 This is the LSTM-VAE model structure diagram. Figure 1As shown in the figure, four types of data sets are used in the training process of the model - QuickDraw, DiDiDataSet, VectorMNIST and $family (the $family data set contains a variety of single-stroke handwriting data, and its series of data sets include $1, $N, $Q, etc.). QuickDraw and DiDiDataSet contain a variety of handwriting data such as handwritten symbols and simple paintings. Compared with the simple handwriting that LSTM-VAE needs to generate in the end, the handwriting in QuickDraw and DiDiDataSet mostly consists of multiple strokes. However, since there is less handwriting data for a single symbol or a small number of strokes, the method of pre-training the model first and then performing transfer learning is adopted. First, use the QuickDraw and DiDiDataSet data sets to pre-train LSTM-VAE. Then use the VectorMNIST and $family data sets to fine-tune the model. The final model can generate several new handwriting data after entering a new symbol.

[0051] Figure 3 Flowcharts showing some embodiments of the method for generating time-series handwriting data of the present invention.

[0052] like Figure 3 As shown, the method includes:

[0053] Data acquisition step S102, acquiring input data.

[0054] In the data processing step S104, the input data is input into the time series handwriting generation model for data processing.

[0055] Among them, the components of the time-series handwriting generation model include: encoder and decoder.

[0056] The training steps of the encoder and decoder include:

[0057] The original data is time series data, which can be regarded as the distribution of high-dimensional data. The data in the data set {X1,x2…x n} can be seen as Since P(X) is difficult to obtain, the usual idea is to introduce a hidden variable h and find Space to space mapping. Upsample and then map to space , thus obtaining the newly generated data. It is called latent space. Define x 1:T is a time series data of length T, x t is the data at time t. The encoder will convert x1:T Mapping to latent space Get h 1:T , and then the decoder will h 1:T Mapping back to space get The relationship between encoder e and decoder h can be expressed as:

[0058] h t =e(h t-1 ,x t )

[0059]

[0060] The encoder and decoder methods are derived from RNN (see reference R1). The details are implemented in the original paper. Then the encoder and decoder are trained to generate data. Try to be as similar to the original data as possible. The Loss function used to optimize the encoder and decoder is L ed , L ed MSE is used to measure the distance between the model input and output. The smaller the distance, the more similar the two are:

[0061]

[0062] The encoder consists of a residual block and a two-layer BiLSTM structure. The two layers of convolutional networks in the residual block learn the deep features of the input handwriting data, and at the same time, the input and the convolutional network output are superimposed through the residual structure to retain some of the input features; then the BiLSTM continues to learn the time series information in the feature layer, and finally converts the input data into a high-dimensional data distribution.

[0063] The decoder consists of multiple modules, including: trend module, cycle module, inflection point module, and basic decoder module.

[0064] Trend module: The trend architecture in the N-Beats prediction model is used, and the trend is modeled as a monotonic function. The latent vector z is used to estimate the basis expansion coefficient θ of the trend. tr , the dimension of the coefficient is N×D×P. The trend information of the time series data is reconstructed as V tr =θ tr R, where R is the power matrix of the time vector with dimension P × T.

[0065] Periodic module: Models multiple periodic patterns, each defined by the number of seasons m and the duration of each period d. The latent space vector z is used to estimate the periodic basis expansion coefficient θ sn , the dimension of the coefficient is N × D × m. The periodic pattern is reconstructed by indexing the elements of a specific time period, generating a seasonal estimate Vsn The dimension is N×T×D

[0066] Inflection point module: Extract the inflection point information of the handwriting from the original data, and connect the inflection points with straight lines according to the order to form a simple handwriting topological structure, and use it as a feature layer to affect the final output result. The calculation of the inflection point uses the curvature algorithm. After extracting the inflection point, the network will learn a weight matrix of dimension N×D×I. The weight in the matrix corresponds to the degree to which each dimension of the feature layer is affected by the topological structure. In order to ensure that the generated data does not differ too much from the original data in terms of overall structure, ideally, the closer to the inflection point in the topological structure, the higher the weight of the feature layer of the current module.

[0067] Basic decoder module: The decoder (Decoder) in the present invention also uses the decoder (Decoder) of the basic variational autoencoder (VAE) as the residual branch of the network.

[0068] The output of the final network is the sum of the feature layers of the four modules. This structure enables LSTM-VAE to not only generate high-quality time series data, but also explain the generation process through information such as trends, cycles, and inflection point locations.

[0069] The result output step S106 outputs the handwriting data according to the data processing result.

[0070] In the present invention, compared with using only a single-layer convolution as an encoder structure, the present invention combines convolution and LSTM to form an encoder. In order to prevent the loss of too many shallow features, a residual structure is also added; at the same time, in order to extract the time-series related features in the handwriting information, a two-layer LSTM structure is added. Among them, the two-layer convolution uses ReLU as the activation function, and the two-layer LSTM structure uses tanh as the activation function.

[0071] Figure 4 It is a schematic diagram of the structure of some embodiments of the time series handwriting data generating device of the present invention.

[0072] The data acquisition module 100 is used to acquire input data;

[0073] The data processing module 200 is used to input the input data into the time-series handwriting generation model for data processing;

[0074] The result output module 300 outputs the handwriting data according to the data processing result.

[0075] The components of the temporal handwriting generation model include: an encoder and a decoder;

[0076] The encoder consists of a residual block and a BiLSTM structure.

[0077] The specific definition of a time-series handwriting data generating device can be found in the definition of a time-series handwriting data generating method mentioned above, which will not be repeated here. Each module in the above-mentioned time-series handwriting data generating device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0078] The present invention also provides a computer device, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above-mentioned method for generating time-series handwriting data is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc. Those skilled in the art can understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0079] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned method for generating time-series handwriting data is implemented.

[0080] Those skilled in the art can understand that all or part of the processes in the above method embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0081] So far, the embodiments of the present invention have been described in detail. In order to avoid obscuring the concept of the present invention, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solutions disclosed herein.

[0082] Although some specific embodiments of the present invention have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It should be understood by those skilled in the art that the above embodiments may be modified or some technical features may be replaced by equivalents without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

[0083] References

[0084] The following references provide exemplary procedural or other details supplementary to those set forth herein and are hereby incorporated by reference.

[0085] (R1)GIRSHICK R,DONAHUE J,DARRELL T,et al.Rich feature hierar chiesfor accurate object detection and semantic segmentation;proceedings of theProceedings of the IEEE conference on computer vision and patternrecognition,F,2014[C]。

Claims

1. A method for generating time series handwriting data, characterized in that: include: Data acquisition step, acquiring input data; A data processing step, inputting the input data into a time series handwriting generation model for data processing; A result output step, outputting handwriting data according to the data processing result; The components of the temporal handwriting generation model include: an encoder and a decoder; The encoder consists of a residual block and a BiLSTM structure.

2. The method for generating time series handwriting data according to claim 1, characterized in that: The encoder extracts deep features of input data through the convolutional network in the residual block, superimposes the input data with the output of the convolutional network through the residual block, and processes the timing information in the input data through BiLSTM, thereby converting the input data into a high-dimensional data distribution.

3. The method for generating time series handwriting data according to claim 1, characterized in that: The decoder comprises: an inflection point module; The inflection point module is used to extract the inflection points of the handwriting from the input data, connect the inflection points with straight lines in order to form a handwriting topological structure, and learn a weight matrix, wherein the weight values ​​in the weight matrix represent the degree to which the feature layer is affected by the topological structure in each dimension; The feature layer in the inflection point module is used for data processing.

4. The method for generating time series handwriting data according to claim 3, characterized in that: The inflection point is calculated using a curvature algorithm; the dimension of the weight matrix is ​​N×D×I; The closer to the inflection point in the topological structure, the higher the weight value in the feature layer.

5. The method for generating time series handwriting data according to claim 3, characterized in that: The decoder further comprises: a trend module, a cycle module, and a basic decoder module; The trend module is used to form a feature layer according to the trend information in the input data; The periodic module is used to form a feature layer according to periodic information in the input data; The basic decoder module, as a residual branch, forms a feature layer according to the input data; The feature layers of each module in the decoder are processed by summing, and the summed feature layers are used for data processing.

6. The method for generating time series handwriting data according to claim 1, characterized in that: The training steps of the encoder and decoder include: The original data is input into the encoder to output the latent space data, and the latent space data is input into the decoder to output the generated data. The loss functions of the encoder and decoder are calculated through the original data and the generated data. The model parameters of the encoder and decoder are optimized according to the loss functions of the encoder and decoder to complete the training of the encoder and decoder.

7. The method for generating time series handwriting data according to claim 6, characterized in that: The training steps of the encoder and decoder include: Treat the original data as a distribution of high-dimensional data The data set {x1,x2…x n } as from The samples obtained by sampling in the 1:T is a time series data of length T, x t is the data at time t, the encoder converts x 1:T Mapping to latent space Get h 1:T , and then the decoder will h 1:T Mapping back to space get The relationship between encoder e and decoder h is: h t =e(h t-1 ,x t ) The loss function used to optimize the model parameters of the encoder and decoder is L ed , the calculation formula is as follows:

8. A device for generating time series handwriting data, characterized in that: include: A data acquisition module, used for acquiring input data; A data processing module, used for inputting input data into the time-series handwriting generation model for data processing; A result output module, outputting handwriting data according to the data processing result; The components of the temporal handwriting generation model include: an encoder and a decoder; The encoder consists of a residual block and a BiLSTM structure.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images