A multi-style calligraphy digital ink simulation method and system based on deep learning

Abstract

The present invention provides a method and system for simulating multi-style calligraphy digital ink based on deep learning, the method comprising: preprocessing an input image to obtain a preprocessed input image; initializing a style feature vector; extracting input image features and preliminarily fusing them with the style feature vector; further fusing input image features and style feature vectors using a feature fusion network; decoding using a decoder to generate an image with an input image stroke structure and a specified calligraphy style; calculating a loss function based on a real image and a generated image, and training the network through back propagation. The technical solution of the present invention does not require cumbersome steps, and end-to-end calligraphy digital ink simulation is achieved; it can easily support simulation of multiple styles, and the simulation effect is more realistic and natural; it can be used not only for calligraphy digital ink simulation, but also for changing the style of paper writing images or computer font images, and has a wider range of uses.

Description

Technical Field

[0001] The present invention belongs to the field of computer image processing and computer vision technology, and in particular to a multi-style calligraphy digital ink simulation method and system based on deep learning. Background Art

[0002] Calligraphy is the art of writing Chinese characters and the crystallization of my country's 5,000-year-old brilliant civilization. The beauty of calligraphy not only includes the beauty of the stroke layout of Chinese characters, but more importantly, the beauty of the handwriting created by the techniques of lifting, pressing, turning, etc. However, in today's information age, people often use touch screens instead of paper to write, lacking traditional brushwork. The original digital ink handwriting is uniform in thickness, looks stiff, rigid, and not beautiful. Therefore, how to simulate real and beautiful digital ink is worth studying.

[0003] Although the existing methods can produce more beautiful results, they still have some shortcomings. For example, patent CN200910193516.1 "A method for beautifying handwritten Chinese characters based on trajectory analysis" and patent CN201110385275.8 "Electronic brush writing handwriting generation method" both first process the writing trajectory information and then use a simulated brush to render the strokes. These methods can achieve good results, but the shape of the brush needs to be manually debugged to get better results, and the simulated brush usually uses a single shape such as an ellipse or a circle, which is difficult to simulate the various strokes of real writing. Some methods also obtain more input trajectory information through a dedicated input device to obtain a better simulation effect. For example, patent CN201110412817.6 "A method and device for beautifying handwriting input" obtains additional handwriting pressure information from the input device. Although this method can improve the simulation effect to a certain extent, it requires additional input devices, is costly and not universal enough.

[0004] In addition, there are also methods of simulating calligraphy digital ink by fusion of strokes, such as patent CN201310407336.5 "A method for generating personalized Chinese digital ink" and patent CN201910377779.1 "A method and device for beautifying handwritten fonts using AI technology", which first disassemble the standard font library and input trajectory, then match the standard strokes that are most similar to the input strokes, and finally fuse the standard strokes with the input strokes to simulate calligraphy handwriting. Although this method can obtain better results and supports a variety of optional styles, it requires the font library to be disassembled, so it takes a certain cost to construct a font library of multiple styles, and the result of stroke fusion is unnatural.

[0005] In recent years, neural style transfer based on deep learning and convolutional neural networks has achieved great success. For example, patent CN201911060607.8 "An image processing method and related equipment" proposes an image processing method based on neural style transfer, but this image processing method is for ordinary images, and its effect is to change the color, texture, brightness and other styles of the image, rather than changing the calligraphy style of the text in the image. Moreover, the style mapping parameter set is obtained by training the parameter neural network and weighted averaging and splicing the intermediate layers, which requires additional training of the parameter neural network.

[0006] There are also some methods that use neural style transfer to generate new fonts, but these methods can generate fonts of various styles well, but these methods fuse the content of the input image with the target style. The effect is to generate an image with the text in the input image and the target style, but the stroke structure of the generated image is usually different from the input image. Therefore, when the target style is the same, the content of the input image is the same, but the writing method is different, the generated result is the same. Calligraphy digital ink simulation, on the other hand, fuses the input image content, stroke style, and target style. The effect is to generate an image with the text and stroke structure in the input image, as well as the target calligraphy style. When the input image content is the same but the writing method is different, even if the target style is the same, the generated result is different. Therefore, font style transfer is essentially different from calligraphy digital ink simulation. Summary of the invention

[0007] In response to the above technical problems, the present invention discloses a multi-style calligraphy digital ink simulation method and system based on deep learning, establishes a style feature vector for each style, and uses an encoder-decoder framework based on deep learning neural network to first extract input image features and fuse them with style features, and then generate an image with the input stroke structure and specific calligraphy style through the fused features, that is, a calligraphy digital ink simulation image.

[0008] To this end, the technical solution adopted by the present invention is:

[0009] A multi-style calligraphy digital ink simulation method based on deep learning, comprising the following steps:

[0010] Step S1, preprocessing the input image to obtain a preprocessed input image;

[0011] Step S2, initializing the style feature vector;

[0012] Step S3, extracting input image features and preliminarily integrating them with the style feature vector;

[0013] Step S4, using a feature fusion network to further fuse the input image features and the style feature vector;

[0014] Step S5, decoding using a decoder to generate an image having the stroke structure of the input image and a specified calligraphy style;

[0015] Step S6, calculating the loss function based on the real image and the generated image, and training the network through back propagation.

[0016] The input image in step S1 is an image containing any handwriting, including any digital ink writing track, any paper writing, any computer font, etc. The real image in step S6 is a calligraphy image with the same stroke structure as the input image and a specified style.

[0017] As a further improvement of the present invention, if the input is an online writing trajectory, it is first converted into an image and then input.

[0018] As a further improvement of the present invention, step S1 includes:

[0019] Step S11, converting the input image into a grayscale image; further, performing binarization processing on the input image;

[0020] Step S12, extracting the skeleton of the grayscale image;

[0021] Step S13, scaling the image to a fixed size;

[0022] Step S14, normalize the image; further, control the grayscale value of the image between [0, 1].

[0023] As a further improvement of the present invention, in step S2, the style feature vectors include multiple types, and each style corresponds to a style feature vector.

[0024] As a further improvement of the present invention, in step S2, the initialization style feature vector uses a specific value or a random value to initialize the style feature vector.

[0025] As a further improvement of the present invention, step S3 adopts an input-end fusion method, a feature-end fusion method or a style coding fusion method for fusion;

[0026] The input-end fusion method selects a style feature vector according to the desired output style, concatenates it with the preprocessed input image, and then inputs it into the image encoder for encoding to obtain a preliminary fused feature;

[0027] The feature-side fusion method first uses an image encoder to encode the preprocessed input image to obtain input image features, then selects a style feature vector according to the required output style, and finally concatenates the input image features with the style feature vector to obtain preliminary fused features;

[0028] The style coding fusion method first uses an image encoder to encode the preprocessed input image to obtain input image features, then selects a style feature vector according to the required output style, and encodes it using the style encoder, and finally concatenates the input image features with the encoded style feature vector to obtain preliminary fused features.

[0029] As a further improvement of the present invention, the image encoder and the style encoder are both neural networks including multiple convolutional layers and pooling layers.

[0030] As a further improvement of the present invention, in step S5, the decoder upsamples the fused features, and the size of the output image is consistent with the size of the preprocessed input image.

[0031] As a further improvement of the present invention, in step S6, the calculation of the loss function includes:

[0032] Step S61, using a pre-trained neural network, calculating the perceptual loss and style loss of the real image and the generated image;

[0033] Step S62, calculating the pixel level loss between the real image and the generated image;

[0034] Step S63, performing weighted summation of the perceptual loss, style loss and pixel level loss to obtain a loss function.

[0035] Furthermore, in step S6, when training the network, it is possible to choose to train the style feature vector or not to train the style feature vector.

[0036] Furthermore, on the basis of step S63, the network structure of the present invention can be regarded as an image generator, and an additional image discriminator can be constructed to calculate the loss of the generative adversarial network, and the image discriminator and the image generator can be alternately trained using the training method of the generative adversarial network until convergence.

[0037] The present invention also discloses a multi-style calligraphy digital ink simulation system based on deep learning, which comprises:

[0038] An image preprocessing module preprocesses the input image to obtain a preprocessed input image;

[0039] Style encoding module, initializes style feature vectors, and uses style feature vectors to represent various calligraphy styles;

[0040] The preliminary fusion module extracts the input image features and preliminarily fuses them with the style feature vector;

[0041] The fusion module uses a feature fusion network to further fuse the input image features and the style feature vector;

[0042] A decoding module, which uses a decoder to decode and generate an image with the stroke structure of the input image and a specified calligraphy style;

[0043] The training network module calculates the loss function based on the real image and the generated image, and trains the network through back propagation.

[0044] As a further improvement of the present invention, the image preprocessing module includes:

[0045] Image conversion module, converts the input image into a grayscale image;

[0046] The skeleton extraction module extracts the skeleton of the grayscale image to obtain a skeleton image;

[0047] Image scaling module, which scales the skeleton image to a fixed size;

[0048] A normalization processing module is used to normalize the scaled image;

[0049] In the training network module, the calculation of the loss function includes: using a pre-trained neural network to calculate the perceptual loss and style loss of the real image and the generated image; calculating the pixel-level loss of the real image and the generated image; and obtaining the loss function by weighted summation of the perceptual loss, style loss and pixel-level loss.

[0050] The multi-style calligraphy digital ink simulation system based on deep learning of the technical solution of the present invention can be deployed on various platforms or devices, supporting simulation of multiple styles and achieving realistic and natural calligraphy digital ink simulation effects.

[0051] The present invention also discloses a computer-readable storage medium, characterized in that the computer-readable storage medium includes a computer program, and when the computer program runs on an electronic device, the electronic device executes the multi-style calligraphy digital ink simulation method based on deep learning as described in any of the above items.

[0052] The present invention also discloses a device comprising a connected processor and a memory, characterized in that: the processor is used to execute a computer program stored in the memory to execute the multi-style calligraphy digital ink simulation method based on deep learning as described in any of the above items.

[0053] The technical solution of the present invention uses specific values ​​or random values ​​to initialize style features, and then adjusts the values ​​of style features during training, and adopts three methods: input-end fusion, feature-end fusion, and style encoding fusion. The weighted sum of perceptual loss, style loss, and pixel-level loss is used as the loss function, and better results can be obtained without using a generative adversarial network.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] Firstly, the present invention does not require complicated steps and realizes end-to-end calligraphy digital ink simulation; secondly, the present invention can easily support simulation of various styles, and the simulation effect is more realistic and natural; finally, the present invention only needs to input the position information of the writing trajectory, so it can not only be used for calligraphy digital ink simulation, but also can be used to change the style of paper writing images or computer font images, and has a wider range of uses. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of an embodiment of the present invention.

[0057] Figure 2 Schematic diagram of the network structure of an embodiment of the present invention, which adopts the style coding fusion method.

[0058] Figure 3 This is an experimental result diagram of an embodiment of the present invention rendering a calligraphy font skeleton into its original style, wherein the first and fourth columns are input images, the second and fifth columns are generated images, the third and sixth columns are real images, and each row represents a different font style.

[0059] Figure 4 1 is a graph showing experimental results of generating a variety of calligraphy styles with standard font weights according to an embodiment of the present invention, wherein the first column is an input image, and the other columns are generated images of different calligraphy styles.

[0060] Figure 5 This is a diagram of experimental results of generating a variety of fonts and font weight styles according to an embodiment of the present invention, wherein the first column is an input image, the other columns are generated images of different font weight styles, and each row represents a different font style. DETAILED DESCRIPTION

[0061] In order to more clearly demonstrate the purpose and technical solution of the present invention, an example of the present invention is further described below in detail.

[0062] Figure 1 This is a flowchart of a calligraphy digital ink simulation technology based on deep learning and including multiple styles according to an embodiment of the present invention. Figure 1 To illustrate some specific embodiments of the present invention. The present invention establishes a style feature vector for each style, uses an encoder-decoder framework based on a convolutional neural network, first extracts input image features and fuses them with style features, and then generates an image with an input stroke structure and a specific calligraphy style through the fused features, i.e., a calligraphy digital ink simulation image; the main steps include:

[0063] Step S1, preprocessing the input image to obtain a preprocessed input image I;

[0064] In this embodiment, the preprocessing of the input image includes conversion to grayscale, skeleton extraction, scaling, and normalization. In addition, this embodiment also performs binarization on the grayscale image to make skeleton extraction more accurate, and performs morphological expansion operation on the skeleton image to improve the quality of the generated image. The size of the preprocessed image is 128×128.

[0065] Step S2, initializing the style feature vector V s ; Each style corresponds to a style feature vector, and the style feature vector corresponding to the i-th style (1≤i≤n) is

[0066] In this embodiment, the style feature vector is randomly initialized during training, and the style feature vector is trained together with the network. This embodiment includes 8 font styles, each font has 6 weight styles, so there are 48 styles in total. This embodiment initializes a 128×128 style feature vector for each font and each weight, and then combines the font style feature vector and the weight style feature vector as the feature vector of each style.

[0067] Step S3, extracting input image features and preliminarily integrating them with the style feature vector;

[0068] like Figure 2 As shown in the network structure diagram, this embodiment uses the style coding fusion method, that is, first use the image encoder E I (x) Encode I and then select the style feature vector according to the required output style And use style encoder E S (x) encode it, and finally E I (I) with The initial fusion features are obtained by splicing.

[0069] The image encoder E used in this embodiment I (x) is a neural network containing multiple convolutional layers and pooling layers, and the results of all convolutional layers are batch normalized. Style encoder E S The structure of (x) is exactly the same as that of the image encoder.

[0070] Step S4, using the feature fusion network R(x) to further fuse the input image features and the style feature vector;

[0071] The feature fusion network R(x) used in this embodiment is a residual network including multiple convolutional layers and residual connections.

[0072] Step S5, decoding using a decoder D(x) to generate an image having the stroke structure of the input image and a specified calligraphy style;

[0073] The decoder D(x) used in this embodiment is a neural network including multiple convolutional layers and deconvolutional layers. After decoding, a generated image with a dimension of 128×128 is obtained.

[0074] Step S6, calculating the loss function based on the real image and the generated image, and training the network parameters and style feature vector through back propagation.

[0075] The experimental results of using the above method to render the calligraphy font skeleton into its original style are shown in the figure below. Figure 3 shown.

[0076] The data set used in this embodiment is constructed based on 8 calligraphy fonts with relatively different styles, including Founder Liu Gongquan regular script, Hanyi Xuejun style, Founder Shen Yinmo running script, Founder Liu Bingsen official script, Hanyi Weibei, Founder Cuan Baozibei regular script, Founder Slender Gold script and Founder hard pen running script. Each font itself contains 2 weights, which are expanded to 6 weights through morphological corrosion and expansion operations. The data set contains a total of 48 calligraphy styles. The experimental results of generating multiple calligraphy styles with standard weights using the method of this embodiment are shown in the figure. Figure 4 As shown in FIG. 1 , the first column is the input image, and the other columns are generated images of different calligraphy styles. The experimental results of using the method of this embodiment to generate multiple fonts and weight styles are shown in FIG. Figure 5 As shown, the first column is the input image, the other columns are generated images of different font weights, and each row represents a different font style.

[0077] The training goal of the model in this embodiment is to make the generated image as similar as possible to the real image. This embodiment calculates the perceptual loss l according to the method in the document "Johnson J, Alahi A, Fei-Fei L. Perceptual losses for real-time style transfer and super-resolution [C] / / European conference on computer vision. Springer, Cham, 2016: 694-711." per and style loss l style , the pixel-level loss selected L2 loss l2, and the generative adversarial network loss l was not used GAN ; The final loss function l = l per +500l style +l2.

[0078] The present invention also discloses a multi-style calligraphy digital ink simulation system based on deep learning, which comprises:

[0079] An image preprocessing module preprocesses the input image to obtain a preprocessed input image;

[0080] Style encoding module, initializes style feature vectors, and uses style feature vectors to represent various calligraphy styles;

[0081] The preliminary fusion module extracts the input image features and preliminarily fuses them with the style feature vector;

[0082] The fusion module uses a feature fusion network to further fuse the input image features and the style feature vector;

[0083] A decoding module, which uses a decoder to decode and generate an image with the stroke structure of the input image and a specified calligraphy style;

[0084] The training network module calculates the loss function based on the real image and the generated image, and trains the network through back propagation.

[0085] Wherein, the image preprocessing module includes:

[0086] Image conversion module, converts the input image into a grayscale image;

[0087] The skeleton extraction module extracts the skeleton of the grayscale image to obtain a skeleton image;

[0088] Image scaling module, which scales the skeleton image to a fixed size;

[0089] A normalization processing module is used to normalize the scaled image;

[0090] In the training network module, the calculation of the loss function includes: using a pre-trained neural network to calculate the perceptual loss and style loss of the real image and the generated image; calculating the pixel-level loss of the real image and the generated image; and obtaining the loss function by weighted summation of the perceptual loss, style loss and pixel-level loss.

[0091] The multi-style calligraphy digital ink simulation system based on deep learning in the embodiment of the present invention can be deployed on various platforms or devices, support simulation of multiple styles, and achieve realistic and natural calligraphy digital ink simulation effects.

[0092] An embodiment of the present invention discloses a computer-readable storage medium, characterized in that the computer-readable storage medium includes a computer program, and when the computer program runs on an electronic device, the electronic device executes the multi-style calligraphy digital ink simulation method based on deep learning as described in any one of the above items.

[0093] An embodiment of the present invention discloses a device comprising a connected processor and a memory, wherein the processor is used to execute a computer program stored in the memory to execute a multi-style calligraphy digital ink simulation method based on deep learning as described in any one of the above items.

[0094] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A multi-style calligraphy digital ink simulation method based on deep learning, characterized in that: It includes the following steps: Step S1, preprocessing the input image to obtain a preprocessed input image; Step S2, initializing the style feature vector; Step S3, extracting input image features and preliminarily integrating them with the style feature vector; Step S4, using a feature fusion network to further fuse the input image features and the style feature vector; Step S5, decoding using a decoder to generate an image having the stroke structure of the input image and a specified calligraphy style; Step S6, calculating the loss function according to the real image and the generated image, and training the network through back propagation; Step S1 includes: Step S11, converting the input image into a grayscale image; Step S12, extracting the skeleton of the grayscale image; Step S13, scaling the image to a fixed size; Step S14, normalizing the image; In step S2, the style feature vectors include multiple types, and each style corresponds to a style feature vector; In step S2, the initialization style feature vector uses a specific value or a random value to initialize the style feature vector; Step S3 adopts an input-end fusion method, a feature-end fusion method or a style coding fusion method for fusion; wherein, in the input-end fusion method, a style feature vector is selected according to the desired output style, and it is concatenated with the preprocessed input image, and then input into the image encoder for encoding to obtain a preliminary fused feature; in the feature-end fusion method, an image encoder is first used to encode the preprocessed input image to obtain input image features, and then a style feature vector is selected according to the desired output style, and finally the input image features are concatenated with the style feature vector to obtain a preliminary fused feature; The style coding fusion method first uses an image encoder to encode the preprocessed input image to obtain input image features, then selects a style feature vector according to the required output style and encodes it using the style encoder, and finally concatenates the input image features with the encoded style feature vector to obtain preliminary fused features; In step S5, the decoder upsamples the fused features, and the size of the output image is consistent with the size of the preprocessed input image; In step S6, the calculation of the loss function includes: Step S61, using a pre-trained neural network, calculating the perceptual loss and style loss of the real image and the generated image; Step S62, calculating the pixel level loss between the real image and the generated image; Step S63, weighted sum of the perceptual loss, style loss and pixel level loss to obtain a loss function; the loss function l=l per +500l style +l2; Among them, l per is the perceptual loss, l style is the style loss and l2 is the pixel level loss.

2. The multi-style calligraphy digital ink simulation method based on deep learning according to claim 1 is characterized in that: In step S1, the input image is an image containing any handwriting; if the input is an online writing trajectory, it is first converted into an image and then input.

3. A multi-style calligraphy digital ink simulation system based on deep learning, characterized in that: It includes: An image preprocessing module preprocesses the input image to obtain a preprocessed input image; The style encoding module initializes the style feature vector and uses the style feature vector to represent various calligraphy styles. The preliminary fusion module extracts the input image features and preliminarily fuses them with the style feature vector. The fusion module uses a feature fusion network to further fuse the input image features and the style feature vector; A decoding module, which uses a decoder to decode and generate an image with the stroke structure of the input image and a specified calligraphy style; The training network module calculates the loss function based on the real image and the generated image, and trains the network through back propagation; The image preprocessing module comprises: Image conversion module, converts the input image into a grayscale image; The skeleton extraction module extracts the skeleton of the grayscale image to obtain a skeleton image; Image scaling module, which scales the skeleton image to a fixed size; A normalization processing module is used to normalize the scaled image; In the training network module, the calculation of the loss function includes: using a pre-trained neural network to calculate the perceptual loss and style loss of the real image and the generated image; calculating the pixel-level loss of the real image and the generated image; weighted summing the perceptual loss, style loss and pixel-level loss to obtain a loss function; the loss function l=l per +500l style +l2; Among them, l per is the perceptual loss, l style is the style loss, l2 is the pixel level loss; The preliminary fusion module adopts an input-end fusion method, a feature-end fusion method or a style coding fusion method for fusion; Among them, the input-end fusion method selects a style feature vector according to the required output style, concatenates it with the preprocessed input image, and then inputs it into the image encoder for encoding to obtain preliminary fused features; the feature-end fusion method first uses the image encoder to encode the preprocessed input image to obtain input image features, then selects a style feature vector according to the required output style, and finally concatenates the input image features with the style feature vector to obtain preliminary fused features; The style coding fusion method first uses an image encoder to encode the preprocessed input image to obtain input image features, then selects a style feature vector according to the required output style, and encodes it using the style encoder, and finally concatenates the input image features with the encoded style feature vector to obtain preliminary fused features; the decoder upsamples the fused features, and the output image size is consistent with the preprocessed input image size.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a computer program, and when the computer program is run on an electronic device, the electronic device executes the multi-style calligraphy digital ink simulation method based on deep learning as described in any one of claims 1 to 2.

5. A computer device comprising a processor and a memory connected together, characterized in that: The processor is used to execute the computer program stored in the memory to execute the multi-style calligraphy digital ink simulation method based on deep learning as described in any one of claims 1 to 2.

Images