Handwriting erasing method, device, equipment and readable storage medium

Abstract

The present application relates to a method, device, equipment and readable storage medium for erasing handwriting, and relates to the technical field of image recognition and processing, including inputting a picture to be erased into a target detection model to obtain multiple handwriting areas and vertex coordinates of each handwriting area; cutting the picture corresponding to each detection frame in the detection frame set formed by the vertex coordinates, and semantically segmenting the background, printed text and handwritten text in the cut picture based on a semantic segmentation model to obtain an initial three-dimensional matrix; updating the pixel values ​​corresponding to the background and printed text in the initial three-dimensional matrix to a first preset value, and updating the pixel values ​​corresponding to the handwritten text to a second preset value, to obtain a binary handwriting mask; inputting the binary handwriting mask and its corresponding cut picture into a multivariate image restoration model to obtain a restored image after handwriting is removed; and covering the picture to be erased based on the restored image after handwriting is removed to obtain a target picture without handwriting, thereby effectively avoiding mis-segmentation and missed segmentation, and improving the accuracy of document restoration.

Description

Technical Field

[0001] The present application relates to the technical field of image recognition processing, and in particular to a handwriting erasing method, device, equipment and readable storage medium. Background Art

[0002] With the transformation of society and the change of educational concepts, the function of education has become increasingly socialized, and the subjects of education have become increasingly diversified, which has attracted more and more attention to the influence of parents on their children's academic achievements. Under the background of home-school cooperation, the connection between parents and schools has become increasingly close. Parents' tutoring of primary school students' homework has become an important part of the current home-school cooperation. In the practice of parents' participation in their children's education, parents' participation in homework is also an important part.

[0003] Among them, when parents help their children with their homework, when the child writes the wrong answer on the test paper or exercise book, they need to use an eraser to erase it. If they write the wrong answer again, they need to erase it again. After repeating this process several times, the test paper or exercise book is no longer the same as it was before. It is so unrecognizable that it is impossible to continue writing the answer. Therefore, how to erase the handwriting to restore the original appearance of the test paper has become an urgent problem to be solved. Document restoration technology came into being in this context. It is a restoration technology that electronically removes handwriting on paper and is widely used in education, office and other fields.

[0004] However, since document restoration technology extracts handwriting by directly performing semantic segmentation on the entire image, when the handwriting in the image is very small, it will lead to missed segmentation; and due to the large difference between printed and background features, in the document restoration process, printed text is often mistakenly segmented into handwritten text, resulting in poor accuracy of document restoration. Summary of the invention

[0005] The present application provides a handwriting erasing method, device, equipment and readable storage medium to solve the problem of poor accuracy of document restoration caused by missed segmentation and wrong segmentation in the document restoration technology in the related art.

[0006] In a first aspect, a method for erasing handwriting is provided, comprising the following steps:

[0007] Inputting the image to be erased into the target detection model to obtain multiple handwriting areas containing handwriting and vertex coordinates of each handwriting area;

[0008] A detection frame set is formed based on the vertex coordinates of multiple handwriting areas, and the image corresponding to each detection frame in the detection frame set is cut to obtain multiple cut images;

[0009] Based on the semantic segmentation model, semantic segmentation is performed on the background, printed text and handwritten text in each cut image to obtain the initial three-dimensional matrix corresponding to each cut image;

[0010] Respectively updating the pixel values ​​corresponding to the background and the printed text in the initial three-dimensional matrix corresponding to each cut image to the first preset value, and updating the pixel values ​​corresponding to the handwritten text to the second preset value, so as to obtain a binary handwriting mask corresponding to each cut image;

[0011] Each binary handwriting mask and its corresponding cut image are input into the multivariate image restoration model to obtain the restored image after handwriting removal;

[0012] Based on the restored image after handwriting removal, the handwriting area on the image to be erased corresponding to the restored image after handwriting removal is covered to obtain a target image without handwriting.

[0013] In some embodiments, forming a set of detection frames based on vertex coordinates of multiple handwriting areas includes:

[0014] Forming a corresponding first detection frame according to the vertex coordinates of each handwriting area;

[0015] Detect whether the intersection-over-union ratio and the distance between two adjacent first detection frames are both greater than the corresponding merging threshold;

[0016] If so, merge the two adjacent first detection frames to obtain a second detection frame, and determine whether the width and height of the second detection frame are greater than the corresponding segmentation threshold. When the width of the second detection frame is greater than the width segmentation threshold and / or the height of the second detection frame is greater than the height segmentation threshold, segment the second detection frame to obtain multiple second sub-detection frames. When the width of the second detection frame is less than or equal to the width segmentation threshold and the height of the second detection frame is less than or equal to the height segmentation threshold, the second detection frame is not segmented.

[0017] If not, the two adjacent first detection frames are not merged, and it is determined whether the width and height of the first detection frame are greater than the corresponding segmentation threshold. When the width of the first detection frame is greater than the width segmentation threshold and / or the height of the first detection frame is greater than the height segmentation threshold, the first detection frame is segmented to obtain multiple first sub-detection frames. When the width of the first detection frame is less than or equal to the width segmentation threshold and the height of the first detection frame is less than or equal to the height segmentation threshold, the first detection frame is not segmented.

[0018] The unmerged and unsplit first detection frame, the first sub-detection frame, the unsplit second detection frame and the second sub-detection frame are taken as a detection frame set.

[0019] In some embodiments, before the step of detecting whether the intersection-over-union ratio and the distance between two adjacent first detection frames are both greater than the corresponding merging threshold, the method further includes:

[0020] Non-maximum suppression processing is performed on all first detection frames to delete redundant first detection frames, the confidence threshold in the non-maximum suppression processing is set to 0.05, and the intersection-over-union ratio threshold in the non-maximum suppression processing is set to 0.95.

[0021] In some embodiments, before the step of performing semantic segmentation on the background, printed text and handwritten text in each cut picture based on the semantic segmentation model, the method further includes:

[0022] Merge multiple cut images into one batch data;

[0023] The batch data is input into the semantic segmentation model.

[0024] In some embodiments, the multivariate image restoration model includes an expression network and a generation network, the input of the output layer of the second decoding module in the generation network is a splicing graph formed by the output of the residual upsampling unit of the second decoding module and the output of the first encoding module in the expression network; the input of the output layer of the third decoding module in the generation network is a splicing graph formed by the output of the residual upsampling unit of the third decoding module and the output of the initial residual module in the expression network.

[0025] In some embodiments, the long-short-term attention module in the generative network is used to:

[0026] Calculating short-term inter-layer attention features of the decoded features output by the first decoding module in the generator network;

[0027] Calculating long-term inter-layer attention features of the encoded features output by the second encoding module in the expression network;

[0028] The short-term inter-layer attention features and the long-term inter-layer attention features are aggregated and used as input of the residual upsampling unit of the second decoding module.

[0029] In some embodiments, after the step of performing overwriting processing on the image to be erased and corresponding to the restored image after the handwriting is removed based on the restored image after the handwriting is removed to obtain a target image without handwriting, the step further includes:

[0030] The target image without handwriting is binarized to obtain a binary image without handwriting.

[0031] In a second aspect, a handwriting erasing device is provided, comprising:

[0032] A handwriting detection unit, which is used to input the image to be erased into the target detection model, obtain multiple handwriting areas containing handwriting and vertex coordinates of each handwriting area; form a detection frame set based on the vertex coordinates of the multiple handwriting areas, and cut the image corresponding to each detection frame in the detection frame set to obtain multiple cut images;

[0033] A handwriting segmentation unit is used to perform semantic segmentation on the background, printed text and handwritten text in each cut picture based on the semantic segmentation model to obtain an initial three-dimensional matrix corresponding to each cut picture; update the pixel values ​​corresponding to the background and printed text in the initial three-dimensional matrix corresponding to each cut picture to a first preset value, and update the pixel values ​​corresponding to the handwritten text to a second preset value to obtain a binary handwriting mask corresponding to each cut picture;

[0034] An image restoration unit, which is used to input each binary handwriting mask and its corresponding cut picture into a multivariate image restoration model to obtain a restored image after handwriting removal;

[0035] The image stitching unit is used to cover the handwriting area corresponding to the restored image after handwriting removal on the image to be erased based on the restored image after handwriting removal to obtain a target image without handwriting.

[0036] In a third aspect, a handwriting erasing device is provided, comprising: a memory and a processor, wherein at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the aforementioned handwriting erasing method.

[0037] In a fourth aspect, a computer-readable storage medium is provided, characterized in that: the computer storage medium stores a computer program, and when the computer program is executed by a processor, the aforementioned handwriting erasing method is implemented.

[0038] The beneficial effects brought about by the technical solution provided by the present application include: effectively avoiding mis-segmentation and missed segmentation, thereby improving the accuracy of document restoration.

[0039] The present application provides a method, device, equipment and readable storage medium for erasing handwriting, comprising inputting a to-be-erased image into a target detection model to obtain a plurality of handwriting regions containing handwriting and vertex coordinates of each handwriting region; forming a detection frame set based on the vertex coordinates of the plurality of handwriting regions, and cutting the image corresponding to each detection frame in the detection frame set to obtain a plurality of cut images; performing semantic segmentation on the background, printed text and handwriting in each cut image based on a semantic segmentation model to obtain an initial three-dimensional matrix corresponding to each cut image; updating the pixel values ​​corresponding to the background and printed text in the initial three-dimensional matrix corresponding to each cut image to a first preset value and the pixel values ​​corresponding to the handwriting to a second preset value to obtain a binary handwriting mask corresponding to each cut image; inputting each binary handwriting mask and its corresponding cut image into a multivariate image restoration model to obtain a restored image after handwriting is removed; and performing overwriting processing on the handwriting region corresponding to the restored image after handwriting is removed on the to-be-erased image based on the restored image after handwriting is removed to obtain a target image without handwriting. The present application detects and locates handwriting in the image to be erased, and cuts the image corresponding to the located detection frame to obtain multiple smaller cut images. The smaller images have fewer interference items, and the features of handwritten handwriting in the images are more obvious. At the same time, semantic segmentation of three targets including background, printed text and handwritten text is performed on the smaller images, effectively avoiding mis-segmentation and missed segmentation, thereby improving the accuracy of document restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A schematic diagram of a process for erasing handwriting provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of the image restoration model structure provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of the structure of a handwriting erasing device provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of the structure of a handwriting erasing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0046] The embodiments of the present application provide a handwriting erasing method, device, equipment and readable storage medium, which can solve the problem of poor document restoration accuracy caused by missed segmentation and incorrect segmentation in the document restoration technology in the related art.

[0047] Figure 1 A method for erasing handwriting provided in an embodiment of the present application comprises the following steps:

[0048] Step S10: inputting the image to be erased into the target detection model to obtain a plurality of handwriting regions including handwriting and vertex coordinates of each handwriting region;

[0049] Exemplarily, in an embodiment of the present application, a request for erasing handwriting is first obtained, the request data is decoded to obtain the image to be restored corresponding to the document to be restored, and it is determined whether there is any abnormality in the image to be restored. If the image to be restored cannot be converted from Base64 (Base64 is one of the most common encoding methods for transmitting 8-bit bytecodes, which is a method of representing binary data based on 64 printable characters) format to a format available to OpenCV (OpenCV is a cross-platform computer vision and machine learning software library that can run on different operating systems), the image is considered to be abnormal, and the original image will be directly returned at this time, and operations such as segmentation, image restoration, and image stitching will no longer be performed; and if the image to be restored can be converted from Base64 format to a format available to OpenCV, the image is considered to have no abnormality, and the image to be erased is obtained, and handwriting detection will be performed on the image to be erased.

[0050] Since the handwriting size varies and the content aspect ratio varies, this embodiment adopts the anchor-free target detection algorithm VFNet (VarifocalNet, a new network for single-stage target detection) as the target detection model and trains it. VFNet is an improved algorithm based on FCOS (first-order full convolution target detection). It is not only a pixel-based (semantic segmentation based on pixel level) prediction, but also considers the target confidence and positioning accuracy while predicting, so that when the application performs non-maximum suppression (NMS) on the detection frame, it can avoid the problem of removing the prediction frame due to "more accurate positioning, but lower confidence", so VFNet not only has a high recall rate, but also a high accuracy rate.

[0051] Compared with FCOS, VFNet has the following optimizations: FCOS algorithm only predicts the target confidence, while VFNet uses Varifocal Loss to predict IACS (IoU-aware classification scores), that is, it considers both target confidence and position accuracy to make positioning more accurate; Star-Shaped Box Feature Representation is used. Representation) can capture the geometric information of the bounding box and the nearby contextual information at the same time. For example, for a pixel point (x, y) on the image, the distances from this position to the upper, lower, left and right of the bounding box are (l, t, r, b) respectively. The star-shaped bounding box feature representation is (x, y), (xl, y), (x, yt), (x+r, y), (x, y+b), (xl, yt), (x+l, yt), (xl, y+b) and (x+r, y+b); VFNet also adds a "bounding box fine-tuning" sub-branch to the regression branch of the FCOS algorithm, that is, a sub-branch is added to the regression branch (l, t, r, b). This sub-branch uses the "star-shaped box feature representation" and uses variable convolution to learn 4 distance scaling factors (Δl, Δt, Δr, Δb). Finally, the fine-tuned bounding box is obtained: (l * ,t * ,r * ,b * )=(Δl×l,Δt×t,Δr×r,Δb×b).

[0052] When creating the VFNet model, in order to enhance the generalization ability of the model, this embodiment also uses data enhancement methods such as grayscale, rotation, flipping, random cropping, smoothing, noise, and color jitter during training. Therefore, when this embodiment inputs the image to be erased into the VFNet model, the position of the handwriting can be used to locate the area where each handwriting in the image to be erased and the vertex coordinates of the area are obtained, that is, multiple handwriting areas containing handwriting and the vertex coordinates of each handwriting area are obtained.

[0053] Step S20: forming a detection frame set based on vertex coordinates of multiple handwriting areas, and cutting the image corresponding to each detection frame in the detection frame set to obtain multiple cut images;

[0054] Furthermore, forming a detection frame set based on vertex coordinates of multiple handwriting areas specifically includes the following steps:

[0055] Forming a corresponding first detection frame according to the vertex coordinates of each handwriting area;

[0056] Detect whether the intersection-over-union ratio and the distance between two adjacent first detection frames are both greater than the corresponding merging threshold;

[0057] If so, merge the two adjacent first detection frames to obtain a second detection frame, and determine whether the width and height of the second detection frame are greater than the corresponding segmentation threshold. When the width of the second detection frame is greater than the width segmentation threshold and / or the height of the second detection frame is greater than the height segmentation threshold, segment the second detection frame to obtain multiple second sub-detection frames. When the width of the second detection frame is less than or equal to the width segmentation threshold and the height of the second detection frame is less than or equal to the height segmentation threshold, the second detection frame is not segmented.

[0058] If not, the two adjacent first detection frames are not merged, and it is determined whether the width and height of the first detection frame are greater than the corresponding segmentation threshold. When the width of the first detection frame is greater than the width segmentation threshold and / or the height of the first detection frame is greater than the height segmentation threshold, the first detection frame is segmented to obtain multiple first sub-detection frames. When the width of the first detection frame is less than or equal to the width segmentation threshold and the height of the first detection frame is less than or equal to the height segmentation threshold, the first detection frame is not segmented.

[0059] The unmerged and unsplit first detection frame, the first sub-detection frame, the unsplit second detection frame and the second sub-detection frame are taken as a detection frame set.

[0060] Exemplarily, in an embodiment of the present application, the VFNet model forms a detection frame based on vertex coordinates. Specifically, a corresponding first detection frame is formed according to the vertex coordinates of each handwriting area, that is, there are several first detection frames for several handwriting areas; then, it is detected whether the intersection and union ratio and the distance between two adjacent first detection frames are both greater than the corresponding merging threshold. For example, if the intersection and union ratio merging threshold is 0 and the distance merging threshold between two adjacent first detection frames is 5, it means that two adjacent first detection frames with an intersection and union ratio greater than 0 and a distance greater than 5 need to be merged, otherwise they do not need to be merged; the two adjacent first detection frames to be merged are merged to obtain a second detection frame, and the width and height of the second detection frame are determined. Whether it is greater than the corresponding segmentation threshold, for example, the width segmentation threshold is 800 and the height segmentation threshold is 800, which means that as long as at least one of the width and height in the second detection frame is greater than 800, the second detection frame needs to be segmented. The segmentation can be equal or unequal, which can be determined according to actual needs and is not limited here; therefore, when the width of the second detection frame is greater than the width segmentation threshold and / or the height of the second detection frame is greater than the height segmentation threshold, the second detection frame is segmented to obtain multiple second sub-detection frames. When the width of the second detection frame is less than or equal to the width segmentation threshold and the height of the second detection frame is less than or equal to the height segmentation threshold, the second detection frame is not segmented;

[0061] In addition, it is also necessary to judge whether the first detection frame that has not been merged needs to be segmented. The judgment method is the same as the method of whether the second detection frame needs to be segmented. Therefore, when the width of the first detection frame is greater than the width segmentation threshold and / or the height of the first detection frame is greater than the height segmentation threshold, the first detection frame is segmented to obtain multiple first sub-detection frames. When the width of the first detection frame is less than or equal to the width segmentation threshold and the height of the first detection frame is less than or equal to the height segmentation threshold, the first detection frame is not segmented. It can be seen that in the whole process, four types of detection frames may be generated, namely, the unmerged and unsegmented first detection frame, the first sub-detection frame, the unsegmented second detection frame and the second sub-detection frame, thereby forming a detection frame set. Then, the image corresponding to each detection frame in the detection frame set is cut to obtain multiple smaller cut images. Since the smaller image has fewer interference items and the features of the handwriting in the image are more obvious, the missed segmentation and mis-segmentation of the semantic segmentation model can be effectively avoided, thereby improving the segmentation accuracy.

[0062] Furthermore, before the step of detecting whether the intersection-over-union ratio and the distance between two adjacent first detection frames are both greater than the corresponding merging threshold, the method further includes:

[0063] Non-extreme value suppression processing is performed on all first detection frames to delete redundant first detection frames, the confidence threshold in the non-extreme value suppression processing is set to 0.05, and the intersection-over-union ratio threshold in the non-extreme value suppression processing is set to 0.95.

[0064] For example, in an embodiment of the present application, before determining whether the first detection frame needs to be merged, the first detection frame can be subjected to non-extreme value suppression processing to delete redundant first detection frames, and in this process, the confidence threshold in the non-extreme value suppression processing can be set to a lower value, such as 0.05, and the intersection-over-union ratio threshold in the non-extreme value suppression processing can be set to a higher value, such as 0.95, thereby effectively preventing missed detections. Of course, the confidence threshold and the intersection-over-union ratio threshold can also be determined according to actual needs, which is not limited here.

[0065] Step S30: Based on the semantic segmentation model, semantic segmentation is performed on the background, printed text and handwritten text in each cut picture to obtain an initial three-dimensional matrix corresponding to each cut picture;

[0066] Exemplarily, in an embodiment of the present application, OCRnet (a semantic segmentation model based on contextual information of object areas) is used as a semantic segmentation model and corresponding training is performed. Its input is a normalized three-channel image tensor, and its output is a binary mask with the same length and width as the input. The value of each position in the output binary mask represents whether the pixel corresponding to the position is part of the "handwriting". For example, if a pixel value is 255, then this pixel point is handwriting, and if a pixel value is 0, then this pixel point is the background or printed text.

[0067] Since the difference between printed features and background features is large, while the difference between printed features and handwritten features is small, it is often easy to mistakenly identify printed as handwritten, and then the problem of mis-segmentation occurs. Therefore, this embodiment divides the existing scheme of dividing the two types of targets into background and handwritten into three types of targets, background, printed and handwritten, and gives weight factors (0.1, 0.3, 0.6) to the cross entropy (cost function), that is, the weight factor of the background is 0.1, the weight factor of the printed body is 0.3, and the weight factor of the handwritten body is 0.6, so as to highlight the importance of handwriting, thereby improving the handwriting segmentation accuracy of the OCRnet model. At the same time, in order to enhance the generalization ability of the OCRnet model, this embodiment also adopts data enhancement methods such as grayscale image, rotation, flipping, random cropping, smoothing, color jittering, etc. Therefore, the segmented picture is input into the OCRnet model, and the OCRnet model performs background, printed body and handwritten segmentation on the segmented picture to obtain the initial three-dimensional matrix corresponding to the segmented picture containing three types of targets: background, printed body and handwritten.

[0068] Furthermore, before the step of performing semantic segmentation on the background, printed text and handwritten text in each cut picture based on the semantic segmentation model, the method further includes:

[0069] Merge multiple cut images into one batch data;

[0070] The batch data is input into the semantic segmentation model.

[0071] For example, in an embodiment of the present application, in order to make full use of the computing power of the GPU (Graphics Processing Unit), in the inference and prediction stage of the OCRnet model, several segmented images can be merged into a batch (i.e., batch data) as the input of the OCRnet model. This method takes advantage of the fast speed of parallel computing, which can reduce the inference and prediction time of the OCRnet model, thereby improving the segmentation efficiency of the OCRnet model.

[0072] Step S40: respectively updating the pixel values ​​corresponding to the background and printed text in the initial three-dimensional matrix corresponding to each cut picture to the first preset value, and updating the pixel values ​​corresponding to the handwritten text to the second preset value, so as to obtain a binary handwriting mask corresponding to each cut picture;

[0073] For example, in the embodiment of the present application, the initial three-dimensional matrix is ​​processed according to different target categories, for example, the pixel values ​​at the background and the printed body are changed to the first preset value, which can be set to 0, and the pixel values ​​at the handwritten body are changed to the second preset value, which can be set to 255, and finally a binary handwriting mask (i.e., a binary handwriting mask) is obtained. It should be noted that the first preset value and the second preset value can be determined according to actual needs and are not limited here.

[0074] Step S50: input each binary handwriting mask and its corresponding cut picture into a multivariate image restoration model to obtain a restored image after handwriting removal;

[0075] Furthermore, the multivariate image restoration model includes an expression network and a generation network, the input of the output layer of the second decoding module in the generation network is a splicing graph formed by the output of the residual upsampling unit of the second decoding module and the output of the first encoding module in the expression network; the input of the output layer of the third decoding module in the generation network is a splicing graph formed by the output of the residual upsampling unit of the third decoding module and the output of the initial residual module in the expression network.

[0076] For example, in the embodiment of the present application, after the aforementioned handwriting segmentation, a relatively accurate binary handwriting mask is obtained; the binary handwriting mask and its corresponding segmented image are then input into the multivariate image restoration model, and a restored image after the handwriting is removed after the filled pixels are output. In this embodiment, the PIC (Pluralistic Image Completion) model is improved to obtain a multivariate image restoration model.

[0077] Specifically, the PIC model is a multivariate image restoration algorithm based on a generative adversarial network. The model includes an expression network, an inference network, and a generation network, and has two parallel routes: one is the reconstruction route ( Figure 2 The upper half of the figure), which uses the only given ground truth to obtain the prior distribution of the missing part and reconstructs the original image based on the distribution. This route is only used in the training stage; the other is the generation route ( Figure 2 The conditional prior is coupled with the distribution obtained in the reconstruction path, which is used in the training and testing phases. Under the premise of ensuring the quality of the model output image, in order to reduce the model's dependence on the input mask, this embodiment improves the PIC model structure and input and generates a multivariate image restoration model (see Figure 2 shown), where Figure 2 The dotted arrows in the figure represent the reconstruction route, which is only used during training; the realized arrows represent the generated route, which is used during training and testing.

[0078] The structures of the expression network, inference network and generation network on the reconstruction route and the generation route are basically the same. The expression networks on the reconstruction route and the generation route both include 1 initial residual module and multiple encoding modules; the inference network in the reconstruction route includes 1 residual module and an acquisition distribution module, while the inference network in the generation route includes 7 residual modules and an acquisition distribution module; the generation networks on the reconstruction route and the generation route both include 1 residual upsampling module, 4 decoding modules and 1 long-short-term attention module.

[0079] Before inputting the data into the multivariate image restoration model, the input cut image and binary handwriting mask need to be processed to obtain the input data of the model. The specific processing process is as follows: 1) The binary handwriting mask and the corresponding cut image are spliced ​​in the image channel dimension to obtain the model input I m , as the input of the generated route; 2) after the binary handwriting mask is inverted, it is spliced ​​with the corresponding cut picture in the image channel dimension to obtain the model input I c , as the input of the reconstruction route, making the network less dependent on the input binary handwriting mask during the testing phase.

[0080] When the multi-image restoration model is in the training phase: in the expression network, multiple encoding modules consisting of residual downsampling modules are used for multi-scale feature extraction; in the inference network, the residual network is used to infer the potential distribution of the input data; the distribution is sampled and added to the non-mask area features obtained by the last encoder in the inference network of the generation route to obtain the initial input of the generation network; in the generation network, multiple decoding modules consisting of upsampling residual modules and output layers and long-short-term attention modules are used to perform image generation operations. In this process, low-level features are also added to improve the quality of the output image, and finally the reconstructed image and the generated image are obtained. When the multi-image restoration model is in the testing phase: it is basically similar to the training phase, but only the operations on the generation route are performed to finally obtain the generated image.

[0081] Among them, the initial residual module used in the expression network contains two parallel process structures. The structure of process 1 is: 1) convolution layer with a convolution kernel of 3×3, 2) LeakyRelu (0.1) activation layer, where 0.1 represents the value of the hyperparameter in LeakyRelu, 3) convolution layer with a convolution kernel of 3×3, and 4) average pooling layer; the structure of process 2 is: 1) average pooling layer, 2) convolution layer with a convolution kernel of 1×1; finally, the outputs of the two processes are added together to obtain the output of the initial residual module. The role of the encoding module in the expression network is to obtain features at different scales. It is mainly composed of a residual downsampling module and contains two parallel processes. The structure of process 1 is: 1) LeakyRelu (0.1) activation layer, 2) convolution layer with a convolution kernel of 3×3, 3) LeakyRelu (0.1) activation layer, 4) convolution layer with a convolution kernel of 3×3; the structure of process 2 is a convolution layer with a convolution kernel of 1×1, and finally the outputs of the two processes are added as the output of the encoding module. The structure of the encoding module is similar to the initial residual module, except that an average pooling layer is added after the convolution layer with a convolution kernel of 3×3 and the convolution layer with a convolution kernel of 1×1, and different encoding modules output feature maps of different sizes.

[0082] The role of the inference network is to infer the potential distribution of the input data, which is convenient for subsequent sampling, thereby obtaining a multivariate output result. In order to obtain a multivariate output, this embodiment samples the distribution based on the idea of ​​CVAE (conditional variational autoencoder), which involves the variational lower bound of the conditional log-likelihood of the supervised training instance. In this embodiment, the missing partial image (as a superset of the complete image) is also required to come from the latent space distribution, with a smooth prior probability p(Z c ), Z c is the potential vector. c, the latent vector of the missing part image with more pixels should have a larger potential prior variance than the latent vector of the missing part image with fewer pixels. Then the feature f output by the fourth encoding module in the expression network is expressed as m The sampled features output by the inference network are concatenated and used as the input of the generation network.

[0083] The decoding module in the generative network consists of a residual upsampling module and an output layer. The residual upsampling module structure contains two parallel processes. The structure of process 1 is: 1) normalization layer, 2) LeakyRelu (0.1) activation layer, 3) convolution layer with a convolution kernel of 3×3, 4) normalization layer, 5) LeakyRelu (0.1) activation layer, 6) deconvolution layer with a convolution kernel of 3×3; the structure of process 2 is: deconvolution layer with a convolution kernel of 3×3, and finally the outputs of the two processes are added as the output of the module; the output layer is composed of a LeakyReLU (0.1) activation layer, a convolution layer with a convolution kernel of 3×3, and a Tanh function activation layer.

[0084] In order to avoid excessive loss of information outside the mask area during the generation process, resulting in poor quality of the generated image, this embodiment, based on the original PIC network, splices (Cancat) the output of the residual upsampling module of the second decoding module in the generation network with the output of the first encoding module in the expression network to form a splicing graph, and uses the splicing graph as the input of the output layer of the second decoding module in the generation network; splices (Cancat) the output of the residual upsampling module of the third decoding module in the generation network with the output of the initial residual module in the expression network to form a splicing graph, and uses the splicing graph as the input of the output layer of the third decoding module in the generation network, thereby introducing more low-level features in the generation network, so that the generation network learns more non-mask area information during the training process, and improves the quality of the generated image.

[0085] Furthermore, the long-short-term attention module in the generation network is used to: calculate the short-term inter-layer attention features of the decoding features output by the first decoding module in the generation network; calculate the long-term inter-layer attention features of the encoding features output by the second encoding module in the expression network; aggregate the short-term inter-layer attention features and the long-term inter-layer attention features and use them as input to the residual upsampling unit of the second decoding module.

[0086] For example, the role of the long-term and short-term attention module in the generator network is to enable the network to choose to focus on smaller fine-grained features in the encoder or more semantic generative features in the decoder according to the specific situation. First, the decoded feature f of the output of the first decoding module in the generator network is calculated using the attention score d The short-term inter-layer attention features and output y d; In order to pay attention to the encoding feature f output by the second encoding module in the expression network e , this embodiment calculates the encoding feature f of the output of the second encoding module e The long-term inter-layer attention features and output y e , which is consistent with the decoded feature f that generates complete image information d Different, encoding feature f e Only the visible part I m ,Finally, the short-term and long-term attention features are aggregated and fed back to the next ,decoder layer.

[0087] Step S60: Based on the restored image after handwriting removal, the handwriting area on the image to be erased corresponding to the restored image after handwriting removal is covered to obtain a target image without handwriting.

[0088] Furthermore, after the step of performing overwriting processing on the handwriting area corresponding to the restored image after handwriting removal on the image to be erased based on the restored image after handwriting removal to obtain a target image without handwriting, the method further includes the following steps:

[0089] The target image without handwriting is binarized to obtain a binary image without handwriting.

[0090] Exemplarily, in an embodiment of the present application, according to the vertex coordinates of the handwriting area obtained by the target detection model, the restored image after removing the handwriting obtained after image restoration is overlaid on the corresponding position to obtain a complete image after erasing the handwriting, that is, a target image without handwriting; then scikit-image (an image processing package) is used to binarize the complete image after erasing the handwriting, and finally a binary image after erasing the handwriting is obtained.

[0091] It can be seen that the present application detects and locates the handwriting in the image to be erased, and cuts the image corresponding to the located detection frame to obtain multiple smaller cut images. The smaller images have fewer interference items, and the features of handwritten handwriting in the image will be more obvious. At the same time, semantic segmentation of three targets including background, printed text and handwritten text is performed on the smaller images, which effectively avoids mis-segmentation and missed segmentation, thereby improving the accuracy of document restoration.

[0092] See also Figure 3 As shown, the embodiment of the present application also provides a handwriting erasing device, comprising:

[0093] A handwriting detection unit, which is used to input the image to be erased into the target detection model, obtain multiple handwriting areas containing handwriting and vertex coordinates of each handwriting area; form a detection frame set based on the vertex coordinates of the multiple handwriting areas, and cut the image corresponding to each detection frame in the detection frame set to obtain multiple cut images;

[0094] A handwriting segmentation unit is used to perform semantic segmentation on the background, printed text and handwritten text in each cut picture based on the semantic segmentation model to obtain an initial three-dimensional matrix corresponding to each cut picture; update the pixel values ​​corresponding to the background and printed text in the initial three-dimensional matrix corresponding to each cut picture to a first preset value, and update the pixel values ​​corresponding to the handwritten text to a second preset value to obtain a binary handwriting mask corresponding to each cut picture;

[0095] An image restoration unit, which is used to input each binary handwriting mask and its corresponding cut picture into a multivariate image restoration model to obtain a restored image after handwriting removal;

[0096] The image stitching unit is used to cover the handwriting area corresponding to the restored image after handwriting removal on the image to be erased based on the restored image after handwriting removal to obtain a target image without handwriting.

[0097] The device of the present application is deployed as a cloud service, mainly by a microservice framework built by Docker (Docker is an open source application container engine) and based on Flask (Flask is a lightweight Web application framework written in Python) / Gunicorn (Gunicorn is a high-performance server); wherein the handwriting detection unit is used to create and train a target detection model, the handwriting segmentation unit is used to create and train a semantic segmentation model, and the image restoration unit is used to create and train a multivariate image restoration model.

[0098] The present application detects and locates the handwriting in the image to be erased through a handwriting detection unit, and cuts the image corresponding to the located detection frame to obtain multiple smaller cut images. The smaller images have fewer interference items, and the features of handwritten handwriting in the images are more obvious. At the same time, the handwriting segmentation unit performs semantic segmentation of three targets including background, printed text and handwritten text on the smaller image, and then the image restoration unit performs image restoration on the segmented results, which effectively avoids mis-segmentation and missed segmentation, thereby improving the accuracy of document restoration.

[0099] Furthermore, in the embodiment of the present application, the handwriting detection unit is specifically used for:

[0100] Forming a corresponding first detection frame according to the vertex coordinates of each handwriting area;

[0101] Detect whether the intersection-over-union ratio and the distance between two adjacent first detection frames are both greater than the corresponding merging threshold;

[0102] If so, merge the two adjacent first detection frames to obtain a second detection frame, and determine whether the width and height of the second detection frame are greater than the corresponding segmentation threshold. When the width of the second detection frame is greater than the width segmentation threshold and / or the height of the second detection frame is greater than the height segmentation threshold, segment the second detection frame to obtain multiple second sub-detection frames. When the width of the second detection frame is less than or equal to the width segmentation threshold and the height of the second detection frame is less than or equal to the height segmentation threshold, the second detection frame is not segmented.

[0103] If not, the two adjacent first detection frames are not merged, and it is determined whether the width and height of the first detection frame are greater than the corresponding segmentation threshold. When the width of the first detection frame is greater than the width segmentation threshold and / or the height of the first detection frame is greater than the height segmentation threshold, the first detection frame is segmented to obtain multiple first sub-detection frames. When the width of the first detection frame is less than or equal to the width segmentation threshold and the height of the first detection frame is less than or equal to the height segmentation threshold, the first detection frame is not segmented.

[0104] The unmerged and unsplit first detection frame, the first sub-detection frame, the unsplit second detection frame and the second sub-detection frame are taken as a detection frame set.

[0105] Furthermore, in the embodiment of the present application, the handwriting detection unit is also used for:

[0106] Non-maximum suppression processing is performed on all first detection frames to delete redundant first detection frames, the confidence threshold in the non-maximum suppression processing is set to 0.05, and the intersection-over-union ratio threshold in the non-maximum suppression processing is set to 0.95.

[0107] Furthermore, in the embodiment of the present application, the handwriting detection unit is also used for:

[0108] Merge multiple cut images into one batch data;

[0109] The batch data is input into the semantic segmentation model.

[0110] Furthermore, in an embodiment of the present application, the multivariate image restoration model includes an expression network and a generation network, and the input of the output layer of the second decoding module in the generation network is a splicing graph formed by the output of the residual upsampling unit of the second decoding module and the output of the first encoding module in the expression network; the input of the output layer of the third decoding module in the generation network is a splicing graph formed by the output of the residual upsampling unit of the third decoding module and the output of the initial residual module in the expression network.

[0111] Furthermore, in an embodiment of the present application, the long-short-term attention module in the generation network is used to:

[0112] Calculating short-term inter-layer attention features of the decoded features output by the first decoding module in the generator network;

[0113] Calculating long-term inter-layer attention features of the encoded features output by the second encoding module in the expression network;

[0114] The short-term inter-layer attention features and the long-term inter-layer attention features are aggregated and used as input of the residual upsampling unit of the second decoding module.

[0115] Furthermore, in the embodiment of the present application, the image stitching unit is also used to: perform binarization processing on the target image without handwriting to obtain a binary image without handwriting.

[0116] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each unit can refer to the corresponding process in the aforementioned handwriting erasing method embodiment, and will not be repeated here.

[0117] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 4 The handwriting eraser is shown running on the device.

[0118] An embodiment of the present application also provides a handwriting erasing device, comprising: a memory, a processor and a network interface connected via a system bus, wherein at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement all or part of the steps of the aforementioned handwriting erasing method.

[0119] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0120] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The processor is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0121] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a video playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as video data, image data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device or other volatile solid-state storage device.

[0122] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, all or part of the steps of the aforementioned handwriting erasing method are implemented.

[0123] The embodiment of the present application implements all or part of the aforementioned process, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of each of the above methods when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0124] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, servers or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program codes.

[0125] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0127] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0128] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A method for erasing handwriting, characterized in that: The following steps are involved: Inputting the image to be erased into the target detection model to obtain multiple handwriting areas containing handwriting and vertex coordinates of each handwriting area; A detection frame set is formed based on the vertex coordinates of multiple handwriting areas, and the image corresponding to each detection frame in the detection frame set is cut to obtain multiple cut images; Based on the semantic segmentation model, semantic segmentation is performed on the background, printed text and handwritten text in each cut image to obtain the initial three-dimensional matrix corresponding to each cut image; Respectively updating the pixel values ​​corresponding to the background and the printed text in the initial three-dimensional matrix corresponding to each cut image to the first preset value, and updating the pixel values ​​corresponding to the handwritten text to the second preset value, so as to obtain a binary handwriting mask corresponding to each cut image; Each binary handwriting mask and its corresponding cut image are input into the multivariate image restoration model to obtain the restored image after handwriting removal; Based on the restored image after handwriting removal, the handwriting area on the to-be-erased image corresponding to the restored image after handwriting removal is covered to obtain a target image without handwriting.

2. The handwriting erasing method according to claim 1, characterized in that: The forming of a detection frame set based on vertex coordinates of a plurality of handwriting regions comprises: Forming a corresponding first detection frame according to the vertex coordinates of each handwriting area; Detect whether the intersection-over-union ratio and the distance between two adjacent first detection frames are both greater than the corresponding merging threshold; If so, merge the two adjacent first detection frames to obtain a second detection frame, and determine whether the width and height of the second detection frame are greater than the corresponding segmentation threshold. When the width of the second detection frame is greater than the width segmentation threshold and / or the height of the second detection frame is greater than the height segmentation threshold, segment the second detection frame to obtain multiple second sub-detection frames. When the width of the second detection frame is less than or equal to the width segmentation threshold and the height of the second detection frame is less than or equal to the height segmentation threshold, the second detection frame is not segmented. If not, the two adjacent first detection frames are not merged, and it is determined whether the width and height of the first detection frame are greater than the corresponding segmentation threshold. When the width of the first detection frame is greater than the width segmentation threshold and / or the height of the first detection frame is greater than the height segmentation threshold, the first detection frame is segmented to obtain multiple first sub-detection frames. When the width of the first detection frame is less than or equal to the width segmentation threshold and the height of the first detection frame is less than or equal to the height segmentation threshold, the first detection frame is not segmented. The unmerged and unsplit first detection frame, the first sub-detection frame, the unsplit second detection frame and the second sub-detection frame are taken as a detection frame set.

3. The handwriting erasing method according to claim 2, characterized in that: Before the step of detecting whether the intersection-over-union ratio and the distance between two adjacent first detection frames are both greater than the corresponding merging threshold, the method further includes: Non-maximum suppression processing is performed on all first detection frames to delete redundant first detection frames, the confidence threshold in the non-maximum suppression processing is set to 0.05, and the intersection-over-union ratio threshold in the non-maximum suppression processing is set to 0.

95.

4. The handwriting erasing method according to claim 1, characterized in that: Before the step of performing semantic segmentation on the background, printed text and handwritten text in each cut picture based on the semantic segmentation model, the method further includes: Merge multiple cut images into one batch data; The batch data is input into the semantic segmentation model.

5. The handwriting erasing method according to claim 1, characterized in that: The multivariate image restoration model includes an expression network and a generation network, wherein the input of the output layer of the second decoding module in the generation network is a splicing graph formed by the output of the residual upsampling unit of the second decoding module and the output of the first encoding module in the expression network; the input of the output layer of the third decoding module in the generation network is a splicing graph formed by the output of the residual upsampling unit of the third decoding module and the output of the initial residual module in the expression network.

6. The handwriting erasing method according to claim 5, characterized in that: The long-short-term attention module in the generative network is used to: Calculating short-term inter-layer attention features of the decoded features output by the first decoding module in the generator network; Calculating long-term inter-layer attention features of the encoded features output by the second encoding module in the expression network; The short-term inter-layer attention features and the long-term inter-layer attention features are aggregated and used as input of the residual upsampling unit of the second decoding module.

7. The handwriting erasing method according to claim 1, characterized in that: After the step of performing overwriting processing on the image to be erased and the handwriting area corresponding to the restored image after handwriting removal based on the restored image after handwriting removal to obtain a target image without handwriting, the method further includes: The target image without handwriting is binarized to obtain a binary image without handwriting.

8. A handwriting erasing device, characterized in that: include: A handwriting detection unit, which is used to input the image to be erased into the target detection model to obtain a plurality of handwriting areas containing handwriting and the vertex coordinates of each handwriting area; A detection frame set is formed based on the vertex coordinates of multiple handwriting areas, and the image corresponding to each detection frame in the detection frame set is cut to obtain multiple cut images; A handwriting segmentation unit, which is used to perform semantic segmentation on the background, printed text and handwritten text in each cut picture based on a semantic segmentation model, and obtain an initial three-dimensional matrix corresponding to each cut picture; Respectively updating the pixel values ​​corresponding to the background and the printed text in the initial three-dimensional matrix corresponding to each cut image to the first preset value, and updating the pixel values ​​corresponding to the handwritten text to the second preset value, so as to obtain a binary handwriting mask corresponding to each cut image; An image restoration unit, which is used to input each binary handwriting mask and its corresponding cut picture into a multivariate image restoration model to obtain a restored image after handwriting removal; The image stitching unit is used to cover the handwriting area corresponding to the restored image after handwriting removal on the image to be erased based on the restored image after handwriting removal to obtain a target image without handwriting.

9. A handwriting erasing device, characterized in that: include: A memory and a processor, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the handwriting erasing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a processor, the handwriting erasing method according to any one of claims 1 to 7 is implemented.

Images