Method and apparatus for erasing handwritten content in a document image

Abstract

The application discloses a method for erasing handwritten content in a document image, comprising the following steps. Step S1: a first neural network for image segmentation is used to detect the image to be processed, and the handwritten content to be erased is extracted therefrom. Step S2: a third neural network for image restoration is used to fill the handwritten content to be erased in the image to be processed with a background color, and the filled image to be processed is restored to an unwritten state, referred to as a recovered image. Step S3: a fourth neural network for image sharpening is used to perform image optimization processing on the recovered image, including: removing traces left after erasing the handwritten content, removing handwriting from the back or below the paper, removing image noise, removing shadows, and removing objects unrelated to the document. The application has the advantages of fast processing speed, more natural erasing, and the ability to preserve the integrity of printed content when erasing handwritten content.

Description

Technical Field

[0001] This invention relates to a method for erasing handwritten content in document images and for image optimization. Background Technology

[0002] In educational settings, there's a common need to organize handwritten or annotated assignments and tests, often requiring the restoration of completed work for redoing. Similarly, in office settings, there's a need to restore filled-out documents and forms for reuse. Restoration in this context involves erasing handwritten content.

[0003] Chinese invention patent application CN114332150A, published on April 12, 2022, entitled "Method, Apparatus, Device and Readable Storage Medium for Erasing Handwritten Handwriting," discloses a method for erasing handwritten handwriting in document images. This method first detects and then segments the image. Its main drawbacks are slow speed and the erasing accuracy being affected by both detection and segmentation precision. The method does not specify how to handle overlapping areas of handwritten and printed text; if it is determined to be handwritten, the printed text will have broken strokes after erasing.

[0004] Chinese invention patent application CN114708601A, published on July 5, 2022, entitled "A Deep Learning-Based Method for Erasing Handwritten Characters," discloses a method for erasing handwritten characters in document images. This method does not specify how to handle overlapping areas between handwritten and printed content. If the text is determined to be handwritten, the printed text will appear broken after erasing. The method includes filling in pixels in the handwritten area to blend it with the background when creating training samples, which results in unnatural residual traces at the edges of the erased text after erasing. Summary of the Invention

[0005] The technical problem to be solved by this invention is to propose a method for erasing handwritten content and optimizing images in document images.

[0006] To address the aforementioned technical problems, this invention discloses an optimization method for erasing handwritten content in document images, comprising the following steps: Step S1: A first neural network for image segmentation is used to detect the image to be processed and extract the handwritten content to be erased. The first neural network is used to classify all pixels in the image to be processed into three categories: background pixels, printed pixels, and handwritten pixels. Pixels that are covered by both printed and handwritten content are designated as printed pixels. The set of all handwritten pixels in the image to be processed is the handwritten content to be erased. Step S2: A third neural network for image restoration is used to fill the handwritten content to be erased in the image to be processed with the background color. The image to be processed after filling is restored to its unwritten state, referred to as the restored image. Step S3: The restored image is processed using a fourth neural network for image sharpening, including: removing residual traces after erasing the handwritten content, removing text visible from the back or bottom of the paper, removing image noise, removing shadows, and removing objects unrelated to the document.

[0007] Optionally, step S1 can be replaced by step S1a. Step S1a: A second neural network for image segmentation is used to detect the image to be processed and extract the handwritten content to be erased from it; the second neural network is used to classify all pixels in the image to be processed into four categories: background pixels, printed pixels, handwritten pixels, and overlapping printed and handwritten pixels; the set of all handwritten pixels in the image to be processed is the handwritten content to be erased.

[0008] Preferably, the first neural network and the second neural network are U 2 The third neural network is any one of U-Net, MPRNet, and pix2pix; the fourth neural network is any one of LaMa, DeepFillv2, and HiFill; and the fifth neural network is any one of U-Net, U... 2 Any one of -Net, U-2-NETp, MPRNet, or pix2pix.

[0009] Furthermore, during training, the first and second neural networks incorporate boundary detection constraints, that is, the prediction results and annotations of the neural networks are fed into the Sobel filter, so that the texture edges of the handwritten content obtained by the neural network prediction results are consistent with the texture edges of the handwritten content in the annotation.

[0010] Furthermore, during training, each pair of training data for the fourth neural network consists of an original image and a processed image. The original image includes one or more defects, such as traces left after image processing, text visible on the back or underside of the paper, image noise, shadows, or objects unrelated to the document image. The processed image is obtained by processing the original image through three steps: (a) passing the original image through one or more image processing filters for image sharpening, text removal, noise reduction, and shadow removal; (b) manually removing remaining defects; if the content of the document image is damaged in step (a), it is also manually restored; and (c) converting it to a grayscale image.

[0011] Optionally, step S2 can be replaced by step S2a. Step S2a: An image restoration technique is used to fill the handwritten content to be erased in the image to be processed with the background color. The image to be processed after filling is restored to its unwritten state, which is called the restored image.

[0012] Optionally, the image to be processed is the input image, or a document region of the input image referred to as the document image, or a document image after orientation correction.

[0013] Optionally, step S4 is included after step S3. Step S4: The optimized document image is corrected for curvature based on text lines and / or table lines.

[0014] Furthermore, in step S4, a sixth neural network for image segmentation is first used to extract text lines and / or table lines from the optimized document image. Then, based on the original state and the state after flattening the extracted text lines and / or table lines into straight lines, a mapping matrix is ​​calculated on the entire optimized document image. Finally, the entire optimized document image is bent and flattened.

[0015] Optionally, step S5 is included after step S4. Step S5: The corrected document image is converted into an editable document using optical character recognition and layout restoration methods.

[0016] This invention also discloses an erasure optimization device for handwritten content in document images, including an erasure content detection unit, an erasure unit, and an image optimization unit. The erasure content detection unit uses a first neural network for image segmentation to detect the image to be processed and extract the handwritten content to be erased. The first neural network divides all pixels in the image to be processed into three categories: background pixels, printed pixels, and handwritten pixels. Pixels covered by both printed and handwritten content are designated as printed pixels. The set of all handwritten pixels in the image to be processed is the handwritten content to be erased. The first erasure unit uses a third neural network for image restoration to fill the handwritten content to be erased in the image to be processed with the background color. The filled image is restored to its unwritten state, referred to as the restored image. The image optimization unit uses a fourth neural network for image sharpening to perform image optimization processing on the restored image, including: removing residual traces after erasing the handwritten content, removing text visible from the back or bottom of the paper, removing image noise, removing shadows, and removing objects unrelated to the document.

[0017] Optionally, the erase content detection unit one can be replaced by the erase content detection unit two; the erase content detection unit two is used to detect the image to be processed using a second neural network for image segmentation, and extract the handwritten content to be erased from it; the second neural network is used to divide all pixels in the image to be processed into four categories: background pixels, printed pixels, handwritten pixels, and printed-handwritten overlapping pixels; the set of all handwritten pixels in the image to be processed is the handwritten content to be erased.

[0018] Optionally, the erasure unit one can be replaced by the erasure unit two; the erasure unit two is used to fill the handwritten content to be erased in the image to be processed with the background color using an image restoration technique, and the image to be processed after filling is restored to the unwritten state, which is called the restored image.

[0019] The technical effects achieved by this invention are: automatic removal of handwriting, document sharpening to improve clarity; fast processing speed; more natural erasure; and preservation of printed content integrity when erasing handwritten content. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the optimized method for erasing handwritten content in document images proposed in this invention.

[0021] Figure 2 This is a schematic diagram of the structure of the device for optimizing the erasure of handwritten content in document images proposed in this invention.

[0022] The attached figures are labeled as follows: 1 is the erased content detection unit 1, 2 is the erase unit 1, and 3 is the image optimization unit. Detailed Implementation

[0023] Please see Figure 1 The method for optimizing the erasure of handwritten content in document images proposed in this invention includes the following steps.

[0024] Step S1: A first neural network for image segmentation is used to detect the image to be processed and extract the handwritten content to be erased. Neural Network (NN) is short for Artificial Neural Network (ANN). The first neural network is preferably an encoder-decoder based neural network, such as U... 2 -Net, MPRNet, pix2pix, etc.

[0025] The first neural network categorizes all pixels in the image to be processed into three classes: background pixels, printed pixels, and handwritten pixels. Pixels that are neither covered by printed content nor by handwritten content are background pixels. Pixels covered only by printed content are printed pixels. Pixels covered only by handwritten content are handwritten pixels. Pixels covered by both printed and handwritten content are also considered printed pixels. The set of all handwritten pixels in the image to be processed constitutes the handwritten content to be erased. During training, the first neural network designates or labels pixels at the overlap between handwritten and printed content as printed pixels; thus, the handwritten content to be erased contains no printed content, effectively preserving the integrity of the printed content.

[0026] The training method for the first neural network is as follows: (1) Generate training data. Specifically, generate document images containing printed content and handwritten content, and specify whether each pixel is a background pixel, a printed pixel, or a handwritten pixel. Alternatively, collect some document images containing printed content and handwritten content, and manually label each pixel as a background pixel, a handwritten pixel, or a printed pixel. In particular, designate or label the pixels in the overlapping area of ​​printed content and handwritten content as printed pixels. In this way, the trained first neural network will only classify the pixels of handwritten content that are not covered by printed content as handwritten pixels, and classify the pixels of handwritten content that are covered by printed content as printed pixels. (2) Train the first neural network using the training data so that the output of the first neural network is as consistent as possible with the designated or labeled results. The trained first neural network can be used to classify all pixels in an image (picture) into three categories: background pixels, printed pixels, and handwritten pixels.

[0027] Preferably, the first neural network incorporates boundary detection constraints during training to improve the accuracy of the three pixel classifications. Boundary detection constraints refer to the constraints imposed by the texture edges of the input image (locations where pixel grayscale values ​​change drastically, such as text outlines, image outlines, etc.) on the corresponding texture edges of the output image. The constraint aims to make the texture edges of the input image and the output image tend to be the same. At this time, step (2) of the training method of the first neural network is changed to (2a): the first neural network is trained using the training data, and the prediction results and annotations of the first neural network are fed into the Sobel filter. The Sobel filter is used to calculate the texture edges of the image, so that the texture edges of the handwritten content obtained by the prediction results of the first neural network tend to be consistent with the texture edges of the annotated handwritten content. Finally, the trained first neural network can more accurately classify all pixels in the image into three categories: background pixels, printed pixels, and handwritten pixels.

[0028] Step S2: A third neural network for image inpainting is used to fill the handwritten content to be erased in the image with the background color, effectively erasing the handwritten content. The filled image is restored to its unwritten state, and is called the restored image. The third neural network can be, for example, LaMa, DeepFillv2, or HiFill. Preferably, a neural network like LaMa containing convolution operations is used, but the convolution operations are replaced with Fast Fourier Convolution (FFC) operations. Fast Fourier Convolution is a channel-wise convolution based on the Fast Fourier Transform (FFT), which expands the receptive field of the neural network, improves the inpainting effect, and makes the filling effect more realistic and natural.

[0029] The training method for the third neural network is as follows: (1) Generate training data. Specifically, a document image without handwritten content and a document image with added handwritten content are used as a pair of training data. Multiple pairs of training data are generated in the same way. (2) The training data is used to train the third neural network so that the output of the third neural network is as consistent as possible with the document image without handwritten content. After the trained third neural network repairs the handwritten content in the document image (filling with the background color), it is restored to the state without handwritten content.

[0030] Step S3: The restored image is optimized using a fourth neural network for image sharpening. The third neural network used for image restoration often leaves noise after erasing handwritten content. This step uses the fourth neural network for image sharpening to remove image noise and other defects. Image sharpening refers to compensating for the image's contours, enhancing edges and areas of grayscale abrupt changes, making a blurred image clearer. Image optimization includes: removing traces left after erasing handwritten content (e.g., spots), removing text that shows through from the back or bottom of the paper (referred to as "transparent text"), removing image noise, removing shadows caused by lighting, and removing objects unrelated to the document (e.g., fingers). This step removes various defects in the restored image while ensuring clear printed content and strong contrast, avoiding noise and transparent text from affecting the subsequent bending correction effect, and improving the accuracy of subsequent bending correction. The fourth neural network is, for example, U-Net, U... 2 -Net, U-2-NETp, MPRNet, pix2pix, etc.

[0031] The training method of the fourth neural network is as follows: (1) Generate training data. Each pair of training data consists of an original image and a processed image. The original image includes one or more defects, such as spots or marks left after image processing, transparent text on the back or bottom of the paper, image noise, shadows caused by light or other reasons, and objects unrelated to the document image, such as fingers. The processed image is obtained by processing the original image through three steps, which are: (a) passing the original image through one or more existing image processing filters for image sharpening, removing transparent text, removing noise, and removing shadows; (b) removing remaining defects by manual processing; if the text, pictures, tables, formulas, etc. in the document image are damaged in step (a), they are also restored by manual processing; existing filters are not good enough for document images in complex scenes, the main problems are transparent text residue, noise residue, finger residue, etc., this step is to clean it by manual smearing; (c) converting to grayscale image; this step is to eliminate shadow residue and images that have not been cleaned by manual smearing by converting to grayscale image. The above three steps are indispensable. They are used to create cleaner processed images as training data, and on the other hand, they also reduce the learning difficulty of the fourth neural network. (2) The training data is used to train the fourth neural network so that the output of the fourth neural network is as consistent as possible with the processed image. The trained fourth neural network can be used to remove various defects in the image, that is, to achieve image optimization, making the image clearer, easier to read, noise-free, text-free, and free of background color interference.

[0032] Optionally, step S1 can be replaced by step S1a. Step S1a involves: using a second neural network for image segmentation to detect the image to be processed and extracting the handwritten content to be erased. The second neural network is preferably a codec-based neural network, such as a U... 2 -Net, MPRNet, pix2pix, etc.

[0033] The second neural network categorizes all pixels in the image to be processed into four classes: background pixels, printed pixels, handwritten pixels, and overlapping printed / handwritten pixels. Pixels that are neither covered by printed content nor by handwritten content are background pixels. Pixels covered only by printed content are printed pixels. Pixels covered only by handwritten content are handwritten pixels. Pixels covered by both printed and handwritten content are overlapping printed / handwritten pixels. The set of all handwritten pixels in the image to be processed constitutes the handwritten content to be erased. This ensures that the handwritten content to be erased contains no printed content, thus preserving the integrity of the printed content.

[0034] The training method for the second neural network is as follows: (1) Generate training data. Specifically, generate document images containing printed and handwritten content, where each pixel is specified as a background pixel, a printed pixel, a handwritten pixel, or a printed / handwritten overlapping pixel. Alternatively, collect some document images containing printed and handwritten content, and manually label each pixel as a background pixel, a printed pixel, a handwritten pixel, or a printed / handwritten overlapping pixel. (2) Train the second neural network using the training data, so that the output of the second neural network is as consistent as possible with the specified or labeled results. The trained second neural network can be used to classify all pixels in the image into four categories: background pixels, printed pixels, handwritten pixels, and printed / handwritten overlapping pixels.

[0035] Preferably, the second neural network incorporates boundary detection constraints during training to improve the accuracy of the four pixel classifications. This is the same as the first neural network's use of boundary detection constraints during training, and will not be elaborated further.

[0036] Optionally, step S2 can be replaced by step S2a. Step S2a involves using an image inpainting technique to fill the handwritten content to be erased in the image to be processed with the background color, i.e., erasing the handwritten content from the image to be processed. The filled image is then restored to its unwritten state, referred to as the restored image. The image inpainting technique could be, for example, the cv2.inpaint() function in OpenCV. OpenCV (Open Source Computer Vision Library) is an open-source library containing hundreds of computer vision algorithms.

[0037] The image to be processed is the input image.

[0038] Alternatively, the image to be processed is a document region of the input image, referred to as a document image. For example, a fifth neural network for image segmentation can be used to extract the document region from the input image. This operation reduces the interference of non-document regions in subsequent handwritten pixel judgment, improving the accuracy of subsequent handwritten pixel judgment. The fifth neural network could be, for example, U-2-NETp, MPRNet, pix2pix, etc.

[0039] The training method for the fifth neural network is as follows: (1) Generate training data. Generate images containing document images using existing document images, where the positions of the document image regions are specified, i.e., known. Alternatively, collect some images containing document images, where the positions of the document image regions in each image are manually labeled, i.e., known. (2) Train the fifth neural network using the training data, so that the output of the fifth neural network is as consistent as possible with the specified or labeled results. The trained fifth neural network can be used to extract document image regions (i.e., document regions) from images.

[0040] Preferably, after the fifth neural network detects and locates the document region in the input image, it crops and extracts the document region through perspective transformation. The document region after perspective transformation is a rectangle; that is, the ROI (region of interest) commonly seen in image processing.

[0041] Alternatively, the image to be processed is a document image after orientation correction. For example, the orientation of the document image is corrected based on the text, and / or images, and / or table content in the document image; or the orientation of the document image is corrected according to the method disclosed in Chinese invention patent application CN114267046A, published on April 1, 2022, entitled "A Method and Apparatus for Orientation Correction of Document Images".

[0042] Optionally, step S4 is included after step S3. Step S4 involves: performing curvature correction on the optimized document image based on text lines and / or table lines to improve regularity and aesthetics. For example, the method disclosed in Chinese invention patent application CN111353961A, published on June 30, 2020, entitled "A Method and Apparatus for Document Surface Correction," can be used to flatten the document image. Alternatively, a sixth neural network for image segmentation can be used to extract text lines and / or table lines from the optimized document image. Then, based on the original state and the flattened state of the extracted text lines and / or table lines, a mapping matrix is ​​calculated on the entire optimized document image, and then the entire optimized document image is flattened.

[0043] The training method for the sixth neural network is as follows: (1) Generate training data. Generate images containing text lines and / or table lines, wherein which pixels belong to the text lines and / or table lines is specified and known. Alternatively, collect some images containing text lines and / or table lines, wherein which pixels in each image belong to the text lines and / or table lines is manually labeled and known. (2) Train the sixth neural network using the training data so that the output of the sixth neural network tends to be consistent with the specified or labeled results. The trained sixth neural network can be used to extract text lines and / or table lines from images.

[0044] Optionally, step S5 is included after step S4. Step S5 involves converting the corrected document image into an editable document (e.g., Microsoft Word format) using optical character recognition (OCR) and layout restoration methods. For example, the method disclosed in Chinese invention patent application CN113850068A, published on December 28, 2021, entitled "A Method and Apparatus for Converting Images into Editable Text While Maintaining Layout," can be used.

[0045] Please see Figure 2 The document image erasure optimization device proposed in this invention includes an erasure content detection unit 1, an erasure unit 2, and an image optimization unit 3. Figure 2 The device shown is Figure 1 The method shown corresponds to this.

[0046] The erased content detection unit 1 is used to detect the image to be processed using a first neural network for image segmentation, and to extract the handwritten content to be erased. The first neural network is used to classify all pixels in the image to be processed into three categories: background pixels, printed pixels, and handwritten pixels. Pixels that are covered by both printed and handwritten content are designated as printed pixels. The set of all handwritten pixels in the image to be processed is the handwritten content to be erased.

[0047] The erasure unit 2 is used to fill the handwritten content to be erased in the image to be processed with the background color using a third neural network for image restoration. The image to be processed after filling is restored to the unwritten state, which is called the restored image.

[0048] The image optimization unit 3 is used to perform image optimization processing on the restored image using a fourth neural network for image sharpening, including: removing traces left after erasing handwritten content (such as spots), removing text that seeps through from the back or bottom of the paper (referred to as transparent text), removing image noise, removing shadows caused by light and other reasons, and removing objects in the image that are not related to the document (such as fingers).

[0049] Optionally, the erased content detection unit 1 can be replaced by an erased content detection unit 2. The erased content detection unit 2 uses a second neural network for image segmentation to detect the image to be processed and extract the handwritten content to be erased. The second neural network categorizes all pixels in the image to be processed into four types: background pixels, printed pixels, handwritten pixels, and overlapping printed / handwritten pixels. The set of all handwritten pixels in the image to be processed constitutes the handwritten content to be erased.

[0050] Optionally, the erasure unit 2 can be replaced by an erasure unit 2. The erasure unit 2 is used to fill the handwritten content to be erased in the image to be processed with the background color using an image restoration technique. The image to be processed after filling is restored to the unwritten state, which is called the restored image.

[0051] This invention addresses the problem of handwritten content restoration and quality improvement in educational and office settings, proposing a solution for handwritten content erasure and image quality enhancement. This invention improves the restoration efficiency of documents and exam papers containing handwritten content, achieving automatic handwriting removal and document sharpening to improve clarity. Compared to CN114332150A, this invention has the advantage of faster processing speed because it distinguishes handwritten content using only one neural network (either a first or second neural network). Compared to CN114708601A, this invention uses a third neural network for image restoration to erase handwritten content, resulting in more natural erasure without affecting the original printed area. Furthermore, this invention treats the overlapping area of ​​handwritten and printed content as printed content, preserving the integrity of the printed content while erasing the handwritten content—a feat not achieved in the aforementioned two documents.

[0052] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An optimized method for erasing handwritten content in document images, characterized in that, Includes the following steps; Step S1: A first neural network for image segmentation is used to detect the image to be processed and extract the handwritten content to be erased from it; the first neural network is used to classify all pixels in the image to be processed into three categories: background pixels, printed pixels, and handwritten pixels; Pixels that are covered by both printed and handwritten content are called printed pixels; The set of all handwritten pixels in the image to be processed is the handwritten content to be erased; the first neural network combines boundary detection constraint means during training, that is, the prediction result and annotation of the first neural network are fed into the Sobel filter, so that the texture edge of the handwritten content obtained by the prediction result of the first neural network is consistent with the texture edge of the annotated handwritten content. Step S2: A third neural network for image restoration is used to fill the handwritten content to be erased in the image to be processed with the background color. The image to be processed after filling is restored to the unwritten state, which is called the restored image. Step S3: The restored image is processed using a fourth neural network for image sharpening, including: removing traces left after erasing handwritten content, removing text visible through the back or bottom of the paper, removing image noise, removing shadows, and removing objects unrelated to the document; during training, each pair of training data for the fourth neural network consists of an original image and a processed image; the original image includes one or more of the following defects: traces left after image processing, text visible through the back or bottom of the paper, image noise, shadows, or objects unrelated to the document image; the processed image is obtained by processing the original image through three steps, namely: (a) passing the original image through one or more image processing filters for image sharpening, text removal, noise removal, and shadow removal; (b) manually removing remaining defects; if the content in the document image is damaged in step (a), it is also manually restored; (c) converting to a grayscale image.

2. The method for optimizing the erasure of handwritten content in a document image according to claim 1, characterized in that, Step S1 is replaced by step S1a; Step S1a: A second neural network for image segmentation is used to detect the image to be processed and extract the handwritten content to be erased from it; the second neural network is used to classify all pixels in the image to be processed into four categories: background pixels, printed pixels, handwritten pixels, and overlapping printed and handwritten pixels; The set of all handwritten pixels in the image to be processed is the handwritten content to be erased; During training, the second neural network incorporates boundary detection constraints, which means that the prediction results and annotations of the second neural network are fed into the Sobel filter, so that the texture edges of the handwritten content obtained by the prediction results of the second neural network are consistent with the texture edges of the handwritten content in the annotation.

3. The method for optimizing the erasure of handwritten content in a document image according to claim 1 or 2, characterized in that, The first neural network and the second neural network are U 2 The third neural network is any one of U-Net, MPRNet, and pix2pix; the fourth neural network is any one of LaMa, DeepFillv2, and HiFill; and the fifth neural network is any one of U-Net, U... 2 Any one of -Net, U-2-NETp, MPRNet, or pix2pix.

4. The method for optimizing the erasure of handwritten content in a document image according to claim 1 or 2, characterized in that, Step S2 is replaced by step S2a; Step S2a: An image restoration technique is used to fill the handwritten content to be erased in the image to be processed with the background color. The image to be processed after filling is restored to the unwritten state, which is called the restored image.

5. The method for optimizing the erasure of handwritten content in a document image according to claim 1 or 2, characterized in that, The image to be processed is the input image, or a document region of the input image called a document image, or a document image after orientation correction.

6. The method for optimizing the erasure of handwritten content in a document image according to claim 1 or 2, characterized in that, in Step S3 is followed by step S4; Step S4: Correct the curvature of the optimized document image based on text lines and / or table lines.

7. The method for optimizing the erasure of handwritten content in a document image according to claim 6, characterized in that, In step S4, a sixth neural network for image segmentation is first used to extract text lines and / or table lines from the optimized document image. Then, based on the original state and the state after flattening the extracted text lines and / or table lines into straight lines, a mapping matrix is ​​calculated on the entire optimized document image. Finally, the entire optimized document image is bent and flattened.

8. The method for optimizing the erasure of handwritten content in a document image according to claim 6, characterized in that, Step S4 is followed by step S5; Step S5: Convert the corrected document image into an editable document using optical character recognition and layout restoration methods.

9. An optimized device for erasing handwritten content in a document image, characterized in that, It includes a content detection unit, an erasure unit, and an image optimization unit; The erased content detection unit is used to detect the image to be processed using a first neural network for image segmentation, and extract the handwritten content to be erased from it. The first neural network is used to classify all pixels in the image to be processed into three categories: background pixels, printed pixels, and handwritten pixels; Pixels that are covered by both printed and handwritten content are called printed pixels; The set of all handwritten pixels in the image to be processed is the handwritten content to be erased; the first neural network combines boundary detection constraint means during training, that is, the prediction result and annotation of the first neural network are fed into the Sobel filter, so that the texture edge of the handwritten content obtained by the prediction result of the first neural network is consistent with the texture edge of the annotated handwritten content. The erasing unit is used to fill the handwritten content to be erased in the image to be processed with the background color using a third neural network for image restoration. The image to be processed after filling is restored to the unwritten state, which is called the restored image. The image optimization unit is used to perform image optimization processing on the restored image using a fourth neural network for image sharpening, including: removing traces left after erasing handwritten content, removing text visible through the back or bottom of the paper, removing image noise, removing shadows, and removing objects unrelated to the document; during training, each pair of training data for the fourth neural network consists of an original image and a processed image; the original image includes one or more of the following defects: traces left after image processing, text visible through the back or bottom of the paper, image noise, shadows, or objects unrelated to the document image; the processed image is obtained by processing the original image through three steps, namely: (a) passing the original image through one or more image processing filters for image sharpening, text removal, noise removal, and shadow removal; (b) manually removing remaining defects; if the content in the document image is damaged in step (a), it is also manually restored; (c) converting it to a grayscale image.

10. The device for erasing and optimizing handwritten content in a document image according to claim 9, characterized in that, The erased content detection unit one is replaced by the erased content detection unit two; the erased content detection unit two is used to detect the image to be processed using a second neural network for image segmentation, and extract the handwritten content to be erased from it; The second neural network is used to classify all pixels in the image to be processed into four categories: background pixels, printed pixels, handwritten pixels, and overlapping printed and handwritten pixels; The set of all handwritten pixels in the image to be processed is the handwritten content to be erased; During training, the second neural network incorporates boundary detection constraints, which means that the prediction results and annotations of the second neural network are fed into the Sobel filter, so that the texture edges of the handwritten content obtained by the prediction results of the second neural network are consistent with the texture edges of the handwritten content in the annotation.

11. The device for erasing and optimizing handwritten content in a document image according to claim 9, characterized in that, The erasure unit one is replaced by the erasure unit two; the erasure unit two is used to fill the handwritten content to be erased in the image to be processed with the background color using an image restoration technology. The image to be processed after filling is restored to the unwritten state, which is called the restored image.

Images