Method for removing smudges in a handwritten document image and related products

Abstract

The invention name of the present disclosure is "Method for removing smudge in handwritten document image and related products". The present disclosure provides a method for removing smudge in handwritten document image, comprising detecting the boundary box of the smudge using a deep learning detection model, and removing the smudge based on the handwriting within the boundary box and the relationship between the boundary box. The present disclosure also provides an apparatus, a device and a machine readable storage medium corresponding to the method. By using the method, apparatus, device and machine readable storage medium provided by the present disclosure, the accuracy of smudge removal can be improved at least.

Description

TECHNICAL FIELD

[0001] The non-limiting and exemplary embodiments of the present disclosure generally relate to image processing, and in particular to methods, apparatuses, devices and machine-readable storage media for removing smudges in handwritten document images. BACKGROUND

[0002] Handwriting is the most common method for humans to express knowledge and record information. Automatic detection of the content of handwritten document images is an important research topic in the field of digital document analysis. Although this problem is very challenging due to the diversity of human writing, significant progress has been made in detection effectiveness in recent years with the development and evolution of deep learning detection models based on, for example, Convolutional Neural Networks (CNN).

[0003] Currently, most detection models assume that the handwriting on the image is normal when detecting the handwriting on a handwritten document image, i.e. there are no smudges in the handwriting.

[0004] However, during the process of handwriting, people often make mistakes, such as spelling errors, inappropriate wording in context, incorrect numbers or symbols, etc. In addition, the author’s thoughts and opinions may develop and change during the writing process, causing previously written content to become less valuable or even incorrect. The author will usually smudge these errors or unwanted content and write the corrected content next to it. These smudges appear in various handwritten documents, such as letters, manuscripts and examination papers, and would be misleading if detected as normal characters.

[0005] Current solutions exist for detecting and removing erasures in handwritten document images. For example, BB Chaudhuri and C Adak in their paper “An approach for detecting and cleaning of struck-out handwritten text” (Pattern Recognition, vol. 61, pp. 282-294, January 2017, doi: https: / / doi.org / 10.1016 / j.patcog.2016.07.032) propose using a mixture model and support vector machine SVM and radial basis function RBF classifiers to distinguish between erasures and normal text, and then to remove the detected erasures. Patent application CN110400309A discloses an erasure removal method, which includes determining erasure trace information in a target image using, for example, a neural network, and performing a cleaning process or a covering process on the determined erasure trace. Patent application CN112052852A describes a method of removing erasure characters, in which two data sets of high and low quality are prepared to solve the problem of erasure character detection, two convolutional recurrent neural network CRNN models with recognition differences are trained, the same detection results are taken as output values, and the differences between the data sets are used to remove erasure characters. SUMMARY

[0006] Current erasure detection and removal solutions have the technical problems of low detection accuracy and inaccurate removal, where low detection accuracy can result in detecting part of normal handwriting as erasure handwriting and / or detecting erasure handwriting as normal handwriting, and inaccurate removal can result in removing normal handwriting near the erasure handwriting during the removal of the erasure handwriting.

[0007] One purpose of the present disclosure is to eliminate or alleviate the technical problem of inaccurate removal described above. To achieve this purpose, the inventors of the present disclosure have conceived that, when removing erasures detected by a deep learning detection model, the relationship between the handwriting within the bounding box of the erasures and the bounding box is considered to avoid removing normal handwriting (if any) within the bounding box.

[0008] According to a first aspect of the present disclosure, the above purpose is achieved by a method for removing erasures in a handwritten document image. The method comprises: detecting a bounding box of the erasures using a deep learning detection model; and removing the erasures based on the relationship between the handwriting within the bounding box and the bounding box.

[0009] According to a second aspect of the present disclosure, the above object is achieved by a device for removing a smear in a handwritten document image. The device comprises a detecting component configured to detect a bounding box of the smear using a deep learning detection model; and a removing component configured to remove the smear based on a relationship between the handwriting within the bounding box and the bounding box.

[0010] According to a third aspect of the present disclosure, the above object is achieved by an apparatus for removing a smear in a handwritten document image. The apparatus comprises a processor and a memory storing instructions which, when executed by the processor, cause the apparatus to perform the method according to the first aspect.

[0011] According to a fourth aspect of the present disclosure, the above object is achieved by a machine-readable storage medium. The machine-readable storage medium stores instructions which, when executed on an apparatus, cause the apparatus to perform the method according to the first aspect.

[0012] Another object of the present disclosure is to eliminate or alleviate the above technical problem of low detection accuracy. To achieve this object, the inventors of the present disclosure actually compared a plurality of detection models and finally determined that the Swin-Transformer+CascadeMask-RCNN detection model has the best effect and is used as the deep learning detection model. Therefore, in some embodiments, the Swin-Transformer+CascadeMask-RCNN detection model is used as the deep learning detection model for detecting smears, thereby improving the accuracy of smear detection.

[0013] Another object of the present disclosure is to provide sample images for training the deep learning detection model. To achieve this object, the inventors of the present disclosure thought of splicing a plurality of sub-images from a handwritten document image to form the sample images. Therefore, in some embodiments, the sample images for training the deep learning detection model are formed by selecting a plurality of sub-images from a handwritten document image and splicing the plurality of sub-images on a blank image, thereby effectively augmenting the learning data samples for training of the detection model. BRIEF DESCRIPTION OF DRAWINGS

[0014] The above and other aspects, features, and benefits of the present disclosure will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings. In the drawings, like reference numerals or letters are used to refer to like or equivalent elements throughout. The drawings are shown for the purpose of better understanding the embodiments of the present disclosure and are not necessarily drawn to scale, wherein:

[0015] Figure 1 An example framework of R-CNN is shown;

[0016] Figure 2Fig. 1 shows a partial answer sheet example image of a certain math exam;

[0017] Figure 3 Fig. 1 shows a partial answer sheet example image of a certain math exam;

[0018] Figure 4 Fig. 1 shows a partial answer sheet example image of a certain math exam; Figure 2 Fig. 1 shows a partial answer sheet example image of a certain math exam;

[0019] Figure 5 Fig. 1 shows a partial answer sheet example image of a certain math exam;

[0020] Figure 6 Fig. 1 shows a partial answer sheet example image of a certain math exam;

[0021] Figure 7 Fig. 1 shows a partial answer sheet example image of a certain math exam;

[0022] Figure 8 Fig. 1 shows a partial answer sheet example image of a certain math exam;

[0023] Figure 9 Fig. 1 shows a partial answer sheet example image of a certain math exam;

[0024] Figure 10 Fig. 1 shows a partial answer sheet example image of a certain math exam;

[0025] Figure 11 Fig. 1 shows a partial answer sheet example image of a certain math exam; DETAILED DESCRIPTION

[0026] Embodiments herein will be described more fully hereinafter with reference to the accompanying drawings. Embodiments herein may, however, be implemented in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided as non-limiting examples, and the scope of the claims should not be limited by the embodiments set forth in the description.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises", "comprising", "includes" and / or "including", when used herein, specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0028] Furthermore, the use of ordinal numbers such as “first,” “second,” and “third” to modify objects in this article does not imply any priority, precedence, or order of one object relative to another, nor does it imply the chronological order in which the actions of methods are executed. Rather, it serves only as a label to distinguish one object with a specific name from another object with the same name.

[0029] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as generally understood. It will be further understood that terms used herein should be interpreted as having the same meaning as they have in the context of this specification and the relevant technical field, and should not be construed in a restrictive sense unless expressly defined herein.

[0030] Handwriting is the most common method humans use to express knowledge and record information. Automatic content detection in handwritten document images is an important research topic in the field of digital document analysis. Although this problem is very challenging due to the diversity of human handwriting, significant progress has been made in recent years with the development and evolution of deep learning detection models, such as those based on Convolutional Neural Networks (CNNs).

[0031] For example, Region-based Convolutional Neural Network (R-CNN) is a bottom-up detection model that applies a region recommendation strategy to a convolutional neural network. An example framework of R-CNN is... Figure 1 The diagram is shown in the image, with the technologies involved on the left and the main detection stages on the right.

[0032] like Figure 1 As shown, in the R-CNN detection model, the first stage is region proposal, that is, given an image, a selective search algorithm generates and recommends 1000 to 2000 candidate boxes (representing candidate regions) with different shapes and sizes, which can overlap or contain each other; the second stage is feature extraction, that is, using a convolutional neural network (CNN) to extract features for each candidate box; the third stage is classification, that is, using a linear support vector machine (SVM) to classify the features extracted in the previous stage; the fourth stage is overlap removal, that is, applying a non-maximum suppression algorithm to overlapping candidate boxes and selecting the bounding boxes with higher SVM scores.

[0033] However, the distortion problem of R-CNN after candidate region scaling and the repeated calculation when extracting features lead to the bottleneck of model performance and speed. In order to solve these problems, Fast R-CNN adds a network layer spatial pyramid pooling SPP (Spatial Pyramid Pooling) layer at the junction of the convolutional layer and the fully connected layer of R-CNN, so that the input dimension of the fully connected layer can be fixed and the distortion of the candidate region can be avoided, and the problem of repeated calculation of R-CNN is solved.

[0034] The first step of Fast R-CNN processing is to perform multiple convolution kernel pooling processing on the image to obtain a convolution feature map. Each feature vector is fed into a fully connected layer, which branches into two sibling output layers. One of the output layers functions to classify and outputs classification information (softmax probability estimation) for each candidate region about K object classes; the other layer outputs four real-valued positioning information for each candidate region, representing the center horizontal and vertical coordinates and the length and width of the bounding box of the region.

[0035] However, Fast R-CNN still uses the traditional candidate region recommendation algorithm (such as selective search), which does not meet the real-time requirements. In order to solve the bottleneck of candidate region recommendation, Faster R-CNN template detection model is further proposed.

[0036] Faster R-CNN mainly includes two modules: one module is the region proposal network RPN (Region Proposal Network), which is a kind of fully convolutional network FCN (Fully-Convolutional Network), mainly used for generating candidate regions, that is, estimating the position and size of the target object; the other module is the Fast R-CNN detection model, which uses the candidate regions generated by RPN for classification and bounding box regression calculation, and shares the convolutional layer with RPN. Faster R-CNN can be simply understood as the combination of RPN and Fast R-CNN, which replaces the selective search to recommend candidate regions, and speeds up the running speed.

[0037] On the basis of Faster R-CNN, Mask R-CNN is developed, which adds another branch in Faster R-CNN for outputting classification information and positioning information for each candidate region, thereby adding output mask information for each candidate region to represent the mask of the object in the region, that is, the original two tasks (classification + regression) become three tasks (classification + regression + segmentation). Mask R-CNN combines the binary mask with the classification and bounding box from Faster R-CNN, which produces accurate image segmentation.

[0038] Cascade R-CNN uses a cascade of detection models on the basis of Faster R-CNN, and uses an incremental threshold in each stage, so that there is a sufficient number of candidate regions in each stage, and there is no serious overfitting phenomenon.

[0039] Cascade Mask R-CNN is a combination of Mask R-CNN and Cascade R-CNN, and the current stage receives the candidate regions recommended by this stage and the information obtained by regression in the last stage as input, to calculate new output information, so as to further improve the detection effect.

[0040] In recent years, ViT (Vision Transformer) has been increasingly widely concerned and applied to replace CNN for image feature extraction. ViT is a Transformer-based image processing algorithm, and Transformer is mainly used for natural language processing NLP (Natural Language Processing). It can better extract data features through the self-attention mechanism to weight and aggregate each data element in the input sequence.

[0041] The main idea of ViT for using Transformer to process image classification problems is to map the input image pixel blocks (patches) into a sequence of vectors, and then use these vectors as the input sequence of Transformer. In order to capture the local features in the image, ViT maps each pixel block into a fixed-length vector representation through the block embedding operation before inputting the image.

[0042] In addition to block embedding, ViT also introduces two important modules: multi-head self-attention and feed-forward network. Multi-head self-attention allows different features to be captured at different positions and scales, while the feed-forward network is used to process and adjust the features.

[0043] To overcome the computational complexity of ViT, the Swin Transformer was proposed for image feature extraction, featuring a sliding window operation and a hierarchical design. The sliding window operation includes non-overlapping local windows and overlapping cross windows. Restricting attention computation to a single window introduces the localization of CNN convolution operations while saving computational resources. The algorithm still uses the patch concept from ViT. However, unlike ViT which uses a fixed window (16×16 pixels), the Swin Transformer initially uses small windows in the first Transformer layer, subsequently merging these into larger patches in deeper layers. It segments the image into 4×4 pixel patches, each with 3 channels, for a total of 48 feature dimensions, which are linearly transformed into C-dimensional vectors (96 or 192). Unlike ViT, the Swin Transformer computes self-attention only within local windows, reducing algorithm complexity and enabling intensive recognition tasks. The output is then merged through a merging layer, concatenating the vectors of 2x2 adjacent patch groups in the image. The attention window moves relative to the previous layer at each step. For example, in the first layer, attention is limited to the neighborhood of these regions; in the next layer, these regions move (similar to strided convolution). Patches that fall into different windows and cannot communicate in the first layer can communicate in the second layer. These generated patches are then merged by a merging layer. This process is repeated depending on the number of layers selected.

[0044] Of course, besides the deep learning detection models mentioned above, there are many other detection models. However, most detection models assume that the handwriting in the image is normal, i.e., there is no smearing in the handwriting, when detecting handwriting on handwritten document images.

[0045] However, errors often occur during handwriting, such as spelling mistakes, inappropriate wording, incorrect numbers or symbols, etc. Furthermore, an author's thoughts and perspectives may evolve and change during the writing process, rendering previously written content less valuable or even erroneous. Authors typically erase these errors or unnecessary content and write corrections alongside them. These erasures appear in various handwritten documents, such as letters, manuscripts, and exam papers, and can create errors once detected as normal characters. For example, due to the complexity and abstract nature of mathematics, erasures are more likely to occur when solving mathematical problems. Erased text, numbers, and symbols can significantly affect the meaning and interpretation of formulas, severely interfering with formula detection. Figure 2 The image shows a sample answer sheet from a math exam. It can be seen that scribbling occurs frequently and in many different ways, without any uniform characteristics.

[0046] It can be seen that removing the erasures in the handwritten document image helps to accurately detect the meaningful document content on the image. Of course, the erasures must be detected before being removed, and the deep learning detection model described above can also be used to detect the erasures after being trained for the purpose.

[0047] There are currently solutions for detecting and removing erasures in handwritten document images. For example, BB Chaudhuri and C Adak proposed in their paper “An approach for detecting and cleaning of struck-out handwritten text” (Pattern Recognition, vol. 61, pp. 282-294, January 2017, doi: https: / / doi.org / 10.1016 / j.patcog.2016.07.032) to use a hybrid model and support vector machine SVM and radial basis function RBF classifier to distinguish erasures and normal text, and then to remove the detected erasures. Patent application CN110400309A discloses an erasure removal method, which includes determining erasure trace information in a target image using, for example, a neural network, and performing cleaning or covering processing on the determined erasure trace. Patent application CN112052852A describes a method of removing erasure characters, in which two data sets of high and low quality are prepared to solve the problem of erasure character detection, two convolutional recurrent neural network CRNN (Convolutional Recurrent Neural Network) models with recognition differences are trained, the same detection results are used as output values, and the differences between the data sets are used to remove erasure characters.

[0048] The current erasure detection and removal solutions have the technical problems of low detection accuracy and inaccurate removal, where the low detection accuracy can cause some normal handwriting to be detected as erasure handwriting and / or erasure handwriting to be detected as normal handwriting, and the inaccurate removal can cause normal handwriting near the erasure handwriting to be removed together in the process of removing the erasure handwriting.

[0049] To eliminate or alleviate the above technical problems of inaccurate removal and low detection accuracy, the inventors of the present disclosure thought that when removing the erasures within the erasure bounding box detected by the deep learning detection model, the relationship between the handwriting within the bounding box and the bounding box should be considered to avoid removing the normal handwriting (if any) within the bounding box.

[0050] Figure 3is a flowchart of a method 300 for removing smudges in a handwritten document image according to embodiments of the present disclosure. The method comprises the following steps: a step 301 of detecting a bounding box of the smudge using a deep learning detection model; and a step 302 of removing the smudge based on the handwriting within the bounding box and the relationship between the bounding box. Figure 4 In Figure 2 The bounding boxes of smudges obtained after performing step 301 of method 300 are further illustrated on the partial answer sheet example image. As shown in Figure 4 , each smudge has been tightly enclosed by a rectangular bounding box, and the handwriting outside the bounding box is normal handwriting.

[0051] Next, method 300 and its various embodiments will be further described in conjunction with Figures 5 to 9 It can be understood that, although Figures 7-9 examples are shown in relation to mathematical answer sheet images, the handwritten document images to which the schemes of the present disclosure are applicable are not limited to mathematical answer sheet images, but can be any meaningful handwritten document images. The schemes of the present disclosure are not limited in this respect. In addition, although Figures 7-9 the bounding boxes of smudges shown in the examples are rectangular boxes, the bounding boxes that can be used by the schemes of the present disclosure can also be other shapes of boxes suitable for tightly enclosing smudges, such as circular boxes, triangular boxes, irregularly shaped boxes, etc. The schemes of the present disclosure are not limited in this respect.

[0052] In order to find the deep learning detection model with the best detection effect at present, the inventors of the present disclosure used a variety of mainstream deep learning detection models to detect smudges. Figure 5 The average precision AP of smudge detection by each deep learning detection model is shown, in which it can be seen that the detection model that can obtain the highest average precision is the Swin-Transformer+CascadeMask-RCNN detection model, i.e., the Cascade Mask-RCNN detection model using Swin Transformer for feature extraction. Therefore, in one embodiment, the Swin-Transformer+Cascade Mask-RCNN detection model is used as the deep learning detection model in method 300. However, other deep learning detection models can also be used in method 300. The schemes of the present disclosure are not limited in this respect.

[0053] An example architecture 600 of the Swin-Transformer+Cascade Mask-RCNN detection model is shown in Figure 6is shown. In this example architecture 600, the Region Proposal Network (RPN) 602 receives a handwritten document image 601 as input and outputs multiple candidate boxes 603 (i.e., candidate bounding boxes for smudges) representing multiple region proposals; these candidate boxes 603 are then subjected to feature extraction by the Swin-Transformer layer 604; the features 605 extracted at the current stage and the bounding boxes 607 regressed from the previous stage are input into the Bounding Box Regression and Candidate Box Refinement section 606 to obtain the adjusted candidate boxes 607 at the current stage until a predefined accuracy is reached; the refined candidate boxes 608 finally output by the Bounding Box Regression and Candidate Box Refinement section 606 are post-processed by non-maximum suppression 609 to remove redundant candidate boxes, and finally the bounding boxes for smudges 610 are output as the detection result.

[0054] It should be noted that the above example architecture 600 only uses the localization information output by the Swin-Transformer + Cascade Mask-RCNN detection model to detect the bounding boxes for smudges. As described above, Cascade Mask-RCNN can be regarded as a cascaded Mask R-CNN detection model, where Mask R-CNN can output classification information, localization information, and mask information for each proposed region. The inventors of the present disclosure have recognized through experiments that the method 300 according to the embodiments of the present disclosure can use the Swin-Transformer + Cascade Mask-RCNN detection model to detect the bounding boxes for smudges without using the classification information and mask information, making the calculation of the model simpler and not reducing the accuracy of the detection. Therefore, in one embodiment, the classification information and mask information output by the Swin-Transformer + Cascade Mask-RCNN detection model are discarded. However, those skilled in the art can appreciate that according to the specific types of handwritten documents and specific application requirements targeted by the present disclosure solution, the classification information and / or mask information can also be used. For example, the classification information can be used to, for example, specifically classify the shape of the smudge, and the mask information can be used to, for example, intercept the smudged handwriting on the image, etc. The solution of the present disclosure has no limitation in this regard.

[0055] After detecting the bounding boxes for smudges, an operation to remove the smudges within the bounding boxes can be performed. Generally speaking, the bounding boxes do not contain non-smudged handwriting (i.e., normal handwriting), but there may be detection deviations for individual bounding boxes in the above detection steps, such that some non-smudged handwriting is included in the bounding boxes. Figure 7 is an example image where the bounding box includes some normal handwriting, and the top handwriting of the text "From the figure" is within this bounding box.

[0056] The inventors of the present disclosure have observed that if all the strokes in a bounding box are removed directly, the normal strokes (if any) within the bounding box will be harmed, leading to recognition errors of the relevant strokes. Therefore, the inventors of the present disclosure have conceived to remove the erasures based on the relationship between the strokes within the bounding box and the bounding box, thereby avoiding removing the normal strokes (if any) within the bounding box. Such relationship can be one or more of, for example, whether the strokes are in contact with the bounding box, whether the strokes are close to the edges of the bounding box, whether the strokes are close to the center of the bounding box, whether the strokes extend out of the bounding box, and the like. Those skilled in the art can base the removal of erasures on various appropriate relationships between the strokes and the bounding box according to the specific type of handwritten document and the specific application requirements that the present disclosure scheme is directed to. The present disclosure scheme is not limited in this respect.

[0057] For example, for handwritten documents like math answer sheets, the inventors of the present disclosure have observed that most of the erasures have continuous long erasure curves, and all fall in the central position of the bounding box, while the strokes within the bounding box that are isolated and in contact with the bounding box are all normal strokes. Therefore, in one embodiment, removing the erasures based on the relationship between the strokes within the bounding box and the bounding box comprises: preserving the isolated strokes within the bounding box that are in contact with the bounding box, and removing other strokes within the bounding box.

[0058] In further embodiments, the method 300 uses a connected component based heuristic algorithm to identify the isolated strokes. For example, the strokes within the bounding box can be binarized, and then a connected component algorithm can be employed to cluster the strokes.

[0059] In further embodiments, the heuristic algorithm divides the strokes within the bounding box into a first connected component, a second connected component, and a third connected component, and identifies the strokes in the third connected component as the isolated strokes, wherein the first connected component is not in contact with the bounding box, the second connected component is in contact with the bounding box and has a center close to the center of the bounding box, and the third connected component is in contact with the bounding box and has a center far away from the center of the bounding box.

[0060] Figure 8 Example before and after removing erasures within a bounding box are shown. From Figure 8 It can be seen that although the bounding box therein contains part of the normal strokes (see the left half of Figure 8 ), due to the consideration of the relationship between the strokes and the bounding box when removing, the method 300 only removes the erasures in the bounding box, preserving the normal strokes that extend into the bounding box (see the right half of Figure 8 ), thereby preserving the valuable content that is not erased. In particular, Figure 8 the isolated strokes in contact with the bounding box of Isolated handwriting that is reserved, not in contact with the bounding box Continuous handwriting that is removed, in contact with the bounding box Also removed.

[0061] The inventors of the present disclosure also realized that in order to train various kinds of deep learning detection models, it is common to augment the sample data used for training by cropping, scaling, padding, lightening, contrast, etc. on the original data, but these augmentations have limited effect on the text and cannot effectively generate effective sample data that is significantly different from the original data. Therefore, in one embodiment, for the deep learning detection model used by method 300, the inventors of the present disclosure propose to augment the training data of the above deep learning detection model by selecting a plurality of subgraphs from the handwritten document image and splicing the plurality of subgraphs on a blank image to form a sample image for training the deep learning detection model.

[0062] In further embodiments, each of the plurality of subgraphs is selected by randomly selecting a rectangular region within the handwritten document image. For example, for the handwritten document image, a coordinate is randomly selected as the center, and a length and a width are randomly selected. The four variables of the center x-coordinate and y-coordinate and the length and width are used to locate a region on the image as a subgraph.

[0063] In further embodiments, the aspect ratio of the rectangular region ranges from 0.5 to 10.

[0064] In one embodiment, the edges of the selected subgraph do not pass through the handwriting and the bounding box on the handwritten document image, thereby ensuring that there is no part of the handwriting within the selected subgraph, and ensuring that the erased bounding box will be completely within the selected subgraph or completely outside the selected subgraph. If the edges of the selected subgraph pass through the handwriting and / or the bounding box on the handwritten document image, the subgraph can be discarded and the selection can be re-performed.

[0065] In one embodiment, splicing the plurality of subgraphs on the blank image includes, for each subgraph of the plurality of subgraphs, performing the following operations: deforming the subgraph; and pasting the deformed subgraph at a different random position on the blank image until the pasting does not result in overlapping handwriting. That is, pasting a deformed subgraph on the blank image, if the pasting results in overlapping handwriting of the subgraph and the subgraphs already pasted on the blank image, re-pasting the subgraph at a different random position on the blank image, otherwise accepting the pasting, which ensures that no new erasing-like handwriting is generated in the sample image due to handwriting overlap.

[0066] In further embodiments, the deformation refers to random scaling, stretching, and Gaussian blur.

[0067] In a further embodiment, the warping further comprises a random -5° to +5° rotation for subgraphs not containing the bounding box.

[0068] Figure 9 An example process of forming sample images according to embodiments of the present disclosure is schematically shown, which forms sample images with large difference from original images, and can be used to effectively train a deep learning detection model, so that the learning data samples can be effectively augmented for the training of the detection model. Using the sample image data augmented by these embodiments to train the above-mentioned Swin-Transformer+Cascade Mask-RCNN detection model can further improve the detection accuracy thereof (see Figure 5 of the last line).

[0069] In one embodiment, the handwritten document image is an image of a handwritten math answer sheet. However, the scheme of the present disclosure is applicable to any meaningful handwritten document image. The scheme of the present disclosure is not limited in this respect.

[0070] Figure 10 is a schematic block diagram of an apparatus 1000 according to embodiments of the present disclosure. The apparatus is used for removing smudges in a handwritten document image, and comprises: a detection component 1001 for detecting a bounding box of the smudges using a deep learning detection model; and a removal component 1002 for removing the smudges based on the relationship between the handwriting within the bounding box and the bounding box. Various embodiments of the apparatus 1000 correspond to various embodiments of the above-mentioned method, and therefore are not described here again.

[0071] It can be appreciated that the apparatus 1000 described herein can be realized by various components, so that the apparatus 1000 realizing one or more functions described in the above-mentioned embodiments can not only include the components shown in Figure 10 but also include other components for realizing one or more other functions thereof. In addition, the apparatus 1000 can include a single component configured to perform two or more functions, or separate components for respective corresponding functions. In addition, these components can be realized in hardware, firmware, software, or any combination thereof.

[0072] It should be understood that the blocks of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, and / or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer and / or other programmable data processing apparatus, create means for implementing the functions / acts specified in the block diagrams and / or flowchart diagrams.

[0073] It should also be understood that the functions / acts shown in the blocks of the flow diagrams can occur out of the order depicted. For example, two blocks shown in succession can in fact be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality / acts involved. Although some diagrams can include arrows on communication paths to show the primary direction of communication, it is to be understood that communication can occur in the opposite direction to the arrows.

[0074] Furthermore, the schemes of the present disclosure can take the form of a computer program on a storage medium having a computer-usable or computer-readable program code embodied in the medium that is employed to program the instruction execution system to perform particular tasks or implement particular abstract data types and to invoke the elements coupled to the system or in communication with the system. In the context of this document, a storage medium can be any media that can contain, store, or maintain the program for use by or in connection with the instruction execution system, apparatus, or device.

[0075] Accordingly, the present disclosure also provides a device 1100 including a processor 1101 and a memory 1102 for removing smudges in a handwritten document image, a schematic block diagram of which is shown in Figure 11 In the device 1100, the memory 1102 stores instructions which, when executed by the processor 1101, cause the device 1100 to perform the above-described method for removing smudges in a handwritten document image.

[0076] The present disclosure also provides a machine-readable storage medium (not shown) having stored thereon instructions which, when executed on a processor, cause the processor to perform the above-described method for removing smudges in a handwritten document image.

[0077] While this specification contains many specific implementation details, these should not be construed as limitations on the scope or potential requirements of any implementations, but as descriptions of particular embodiments falling under the generic precepts of the disclosure. Certain features that are, for clarity, described above in the context of separate embodiments can also be provided in combinations unless otherwise specifically stated and / or required by the technical context. Conversely, various features that are, for brevity, described in the context of a single embodiment can also be provided separately or in any suitable subcombination. In addition, while features can be described above as functioning in the context of certain combinations, even if claimed separately, an individual feature can function in the context of one or more different combinations than the ones described. All such modifications and variations are within the skills of the artisans and can be practiced and implemented in various of the appropriate environment by skilled people without unreasonably expanding the scope or spirit of the system and techniques described above, and claimed below. Further, it is the usual practice to use terms such as "including", "containing", "comprising", "having", and / or "encompassing" to convey structural binaries of a claimed subject matter. In this specification, these terms are used in their open-ended, conventional sense to mean that when used in the context of a structural binary, the term "comprising" indicates the presence of the stated structural binary, but not excluding the presence of one or more other structural binaries.

[0078] It will be apparent to those skilled in the art that the application concept can be implemented in various ways without departing from the spirit or scope of the disclosure. The above examples are given for the purpose of description and not limitation, and it is understood that modifications and variations can be made without departing from the spirit and scope of the disclosure, as would be apparent to those of ordinary skill in the art. Such modifications and variations are intended to fall within the scope of the disclosure and the appended claims. The scope of the disclosure is defined by the appended claims.

Claims

1. A method for removing smudges from a handwritten document image, comprising: A deep learning detection model is used to detect the bounding box of the smear; as well as The smearing is removed based on the relationship between the handwriting within the bounding box and the bounding box. This removal includes: retaining isolated handwriting within the bounding box that contacts the bounding box, and removing other handwriting within the bounding box. A heuristic algorithm based on connected components is used to classify the handwriting within the bounding box into a third connected component, and the handwriting in the third connected component is identified as the isolated handwriting. The third connected component contacts the bounding box and its center is far from the center of the bounding box.

2. The method as described in claim 1, wherein the deep learning detection model is the Swin-Transformer+Cascade Mask-RCNN detection model.

3. The method as described in claim 2, wherein the classification information and mask information in the Swin-Transformer+Cascade Mask-RCNN detection model are discarded.

4. The method of claim 1, wherein the heuristic algorithm further divides the handwriting within the bounding box into a first connected component, a second connected component, and a third connected component, wherein the first connected component does not touch the bounding box, and the second connected component touches the bounding box and its center is close to the center of the bounding box.

5. The method of any one of claims 1-4, wherein the sample images used to train the deep learning detection model are generated by the following operations: Selecting multiple sub-images from the handwritten document image; and The sample image is formed by stitching the multiple sub-images onto a blank image.

6. The method of claim 5, wherein each of the plurality of sub-images is selected by randomly selecting a rectangular region within the handwritten document image.

7. The method of claim 6, wherein the aspect ratio of the rectangular region is between 0.5 and 10.

8. The method of claim 5, wherein the edges of the selected sub-image do not cross the handwriting on the handwritten document image and the bounding box.

9. The method of claim 5, wherein stitching the plurality of sub-images onto the blank image comprises performing the following operations on each of the plurality of sub-images: Transform the subgraph; and Paste the deformed sub-image at different random locations on the blank image until the pasting does not result in overlapping handwriting.

10. The method of claim 9, wherein the deformation refers to random scaling, stretching, and Gaussian blurring.

11. The method of claim 10, wherein the deformation further includes a random rotation of -5° to +5° for subgraphs that do not contain the bounding box.

12. The method of claim 1, wherein the handwritten document image is an image of a handwritten math answer sheet.

13. An apparatus for removing smudges from a handwritten document image, comprising: A detection component for using a deep learning detection model to detect the bounding box of the smear; as well as A removal component is used to remove the smear based on the relationship between the handwriting within the bounding box and the bounding box. Removing the smear based on the relationship between the handwriting within the bounding box and the bounding box includes: retaining isolated handwriting within the bounding box that contacts the bounding box, and removing other handwriting within the bounding box. This is achieved by using a connected component-based heuristic algorithm to classify the handwriting within the bounding box into a third connected component, identifying the handwriting in the third connected component as the isolated handwriting, and ensuring that the third connected component contacts the bounding box and its center is away from the center of the bounding box.

14. A device for removing smudges from a handwritten document image, comprising: processor; as well as A memory that stores instructions, wherein the instructions, when executed by the processor, cause the device to perform the method as described in any one of claims 1-12.

15. A machine-readable storage medium containing stored instructions that, when executed on a device, cause the device to perform the method as claimed in any one of claims 1-12.

Images