Handwriting erasing method and device, equipment and medium

By performing label and mask processing of handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwritten handwrit remove.

CN119065569AInactive Publication Date: 2024-12-03SHENZHEN CHUANGZHI MINIMALIST TECHNOLOGY CO LTD

Patent Information

Application Number
CN202411565793.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-05
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing handwritten erasing methods have poor results in complex backgrounds, resulting in less accuracy in handwritten erasing.

Method used

By obtaining the pending image, processing it into a label image and a mask image, and inputting it into a handwritten letter erase model based on the LAMA network and the decoder network for iterative training until the trained model meets the expected performance, and the model is to be processed using the model Handwritten traces in the image are erased.

Benefits of technology

It significantly improves the accuracy and efficiency of handwritten writing erasing, enhances its resistance to complex backgrounds, and improves its ability to distinguish handwritten and printed words.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065569A_ABST
    Figure CN119065569A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of deep learning, in particular to a handwritten handwriting erasing method and device, equipment and a medium, and aims to solve the problem that in the prior art, the accuracy of handwritten handwriting erasing cannot be improved, and consequently the influence of a complex background on the erasing effect is large. The method comprises the steps that a to-be-processed image is acquired, and the to-be-processed image is a real image containing printed handwriting and handwritten handwriting; processing the to-be-processed image to obtain a label image and a mask image; inputting the label image and the mask image into a handwritten handwriting erasing model based on an LAMA network and a decoder network, carrying out iterative training until the trained handwritten handwriting erasing model meets expected performance, and obtaining a target handwritten handwriting erasing model; and erasing the handwritten handwriting in the image to be processed by using the target handwritten handwriting erasing model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a handwriting erasing method, device, equipment and medium. Background Art

[0002] With the continuous development of image analysis technology, we can convert various documents into images through mobile terminals and then further process them. Among them, the removal of handwriting in documents has a very wide range of applications in education, office and other fields.

[0003] At present, most practices are to apply scene text erasure technology to solve the problem of handwriting erasure. However, research has found that for complex scenes, the erasure effect of existing handwriting erasure methods is poor, and it is difficult to cleanly erase handwriting traces on paper, which leads to low accuracy of handwriting erasure. Therefore, how to improve the accuracy of handwriting erasure to reduce the impact of complex backgrounds on the erasure effect has become a technical problem that needs to be solved urgently by technicians in this field. Summary of the invention

[0004] Based on this, it is necessary to provide a handwriting erasing method, device, equipment and medium to address the above technical problems, so as to solve the problem that the existing technology cannot improve the accuracy of handwriting erasing, resulting in a greater impact of complex background on the erasing effect.

[0005] A first aspect of an embodiment of the present application provides a method for erasing handwritten traces, the method comprising: Acquire an image to be processed, wherein the image to be processed is a real image containing printed words and handwritten words; Processing the image to be processed to obtain a label image and a mask image; Inputting the label image and the mask image into a handwriting erasure model based on a LAMA network and a decoder network, and performing iterative training until the trained handwriting erasure model meets the expected performance, thereby obtaining a target handwriting erasure model; The target handwriting erasure model is used to erase the handwriting in the image to be processed.

[0006] A second aspect of an embodiment of the present application provides a handwriting erasing device, the handwriting erasing device comprising: An acquisition module, used for acquiring an image to be processed, wherein the image to be processed is a real image containing printed handwriting and handwritten handwriting; A processing module, used for processing the image to be processed to obtain a label image and a mask image; A training module, used for inputting the label image and the mask image into a handwriting erasure model based on a LAMA network and a decoder network, and performing iterative training until the trained handwriting erasure model meets the expected performance, thereby obtaining a target handwriting erasure model; The erasing module is used to erase the handwriting in the image to be processed by using the target handwriting erasure model.

[0007] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the handwriting erasing method as described in the first aspect when executing the computer program.

[0008] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the handwriting erasing method as described in the first aspect is implemented.

[0009] In summary, the present invention provides a method, device, equipment and medium for erasing handwriting, by acquiring an image to be processed, wherein the image to be processed is a real image containing printed handwriting and handwriting, processing the image to be processed to obtain a label image and a mask image, inputting the label image and the mask image into a handwriting erasure model based on a LAMA network and a decoder network, and performing iterative training until the trained handwriting erasure model meets the expected performance, and obtaining a target handwriting erasure model, and using the target handwriting erasure model to erase the handwriting in the image to be processed. Compared with the prior art, the present application obtains a target handwriting erasure model by inputting the generated label image and mask image into a handwriting erasure model based on a LAMA network and a decoder network for iterative training, which ultimately greatly improves the accuracy and efficiency of handwriting erasure, and improves the ability to distinguish handwritten characters from printed characters, thereby reducing the influence of complex background on the erasure effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0011] Figure 1 This is a schematic diagram of an application environment of a handwriting erasing method provided by an embodiment of the present invention; Figure 2It is a flowchart of a method for erasing handwriting provided by an embodiment of the present invention; Figure 3 It is a structural schematic diagram of a handwriting erasing device provided by one embodiment of the present invention; Figure 4 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0014] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0015] As used in the present specification and the appended claims, the term “if” may be interpreted as “when” or “uponce” or “in response to determining”, depending on the context. Similarly, the phrases “if it is determined” or “if matched to [described condition or event]” may be interpreted as meaning “upon determination” or “in response to determination” or “uponce matched to [described condition or event]” or “in response to matching to [described condition or event]”, depending on the context.

[0016] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0017] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0018] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0019] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0020] A method for erasing handwritten characters provided by an embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment, it is designed to solve common defects in images, such as handwriting and complex backgrounds, so as to improve the accuracy of handwriting erasure to produce clearer, more detailed and more beautiful images. Among them, the scanning device communicates with the server through the network. The scanning device is used to obtain the image to be processed and send the image to be processed to the server. The server can use the image to be processed as a training sample image and input the trained target handwriting erasure model, thereby improving the accuracy of handwriting erasure. Among them, the scanning device can be but not limited to various cameras, sensors, laser scanners, scanning measuring instruments, etc. The server can be implemented with an industrial computer, an independent server, or a server cluster consisting of multiple servers.

[0021] See also Figure 2 , is a flow chart of a method for erasing handwritten characters provided by an embodiment of the present invention. The handwritten characters erasing method can be applied to Figure 1 Servers in Figure 2 As shown, the handwriting erasing method may include the following steps: S201: Acquire an image to be processed, wherein the image to be processed is a real image containing printed words and handwritten words.

[0022] In the embodiments of the present disclosure, the image to be processed may be any image that needs to be erased by handwriting, such as an image of a test paper written by a student (such as a test paper that has been answered), or an image of an office document with handwriting (such as a document with a handwritten signature). Printed handwriting may include printed traces such as text, lines, and graphics formed by printing, and handwritten handwriting may include handwritten handwriting such as text, lines, and graphics formed by handwriting. The image to be processed may be obtained by photographing teaching materials (such as books, test papers, and exercise books, etc.) through a camera of a smartphone, a camera of a tablet computer, a lens of a digital camera, etc., or may include an image obtained by scanning teaching materials (such as books, test papers, and exercise books, etc.) through an instrument with a scanning function such as a printer. It should be noted that although the embodiments of the present disclosure all take the image to be processed as an image of a test paper as an example, it should not be regarded as a limitation of the present disclosure.

[0023] Specifically, the image to be processed contains real images containing both printed and handwritten characters, that is, handwritten images and printed images, wherein the handwritten images may also include handwritten text images and handwritten special symbol images. In actual application, the image to be processed may contain only printed images, or only handwritten images, or only printed images and handwritten text images. The specific composition form of the image to be processed is not limited here. It should be noted that although the embodiments of the present disclosure all take the image to be processed as an image containing a printed image and a handwritten image as an example, it should not be regarded as a limitation of the present disclosure. Through the above steps, the present application obtains the image to be processed so as to perform more precise erasing on the image to be processed later.

[0024] S202: Process the image to be processed to obtain a label image and a mask image.

[0025] In the disclosed embodiments, a label image is an image that separates different objects or regions. It can be a grayscale image, where each object is represented by a different grayscale value, or a color image, but usually requires a corresponding color map. Mask images often use binary images or multi-channel images to represent specific objects therein, usually with white (255) representing the target area and black (0) representing the background area. In semantic segmentation, each pixel value is used to represent a different category. This application can better understand and parse the image content and identify objects and their boundaries by processing the image to be processed and then generating a label image and a mask image.

[0026] As an optional embodiment, processing the image to be processed to obtain a label image includes: The handwritten characters in the image to be processed are removed and filled pixel by pixel by using image editing software to obtain a label image.

[0027] Specifically, after obtaining the image to be processed, the handwriting in the image to be processed is removed and filled pixel by pixel to obtain a label image. It should be noted that the image editing software may be Photoshop or PS software, etc., and this application does not make any limitation on this. Through detailed pixel-by-pixel processing, the handwriting can be effectively removed and a corresponding label image can be generated, which is convenient for subsequent application and analysis.

[0028] As an optional embodiment, interpolation processing is performed on the high-quality image to generate a mask image, including: Subtracting the image to be processed from the label image to obtain a difference map; Averaging the values ​​of the three channels of the difference map to obtain a difference grayscale map; Comparing the predicted score of each pixel in the difference grayscale image with a preset threshold, wherein the predicted score is used to indicate the probability that the corresponding pixel is handwritten; The pixel values ​​of the pixels whose prediction scores are greater than the preset threshold are set to 1, and the pixel values ​​of the pixels whose prediction scores are not greater than the preset threshold are set to 0, so as to obtain a mask image, wherein the mask image is a binary mask image.

[0029] Specifically, the pixel difference between the image to be processed and the label image is calculated to obtain a difference map. The values ​​of the three channels of the difference map are averaged to obtain a difference grayscale map, and the predicted score of each pixel in the difference grayscale map is compared with a preset threshold, wherein the predicted score is used to indicate the probability that the corresponding pixel is handwritten, and the pixel value of the pixel whose predicted score is greater than the preset threshold is set to 1, and the pixel value of the pixel whose predicted score is not greater than the preset threshold is set to 0, so as to obtain a mask image, wherein the mask image is a binary mask image. Thus, the difference grayscale map can be binarized to obtain a mask image containing only 0 and 1, which provides data support for subsequent handwriting removal. It should be noted that the preset threshold can be 20 or other values, which can be set according to actual conditions, and this application does not impose any restrictions on this.

[0030] In the implementation of this application, by processing the image to be processed, a label image and a mask image are obtained. This can accurately represent the boundaries and categories of each object in the image, improve the accuracy and efficiency of image processing and analysis, and provide important support for the application of machine learning and deep learning, making the overall process more efficient.

[0031] S203: Input the label image and the mask image into a handwriting erasure model based on a LAMA network and a decoder network, and perform iterative training until the trained handwriting erasure model meets the expected performance, thereby obtaining a target handwriting erasure model.

[0032] In an embodiment of the present application, the LAMA network (Large Multilingual Adapter Model) is a technology based on a large-scale pre-trained language model. It improves the performance of the model in specific tasks or fields by adding a small number of adaptive layers (adapter layers) without training the entire model from scratch. LAMA is usually used in a multilingual environment and can be quickly "fine-tuned" to new tasks, thereby saving computing resources and improving the effect of transfer learning. The decoder network is a common deep learning architecture, mainly used for sequence generation tasks such as machine translation, text summarization, speech recognition or chatbots. In the encoder-decoder model structure, the decoder is responsible for receiving the hidden state information from the encoder and converting it into a probability distribution of the target sequence. The handwriting erasure model of the present application is composed of an image generation module and a mask generation module. The decoder and encoder of the LAMA network are introduced as the image generation module, wherein the decoder obtains feature maps of different scales and channels through three downsamplings, image features representing different receptive fields extracted, and further extracts features through a residual connection block constructed by FFC (the number of connection blocks is 9). Finally, the encoder gradually generates a picture of the erased handwriting through three upsamplings. At the same time, an additional decoder is introduced as a mask generation module, specifically defining a transposed convolution module, performing three convolution upsamplings, and finally generating a mask through a Sigmoid function, wherein the Sigmoid function is a common S-type function in biology, also known as an S-type growth curve, which is usually used as an activation function of a neural network to map variables between 0 and 1. Each module of the handwriting erasure model adopts an encoder-decoder structure, wherein the mask generation module shares parameters with the encoder of the image generation module, and the mask generation module adopts an attention mechanism to generate a spatial domain attention matrix using a mask feature map to guide the generation of the target map.

[0033] In an embodiment of the present application, a handwriting erasure model based on a LAMA network and a decoder network includes a plurality of convolutional neural network structures, each of which has different receptive fields in order from large to small. Among them, the receptive field (Receptive Field) is an area in the input space that affects a specific unit of the network. The input area can be the input of the network or the output of other units in the network, which is not limited here. It should be noted that compared with the erasure method in the prior art, the handwriting erasure model can better capture the handwriting of the image to be processed, and control the size of the image by gradually shrinking the receptive field, thereby improving the accuracy and efficiency of handwriting erasure.

[0034] As an optional embodiment, the handwriting erasure model is trained in the following manner: Obtain training sample data set; Inputting the sample images in the training sample data set into the handwriting erasure model for training to obtain a predicted label image and a predicted mask image; Calculate the reconstruction loss value of the label image and the predicted label image in the training sample data set by using the reconstruction loss function; Calculating the segmentation loss value of the mask image and the predicted mask image in the training sample data set by using a mask loss function; Based on the reconstruction loss value and the segmentation loss value, the handwriting erasure model is trained until the sum of the reconstruction loss value and the segmentation loss value output by the trained handwriting erasure model is less than a preset loss threshold.

[0035] Specifically, it can be obtained from images publicly available on the Internet or collected offline. Take several sample images containing printed handwriting and handwriting, the sample images obtained include but are not limited to educational scene documents, such as test papers, and process the obtained sample images to obtain sample handwriting images and sample printed images in the sample images, thereby obtaining a training sample data set. Among them, the sample handwriting image includes the handwriting of the sample image to which it belongs, but does not include printed handwriting, and the sample printed image includes the printed handwriting of the sample image to which it belongs, but does not include handwriting. That is to say, in the embodiment of the present disclosure, the training sample data set can be composed of multiple sample subsets, and a sample subset includes a sample image, a sample handwriting image in the sample image, and a sample printed image in the sample image. It should be noted that in the embodiment of the present disclosure, the sizes of the sample image, the sample handwriting image, and the sample printed image are the same to facilitate subsequent processing.

[0036] Since the sizes of different sample images vary, in order to facilitate model training, the collected The sample images of different sizes are resized to a uniform size, such as scaling the sample images to a size of 512*512. In addition, the pixel values ​​of each pixel in the sample image can be normalized, and the pixel value range of 0 to 255 in the sample image can be normalized to between 0 and 1. The preprocessed sample images are then used to construct a training sample data set for model training.

[0037] In the disclosed embodiment, after obtaining the training sample data set, the sample images in the training sample data set are input into the handwriting erasure model for training, and the handwriting area and the printed area in the sample image are classified, detected and identified by the handwriting erasure model to obtain the predicted label image and the predicted mask image, and then the reconstruction loss value of the label image and the predicted label image in the training sample data set is calculated by the reconstruction loss function, and the segmentation loss value of the mask image and the predicted mask image in the training sample data set is calculated by the mask loss function, wherein the loss is calculated by the following formula:

[0038] in, Expressed as the total loss function value, Expressed as the reconstruction loss value, Expressed as the segmentation loss value, represents the label image output by the image generation module, represents the mask image output by the mask generation module, Represented as the predicted mask map of the handwriting erasure model, Represented as the predicted label map of the handwriting erasure model, Expressed as a matrix Hadamard product, This means that the background weight is 0.5.

[0039] After determining the total loss function value based on the reconstruction loss value and the segmentation loss value, the present application then updates the network parameters of the handwriting erasure model according to the total loss function value for iterative training until the total loss function value of the handwriting erasure model to be trained is less than or equal to the preset loss threshold, so as to obtain the target handwriting erasure model. It should be noted that the preset loss threshold can be set according to the experience of the technician, and this embodiment does not limit the value of the preset loss threshold.

[0040] It can be understood that model training is a repetitive iterative process. By calculating the loss function value after each iteration and continuously adjusting the model's network parameters for iterative training when the loss function value is greater than the preset value, the model's overall loss function value is less than or equal to the preset value, or the model's overall loss function value no longer changes or changes slowly, the model converges and a trained model is obtained.

[0041] The training method of the handwriting erasure model of the embodiment of the present disclosure obtains a training sample set, the training sample set includes sample images, the sample images include sample handwriting images and sample print images, the sample images are input into the handwriting erasure model to obtain the prediction result of the handwriting erasure model for the sample images, the prediction result includes the sample handwriting prediction map and the sample print prediction map, then the loss function value of the handwriting erasure model is calculated according to the sample handwriting prediction map, the sample handwriting image, the sample print prediction map and the sample print image, and then the handwriting erasure model is updated according to the loss function value. The network parameters of the handwriting erasure model are iteratively trained until the loss function value of the handwriting erasure model is less than or equal to a preset value to obtain a target handwriting erasure model. As a result, the trained target handwriting erasure model can detect and recognize handwriting and print in the image, which provides conditions for the subsequent use of the recognized handwriting prediction map to process the handwriting residues in the recognized print prediction map to obtain an image with handwriting traces erased cleanly, thereby improving the effect of handwriting erasure, and further improving the clarity of the image generated by the target handwriting erasure model in visual perception.

[0042] Optionally, in the process of training the handwriting erasure model, the number of training iterations can also be counted. When the number of training iterations of the handwriting erasure model reaches the iteration number threshold, the handwriting erasure model is considered to converge, and a trained target handwriting erasure model is obtained. Among them, the iteration number threshold can be determined according to the number of sample images in the training sample set. When the loss function value of the handwriting erasure model is greater than the preset value during the training process, but the number of training iterations reaches the iteration number threshold, the training is terminated to obtain a trained target handwriting erasure model. By setting the iteration number threshold, when the number of training iterations of the handwriting erasure model reaches the iteration number threshold, a trained target handwriting erasure model is obtained, and the model training process can be terminated in time when the loss function value of the handwriting erasure model cannot converge, avoiding the situation where the loss function value cannot converge and the model training cannot be terminated. In addition, during the training process, the image data can be enhanced accordingly, so that the model can learn the image data of illumination, watermark, and noise to enhance the model's ability to resist illumination, watermark, and noise. Specifically, during model training, the model can be made to learn sample images containing illumination, watermarks, noise, etc., and the classification of each pixel point also includes illumination class, watermark class, and noise class.

[0043] As an optional embodiment, after obtaining the target handwriting erasure model, the following steps are included: The target handwriting erasure model is optimized based on a network pruning module.

[0044] In this implementation, by selecting a suitable network pruning algorithm, such as pruning according to weight importance, channel pruning, structured pruning, etc. These algorithms can be selected according to the model structure and requirements. According to the selected pruning algorithm, the model is pruned to remove unimportant neurons, connections or layers, thereby reducing the complexity and computational burden of the model. After pruning, the model is fine-tuned to recover the performance loss caused by the pruning operation. The fine-tuning process can be achieved by continuing to train on the pruned model. The pruned and fine-tuned model is evaluated again to check whether its performance on the test set has improved, including indicators such as image quality and running time. That is, using cosine similarity as the similarity indicator, the similarity between the output of each module in the network and the output of the last residual module is calculated. The output of the last residual module is regarded as the standard output of the network. Multiple images are input into the network to obtain the output of each module, and then their cosine similarity with the output of the last module is calculated, and the average is taken to obtain the average similarity of each module. Among them, the similarity reflects the contribution of the module to the improvement of network performance. The greater the increase in similarity, the higher the contribution, which indicates that the network layer is more important to the overall network structure. When the similarity is lower than the preset similarity threshold, the convolution modules below the preset similarity threshold are removed, thereby reducing the number of network parameters and the amount of calculation. In order to ensure that the performance of the pruned network is consistent with the previous one, it is necessary to fine-tune the pruned network. Specifically, the Adam learning rate optimizer is used, the initial learning rate is set to 0.0001, the learning rate decays by half every 100 iterations, and the total training cycle is 300 epochs.

[0045] In this embodiment, by selecting a structured pruning method, pruning is performed according to the weight size, and the part with a smaller weight is deleted. After the pruning operation, we fine-tune the model and continue to train the pruned model on the training set so that it can relearn and adapt to the new structure. Finally, the performance of the model on the test set is re-evaluated. Through the optimization of network pruning, we can improve the efficiency and performance of the target handwriting erasure model, making it better when erasing handwriting in images.

[0046] In an embodiment of the present application, by inputting the label image and the mask image into a handwriting erasure model based on a LAMA network and a decoder network, and performing an iterative training process, a target handwriting erasure model can be obtained that can quickly and accurately erase handwriting contained in an image, thereby achieving a better erasure effect and improving image details and clarity.

[0047] S204: Erasing the handwriting in the image to be processed using the target handwriting erasure model.

[0048] As an optional embodiment, erasing handwriting in the image to be processed using the target handwriting erasure model includes: After performing angle correction on the image to be processed, cropping the image to be processed based on a set picture size to obtain a cropped image to be processed; The cropped image to be processed is input into the target handwriting erasure model to perform handwriting erasure, and a shadow removal operation is performed on the background part of the cropped image to be processed except the text part, so as to complete the handwriting erasure of the image to be processed.

[0049] Specifically, after acquiring the image to be processed and performing angle correction on the image to be processed, the image to be processed is cropped based on the set picture size to obtain a cropped image to be processed, the cropped image to be processed is input into the trained erasure model to erase handwriting, and a de-shading operation is performed on the background part of the cropped image to be processed except the text part, thereby completing the erasure of handwritten characters in the image to be processed.

[0050] It can be seen that in the present application, when handwriting is erased from the cropped image to be processed and the background part of the cropped image to be processed except the text part (including text in various formats) is de-shadowed, the target handwriting erasure model provided by the embodiment of the present disclosure can realize the erasing of handwriting in the image to be processed + background de-shadowing through an end-to-end network. Compared with the prior art that regards de-shadowing as an upstream task of the model, the method provided by the embodiment of the present disclosure is more convenient and more efficient, and thus generates a target image with better erasing effect and better de-shadowing effect, thereby improving the erasing effect and de-shadowing effect of the image to be processed.

[0051] In summary, the present invention provides a method, device, equipment and medium for erasing handwriting, by acquiring an image to be processed, wherein the image to be processed is a real image containing printed handwriting and handwriting, processing the image to be processed to obtain a label image and a mask image, inputting the label image and the mask image into a handwriting erasure model based on a LAMA network and a decoder network, and performing iterative training until the trained handwriting erasure model meets the expected performance, and obtaining a target handwriting erasure model, and using the target handwriting erasure model to erase the handwriting in the image to be processed. Compared with the prior art, the present application obtains a target handwriting erasure model by inputting the generated label image and mask image into a handwriting erasure model based on a LAMA network and a decoder network for iterative training, which ultimately greatly improves the accuracy and efficiency of handwriting erasure, and improves the ability to distinguish handwritten characters from printed characters, thereby reducing the influence of complex background on the erasure effect.

[0052] See also Figure 3 , Figure 3 Schematic diagram of the structure of the handwriting erasing device provided by the embodiment of the present invention. In this embodiment, the terminal includes various units for executing Figure 3 For details, please refer to the steps in the corresponding embodiment. Figure 2 as well as Figure 2 For the convenience of explanation, only the parts related to this embodiment are shown. Figure 3 The handwriting erasing device 30 includes: an acquisition module 31, a processing module 32, a training module 33, and an erasing module 34.

[0053] An acquisition module 31 is used to acquire an image to be processed, wherein the image to be processed is a real image containing printed handwriting and handwritten handwriting; A processing module 32 is used to process the image to be processed to obtain a label image and a mask image; A training module 33 is used to input the label image and the mask image into a handwriting erasure model based on a LAMA network and a decoder network, and perform iterative training until the trained handwriting erasure model meets the expected performance, thereby obtaining a target handwriting erasure model; The erasing module 34 is used to erase the handwriting in the image to be processed by using the target handwriting erasure model.

[0054] Optionally, the processing module 32 is specifically used for: The handwritten characters in the image to be processed are removed and filled pixel by pixel by using image editing software to obtain a label image.

[0055] Optionally, the processing module 32 is further used for: Subtracting the image to be processed from the label image to obtain a difference map; Averaging the values ​​of the three channels of the difference map to obtain a difference grayscale map; Comparing the predicted score of each pixel in the difference grayscale image with a preset threshold, wherein the predicted score is used to indicate the probability that the corresponding pixel is handwritten; The pixel values ​​of the pixels whose prediction scores are greater than the preset threshold are set to 1, and the pixel values ​​of the pixels whose prediction scores are not greater than the preset threshold are set to 0, so as to obtain a mask image, wherein the mask image is a binary mask image.

[0056] Optionally, the training module 33 is specifically used for: Obtain training sample data set; Inputting the sample images in the training sample data set into the handwriting erasure model for training to obtain a predicted label image and a predicted mask image; Calculate the reconstruction loss value of the label image and the predicted label image in the training sample data set by using the reconstruction loss function; Calculating the segmentation loss value of the mask image and the predicted mask image in the training sample data set by using a mask loss function; Based on the reconstruction loss value and the segmentation loss value, the handwriting erasure model is trained until the sum of the reconstruction loss value and the segmentation loss value output by the trained handwriting erasure model is less than a preset loss threshold.

[0057] Optionally, the training module 33 is then specifically used for: The target handwriting erasure model is optimized based on a network pruning module.

[0058] Optionally, the erasing module 34 is specifically used for: After performing angle correction on the image to be processed, cropping the image to be processed based on a set picture size to obtain a cropped image to be processed; The cropped image to be processed is input into the target handwriting erasure model to perform handwriting erasure, and a shadow removal operation is performed on the background part of the cropped image to be processed except the text part, so as to complete the handwriting erasure of the image to be processed.

[0059] It should be noted that the information interaction, execution process and other contents between the above-mentioned units are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0060] Figure 4Schematic diagram of the structure of a computer device provided by an embodiment of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the figure), a memory, and a computer program stored in the memory and executable on at least one processor, and when the processor executes the computer program, the steps in the above-mentioned handwriting erasing method embodiment are implemented.

[0061] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 4 These are merely examples of computer devices and do not constitute limitations on the computer devices. The computer devices may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, they may also include a network interface, a display screen, and an input device.

[0062] In one embodiment, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor in a computer device, the computer device can perform the steps of any embodiment of handwriting erasure disclosed in the present invention, which will not be repeated here. The computer-readable storage medium can be non-volatile or volatile.

[0063] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0064] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of the operating device and the computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store operating devices, application programs, boot loaders (BootLoader), data, and other programs, such as program codes of computer programs, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0065] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0066] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0067] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for erasing handwritten text. It is characterized in that include: Acquire an image to be processed, wherein the image to be processed is a real image containing printed words and handwritten words; Processing the image to be processed to obtain a label image and a mask image; Inputting the label image and the mask image into a handwriting erasure model based on a LAMA network and a decoder network, and performing iterative training until the trained handwriting erasure model meets the expected performance, thereby obtaining a target handwriting erasure model; The target handwriting erasure model is used to erase the handwriting in the image to be processed.

2. The method for erasing handwriting as claimed in claim 1, It is characterized in that The step of processing the image to be processed to obtain a label image includes: The handwritten characters in the image to be processed are removed and filled pixel by pixel by using image editing software to obtain a label image.

3. The method for erasing handwriting as claimed in claim 1, It is characterized in that The step of processing the image to be processed to obtain a mask image includes: Subtracting the image to be processed from the label image to obtain a difference map; Averaging the values ​​of the three channels of the difference map to obtain a difference grayscale map; Comparing the predicted score of each pixel in the difference grayscale image with a preset threshold, wherein the predicted score is used to indicate the probability that the corresponding pixel is handwritten; The pixel values ​​of the pixels whose prediction scores are greater than the preset threshold are set to 1, and the pixel values ​​of the pixels whose prediction scores are not greater than the preset threshold are set to 0, so as to obtain a mask image, wherein the mask image is a binary mask image.

4. The method for erasing handwriting as claimed in claim 1, It is characterized in that The handwriting erasure model is trained in the following way: Obtain training sample data set; Inputting the sample images in the training sample data set into the handwriting erasure model for training to obtain a predicted label image and a predicted mask image; Calculate the reconstruction loss value of the label image and the predicted label image in the training sample data set by using the reconstruction loss function; Calculating the segmentation loss value of the mask image and the predicted mask image in the training sample data set by using a mask loss function; Based on the reconstruction loss value and the segmentation loss value, the handwriting erasure model is trained until the sum of the reconstruction loss value and the segmentation loss value output by the trained handwriting erasure model is less than a preset loss threshold.

5. The method for erasing handwriting as claimed in claim 1, It is characterized in that The step of erasing the handwriting in the image to be processed by using the target handwriting erasure model includes: After performing angle correction on the image to be processed, cropping the image to be processed based on a set picture size to obtain a cropped image to be processed; The cropped image to be processed is input into the target handwriting erasure model to perform handwriting erasure, and a shadow removal operation is performed on the background part of the cropped image to be processed except the text part, so as to complete the handwriting erasure of the image to be processed.

6. The method for erasing handwriting as claimed in claim 1, It is characterized in that After obtaining the target handwriting erasure model, the method includes: The target handwriting erasure model is optimized based on a network pruning module.

7. A handwriting erasing device, It is characterized in that include: An acquisition module, used for acquiring an image to be processed, wherein the image to be processed is a real image containing printed handwriting and handwritten handwriting; A processing module, used for processing the image to be processed to obtain a label image and a mask image; A training module, used for inputting the label image and the mask image into a handwriting erasure model based on a LAMA network and a decoder network, performing iterative training until the trained handwriting erasure model meets the expected performance, thereby obtaining a target handwriting erasure model; The erasing module is used to erase the handwriting in the image to be processed by using the target handwriting erasure model.

8. The handwriting erasing device as claimed in claim 7, It is characterized in that The processing module is also used for: Subtracting the image to be processed from the label image to obtain a difference map; Averaging the values ​​of the three channels of the difference map to obtain a difference grayscale map; Comparing the predicted score of each pixel in the difference grayscale image with a preset threshold, wherein the predicted score is used to indicate the probability that the corresponding pixel is handwritten; The pixel values ​​of the pixels whose prediction scores are greater than the preset threshold are set to 1, and the pixel values ​​of the pixels whose prediction scores are not greater than the preset threshold are set to 0, so as to obtain a mask image, wherein the mask image is a binary mask image.

9. A computer device, It is characterized in that The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the handwriting erasing method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the handwriting erasing method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Hand writing erasing method based on deep learning

    CN114708601A

  • Handwriting erasing method and device, electronic equipment and storage medium

    CN115019323A

  • Erasing optimization method and device for handwritten content in document image

    CN115578403A

  • Smooth surface highlight rapid removal method and system combining U2-Net and LaMa models

    CN115760619A

  • Image processing method and device, network model training method and device, equipment and medium

    CN116863017A

Cited By

  • Handwritten form erasing method and device, electronic equipment and storage medium

    CN121617111A