Handwriting erasing method and device, electronic equipment and computer storage medium

Abstract

The invention relates to an image processing technology, and provides a handwriting erasing method and device, electronic equipment and a computer storage medium, and the method comprises the steps: obtaining a to-be-processed image of which handwriting needs to be erased; inputting the to-be-processed image into a pre-trained handwriting erasing model to obtain an image after handwriting erasing, the handwriting erasing model being obtained by using a pre-trained teacher model to perform knowledge distillation training on a pre-constructed student model based on a preset distillation strategy; the preset distillation strategy comprises label distribution distillation used for adjusting label distribution of student models according to label distribution of teacher models, and feature distillation used for aligning and constraining intermediate layer feature maps of the student models according to intermediate layer feature maps of teacher models; and the attention distillation is used for adjusting the attention weight of the student model according to the attention weight of the teacher model. According to the invention, a high-precision handwriting erasing effect can be achieved by using the lightweight handwriting erasing model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a handwriting erasing method and device, electronic equipment and computer storage medium. BACKGROUND

[0002] With the rapid advancement of office automation and electronic archives, a large number of paper documents and historical archives are digitally stored and managed. In practical applications, there are often handwritten notes in documents, such as annotations, signatures, underlines, and marks. These additional information will interfere with the electronic processing, OCR recognition, text re-editing, and information extraction of the document, affecting the subsequent digital processing effect. Therefore, handwriting erasing has become a key task in document image processing, which aims to accurately remove handwritten traces of various colors, thicknesses, and shapes while preserving the original document background texture and printed text clarity as much as possible. This not only requires the algorithm to maintain the integrity of the text while removing the handwriting, but also needs to have strong adaptability and robustness to complex scenes, different handwriting styles, different paper textures, and lighting conditions of the document. Efficient and accurate handwriting erasing technology has important practical value for digital management, intelligent retrieval, and information security of documents.

[0003] Traditional filtering and morphological methods are prone to damage the text structure and have poor adaptability to different colors, thicknesses, and shapes of handwriting. Although the method based on the SegFormer-B5 network has high accuracy in the handwriting erasing task, it has large parameter quantity and slow inference speed, which cannot be applied to embedded or mobile deployment scenarios. SUMMARY

[0004] The present application relates to the field of image processing, in particular to a handwriting erasing method and device, electronic equipment and computer storage medium.

[0005] Embodiments of the present application can be implemented as follows: In a first aspect, the present application provides a handwriting erasing method, which comprises: obtaining a to-be-processed image in which handwriting needs to be erased; inputting the to-be-processed image into a pre-trained handwriting erasing model to obtain an image after handwriting erasing, the handwriting erasing model being obtained by pre-training a teacher model, and performing knowledge distillation training on a pre-constructed student model based on a preset distillation strategy; the preset distillation strategy comprises label distribution distillation for adjusting the label distribution of the student model according to the label distribution of the teacher model, feature distillation for aligning and constraining the intermediate layer feature map of the student model according to the intermediate layer feature map of the teacher model, and attention distillation for adjusting the attention weight of the student model according to the attention weight of the teacher model.

[0006] In an optional implementation, the training steps of the handwriting erasure model include: Obtain sample images and actual labels, where the actual label is the actual category of each pixel in the sample image; The sample image is input into the teacher model to obtain a first distribution and a first feature map extracted by the intermediate layer of the teacher model. The first distribution is the probability distribution of the predicted category of all pixels in the sample image output by the teacher model. The sample image is input into the student model to obtain soft labels, a second distribution, and a second feature map extracted by the intermediate layer of the student model. The soft labels include the predicted category of each pixel in the sample image output by the student model, and the second distribution is the probability distribution of all predicted categories in the soft labels. Calculate the label distribution distillation loss based on the first distribution and the second distribution; Calculate the characteristic distillation loss based on the first feature map and the second feature map; Calculate the attention distillation loss based on the attention weights of the teacher model and the student model; The student model is trained based on the actual labels, the soft labels, the label distribution distillation loss, the feature distillation loss, and the attention distillation loss, and the trained student model is used as the handwriting erasure model.

[0007] In an optional implementation, the step of calculating the label distribution distillation loss based on the first distribution and the second distribution includes: Using a preset distillation temperature coefficient, the first distribution and the second distribution are smoothed to obtain the processed first distribution and the processed second distribution. Calculate the KL divergence between the processed first distribution and the processed second distribution to obtain the distillation loss of the label distribution.

[0008] In an optional implementation, the step of calculating the characteristic distillation loss based on the first feature map and the second feature map includes: Perform a convolution operation on the second feature map so that the convolutioned second feature map has the same dimension as the first feature map; The L2 loss between the first feature map and the convolved second feature map is calculated to obtain the feature distillation loss, which is then used to constrain the second feature map.

[0009] In an optional implementation, the step of calculating the attention distillation loss based on the attention weights of the teacher model and the attention weights of the student model includes: Extract a first attention matrix, where each element in the first attention matrix represents the attention weight of the corresponding pixel in the teacher model; The second attention matrix of the student model is calculated based on the attention weight of each pixel and the corresponding input feature map of the attention layer. The attention distillation loss is obtained by calculating the L2 loss between the first attention matrix and the second attention matrix.

[0010] In an optional implementation, the step of training the student model based on the actual labels, the soft labels, the label distribution distillation loss, the feature distillation loss, and the attention distillation loss includes: Calculate the cross-entropy loss based on the actual label and the soft label; Determine the optimized region in the sample image, wherein the optimized region is a handwriting boundary region or a handwriting region and a non-handwriting region; Calculate the optimization loss based on the optimization region; The total loss is calculated based on the cross-entropy loss, the feature distillation loss, the attention distillation loss, the label distribution distillation loss, and the optimization loss; The parameters of the student model are adjusted based on the total loss until the preset conditions are met.

[0011] In an optional implementation, the optimization region is the handwriting boundary region, and the step of calculating the optimization loss based on the optimization region is as follows: Obtain the boundary region mask of the handwriting boundary region; The predicted handwriting boundary region is determined based on the soft label; The optimization loss is calculated based on the preset boundary weight coefficient, the boundary region mask, and the predicted handwriting boundary region, wherein the preset boundary weight coefficient is greater than a preset value.

[0012] In an optional implementation, the optimization region is a handwriting region and a non-handwriting region, and the step of calculating the optimization loss based on the optimization region is as follows: Obtain the handwriting region mask of the handwriting region and the background region mask of the non-handwriting region; The handwriting prediction region and background prediction region are determined based on the soft label; The optimization loss is calculated based on the preset handwriting region weight, the preset background region weight, the handwriting region mask, the background region mask, the handwriting prediction region, and the background prediction region, wherein the preset handwriting region weight is greater than the preset background region weight.

[0013] In a second aspect, the present invention provides a handwriting erasure device, the device comprising: The acquisition module is used to acquire the image to be processed from which the writing needs to be erased; The processing module is used to input the image to be processed into a pre-trained handwriting erasure model to obtain an image after erasing the handwriting. The handwriting erasure model is obtained by using a pre-trained teacher model and knowledge distillation training on a pre-constructed student model based on a preset distillation strategy. The preset distillation strategy includes label distribution distillation for adjusting the label distribution of the student model according to the label distribution of the teacher model, feature distillation for aligning and constraining the intermediate layer feature maps of the student model according to the intermediate layer feature maps of the teacher model, and attention distillation for adjusting the attention weights of the student model according to the attention weights of the teacher model.

[0014] Thirdly, the present invention provides an electronic device including a controller and a memory, the memory being used to store a program, and the controller being used to implement the handwriting erasure method as described in the first aspect when executing the program.

[0015] Fourthly, the present invention provides a computer storage medium having a computer program stored thereon, which, when executed by a controller, implements the handwriting erasure method as described in the first aspect.

[0016] Compared with the prior art, the present invention has the following beneficial effects: This invention utilizes a teacher model to train a student model using knowledge distillation, resulting in a handwriting erasure model. Due to the pre-defined distillation strategy employed during training, the label distribution of the student model can be adjusted based on the label distribution of the teacher model, the intermediate feature maps of the student model can be aligned and constrained based on the intermediate feature maps of the teacher model, and the attention weights of the student model can be adjusted based on the attention weights of the teacher model. Through this multi-level knowledge distillation strategy, the lightweight student model achieves a significant reduction in computational complexity while maintaining erasure accuracy and robustness close to that of a large teacher model. This achieves a balance between lightweight design and high accuracy, ultimately resulting in a high-precision handwriting erasure effect using a lightweight handwriting erasure model. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1This is a flowchart illustrating the handwriting erasure method provided in this embodiment.

[0019] Figure 2 This is a flowchart illustrating the process of training a student model to obtain a handwriting erasure model, as provided in this embodiment.

[0020] Figure 3 This is a block diagram illustrating the handwriting erasure device provided in this embodiment.

[0021] Figure 4 This is a block diagram of the electronic device provided in this embodiment.

[0022] Icons: 10-Electronic device; 11-Processor; 12-Memory; 13-Bus; 100-Handwriting erasure device; 110-Acquisition module; 120-Processing module; 130-Training module. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0024] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0025] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0026] In the description of this invention, it should be noted that if terms such as "upper," "lower," "inner," or "outer" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of this invention is usually placed, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0027] Furthermore, the terms "first" and "second" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0028] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.

[0029] To overcome the poor erasing effect of traditional handwriting erasing methods, deep learning networks such as CNN (Convolutional Neural Network), U-Net (U-shaped Convolutional Network), and GAN (Generative Adversarial Network) can be used to erase complex handwriting. However, this erasing method is prone to leaving residues at boundaries or in small text areas. The lightweight SegFormer-B0 network can also be used, but its erasing accuracy is insufficient, especially in complex handwriting and small text areas.

[0030] In view of this, this embodiment provides a handwriting erasure method, apparatus, electronic device, and computer storage medium. Its core improvement lies in employing a multi-level knowledge distillation strategy—label distribution distillation at the output layer, feature distillation at the feature layer, and attention distillation at the attention layer—to significantly reduce the computational complexity of the lightweight student model while maintaining erasure accuracy and robustness close to that of a large teacher model. This will be described in detail below.

[0031] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the handwriting erasure method provided in this embodiment. The method includes the following steps: Step S101: Obtain the image to be processed from which the writing needs to be erased.

[0032] In this embodiment, the image to be processed can be a digital image containing handwritten content. Its data format can be a common bitmap type such as JPEG, PNG, or TIFF. The image source includes, but is not limited to, images of paper documents captured by a scanner, photos of handwritten drafts taken by a mobile device, or screenshots of electronic notes recorded on a tablet terminal. The pixel space of the image to be processed simultaneously contains the ink features of the handwritten area and the paper background features of the non-handwritten area. The image to be processed can also be obtained by preprocessing the original image, such as size normalization, grayscale correction, or noise suppression.

[0033] Step S102: Input the image to be processed into the pre-trained handwriting erasure model to obtain the image after erasing the handwriting. The handwriting erasure model is obtained by using a pre-trained teacher model and knowledge distillation training on a pre-constructed student model based on a preset distillation strategy. The preset distillation strategy includes label distribution distillation for adjusting the label distribution of the student model according to the label distribution of the teacher model, feature distillation for aligning and constraining the intermediate layer feature maps of the student model according to the intermediate layer feature maps of the teacher model, and attention distillation for adjusting the attention weights of the student model according to the attention weights of the teacher model.

[0034] In this embodiment, the handwriting erasure model takes the image to be processed as input and performs end-to-end semantic parsing and pixel-level reconstruction on it through an internal neural network structure, outputting a new image with the same size as the input image but with the handwriting content removed, i.e., the image after the handwriting has been erased. In this output image, the original handwriting area is appropriately filled to match the surrounding background texture, achieving a visually natural and coherent restoration effect.

[0035] In this embodiment, knowledge distillation is a model compression and performance transfer technique aimed at enabling student models with fewer parameters and faster inference speeds to approximate or even approach the predictive performance of large teacher models. The teacher model is typically a deep neural network that has been extensively trained on a large number of labeled samples, possessing powerful feature extraction and distribution modeling capabilities; while the student model adopts a relatively lightweight architecture, facilitating deployment in resource-constrained environments. For example, the teacher model is SegFormer-B5, first trained on a handwritten digit erasure dataset, serving as a "teacher" with fixed parameters. The student model is SegFormer-B0. By introducing a knowledge distillation mechanism, the student model not only learns the hard targets represented by the real labels but also absorbs the soft knowledge generated by the teacher model during inference, thereby improving its generalization ability and detail reproduction accuracy.

[0036] In this embodiment, the teacher model predicts whether each pixel in the image to be processed belongs to "handwriting" or "non-handwriting". The label distribution output by the teacher model is the distribution of the class prediction results of all pixels in the image to be processed by the teacher model. Similarly, the label distribution output by the student model is the distribution of the class prediction results of all pixels in the image to be processed by the student model. Label distribution distillation is used to adjust the label distribution of the student model according to the label distribution output by the teacher model, that is, to guide the student model to imitate the overall distribution trend of the predicted class probability of each pixel by the teacher model, especially the implicit discriminative information on low confidence classes. Feature distillation is applied to the intermediate layer feature map level. By aligning the feature response maps extracted by the teacher model and the student model in the deep layers of the network, it ensures the consistency of their representation in the semantic space, thereby enhancing the student model's ability to perceive complex patterns. Attention distillation focuses on the transmission of the attention mechanism inside the model. By allowing the student model to learn the weight allocation rules of the teacher model for pixels at different positions in the attention layer, it enables the student model to concentrate more computing resources in key areas (such as handwriting edges) and improve the ability to preserve local structure.

[0037] The method described in this embodiment, through this multi-level knowledge distillation strategy, enables the lightweight student model to significantly reduce computational complexity while maintaining erasure accuracy and robustness close to that of a large teacher model. This achieves a balance between lightweight model and high accuracy, ultimately achieving high-precision handwriting erasure results using a lightweight handwriting erasure model.

[0038] In an optional implementation, in order to train a high-precision student model using the teacher model and ultimately obtain a handwriting erasure model that balances lightweight design and high precision, this embodiment also provides an implementation method for training the handwriting erasure model: First, obtain the sample image and the actual label, where the actual label is the actual category of each pixel in the sample image; In this embodiment, the sample image is digitized image data containing handwritten content, and its pixel space can clearly indicate the spatial distribution of handwritten and non-handwritten areas. The actual label serves as the ground truth annotation corresponding to the sample image, specifically manifested as a discrete label of the category to which each pixel belongs, such as the "handwritten" or "non-handwritten" category.

[0039] Secondly, the sample image is input into the teacher model to obtain the first distribution and the first feature map extracted by the intermediate layer of the teacher model. The first distribution is the probability distribution of the predicted category of all pixels in the sample image output by the teacher model. In this embodiment, the first distribution reflects the teacher model's confidence assessment of the category to which each pixel belongs, containing richer discriminative information than a single-category determination. Furthermore, in the intermediate layer of the teacher model, the model extracts feature response maps with high-level semantic meaning, i.e., the first feature maps. These feature maps record the structured representation of the image at different levels of abstraction and can be considered important intermediate products of the teacher model's internal cognitive process.

[0040] Third, the sample image is input into the student model to obtain the soft label, the second distribution, and the second feature map extracted by the intermediate layer of the student model. The soft label includes the predicted category of each pixel in the sample image output by the student model, and the second distribution is the probability distribution of all predicted categories in the soft label. In this embodiment, the second distribution is similar to the first distribution, and the second feature map is similar to the first feature map. The difference is that the second distribution and the second feature map are obtained by inputting the sample images into the student model, while the first distribution and the first feature map are obtained by inputting the sample images into the teacher model.

[0041] Fourth, calculate the label distribution distillation loss based on the first and second distributions; In this embodiment, the label distribution distillation loss aims to make the output distribution of the student model as close as possible to that of the teacher model, especially demonstrating its advantage in learning low-confidence regions. By comparing the differences in the global prediction trends of the two models under the same input, the degree of deviation of the student model in imitating the "thinking style" of the teacher model is quantified. The introduction of this loss term allows the student model to not only focus on the consistency of correct classification results, but also to capture the potential patterns contained in the teacher model's judgments in uncertainty, thereby enhancing its generalization ability.

[0042] Fifth, calculate the characteristic distillation loss based on the first and second characteristic maps; In this embodiment, the feature distillation loss aims to focus on optimizing the consistency of feature representation in the intermediate layers of the model. By performing pointwise error measurement on the feature maps extracted by the teacher model and the student model under the same input, the student model is forced to reproduce the activation pattern of the teacher model in the latent space.

[0043] Sixth, calculate the attention distillation loss based on the attention weights of the teacher model and the student model; In this embodiment, attention weights reflect the distribution of attention intensity at different locations when the model processes images, showing a particularly significant response in key areas such as edges and texture variations. By acquiring the attention weights generated by the teacher model and the student model in their respective attention mechanism modules, the similarity between the two is evaluated, forming an attention distillation loss. This loss term guides the student model to learn the spatial attention strategy of the teacher model, enabling it to allocate higher attention weights to detail-sensitive areas such as handwriting boundaries, thereby improving the local coherence of the restoration results.

[0044] Seventh, the student model is trained based on the actual labels, soft labels, label distribution distillation loss, feature distillation loss, and attention distillation loss, and the trained student model is used as the handwriting erasure model.

[0045] In this embodiment, multiple loss terms are used together to construct a comprehensive optimization objective, including traditional loss based on real annotations and three-term distillation loss, which work together to update parameters during the training process of the student model.

[0046] In an optional implementation, in addition to having the student model learn from actual labels, it can also learn richer semantic information from the judgment and processing process of the teacher model, improving the robustness of erasing blurred, broken, and color-abnormal handwriting. This embodiment provides an implementation method for calculating the label distribution distillation loss: First, the first distribution and the second distribution are smoothed using a preset distillation temperature coefficient to obtain the processed first distribution and the processed second distribution. Next, the KL divergence between the processed first distribution and the processed second distribution is calculated to obtain the label distribution distillation loss.

[0047] In this embodiment, the preset distillation temperature coefficient is a scalar parameter greater than 1, used to smooth the first distribution. The preset distillation temperature coefficient can be set to 2~5. KL divergence (Kullback-Leibler Divergence) is an asymmetric measure of the similarity between two probability distributions, used to quantify the ability of the student model to simulate the output distribution of the teacher model.

[0048] In this embodiment, taking the teacher model as SegFormer-B5 and the student model as SegFormer-B0 as an example, the label distribution distillation loss can be calculated using the following formula: , in, For label distribution distillation loss, The logits output by the teacher model are the predicted values ​​output by the teacher model without the softmax processing. Softmax is a function that "compresses" a vector of arbitrary real values ​​into a probability distribution such that the value of each element is between (0, 1) and the sum of all elements is 1. The logits output by the student model are the predicted values ​​before softmax processing. T is a preset distillation temperature coefficient used to "smooth" the softmax, making the probability distribution "softer" and thus conveying more information about the relative relationships between categories. KL divergence is used to measure the difference between two distributions. and The differences between them. It is the first distribution. It is the second distribution.

[0049] In an optional implementation, to enable the student model to learn feature representations from the teacher model, thereby strengthening its semantic understanding and improving erasure performance when dealing with fine handwriting, intersecting strokes, and low-contrast areas, this embodiment also provides a method for calculating feature distillation loss: First, a convolution operation is performed on the second feature map so that the dimensions of the convolutioned second feature map are the same as those of the first feature map; In this embodiment, the first feature map includes intermediate layer feature maps extracted from the encoder and multilayer perceptron (MLP) layers of the teacher model, and the second feature map includes intermediate layer feature maps extracted from the encoder and MLP layers of the student model. Since the first feature map typically has a higher channel dimension or spatial resolution, a dimension adaptation operation may be needed on the second feature map output by the student model before computation to ensure comparability. The resulting feature distillation loss can effectively narrow the gap in semantic feature extraction capabilities between the two models and improve the representation quality of the student model. One alignment method is to perform a 1×1 convolution mapping on the second feature map to adjust its feature dimension to the same dimension as the corresponding layer of the teacher model. For example, a 1×1 convolution mapping can be performed on the intermediate layer feature map extracted from the encoder layer of the student model to adjust its feature dimension to the same dimension as the intermediate layer feature map extracted from the corresponding encoder layer of the teacher model.

[0050] Next, the L2 loss between the first feature map and the convolved second feature map is calculated to obtain the feature distillation loss, which is then used to constrain the second feature map.

[0051] In this embodiment, L2 loss is used to constrain the feature maps of the student model to be close to those of the teacher model, thereby achieving effective transfer of semantic knowledge from the intermediate layer. This allows the student model to directly acquire the semantic knowledge from the teacher's intermediate layer, improving its ability to recognize fine handwriting and complex boundaries. The feature distillation loss can be calculated using the following formula: ,in, Characteristic distillation loss, This is the feature map of the intermediate layer of the i-th layer in the teacher model. For the feature map of the i-th intermediate layer of the student model, g( ) is the mapping function, which is the result of convolving the feature maps of the i-th intermediate layer of the student model, and n is the number of all selected intermediate layers. The summation symbol in the formula represents the sum of the distillation losses of all selected intermediate layers. The distillation loss of each layer is the L2 distance between the feature maps of the teacher model and the student model after convolving the feature maps of that layer.

[0052] In an optional implementation, to enable the student model to learn higher-level structured knowledge such as integrated global information from the teacher model and improve the erasure effect in scenarios such as overlapping multi-colored handwriting and uneven lighting, this embodiment provides an implementation method for calculating attention distillation loss: First, the first attention matrix is ​​extracted, where each element represents the attention weight of the corresponding pixel in the teacher model. In this embodiment, during the distillation training of the student model, the parameters of the teacher model are fixed, and the first attention matrix can be directly extracted from the Self-Attention module (i.e., the attention module) through forward inference. During the forward propagation of the teacher model, its built-in Self-Attention module generates a normalized weight value for each spatial location in the input feature map, reflecting the importance of that location in information aggregation. These weight values ​​are arranged according to the original spatial resolution to form a two-dimensional or three-dimensional matrix structure, i.e., the first attention matrix, where high-weight regions typically correspond to semantically significant regions such as handwriting outlines and character connections.

[0053] Secondly, the second attention matrix of the student model is calculated based on the attention weight of each pixel in the student model and the corresponding input feature map of the attention layer. In this embodiment, taking the Transformer's Self-Attention module in the student model as an example, the second attention matrix for each layer can be calculated using the following formula: ,in, , , ; This is the input feature map for this layer; All of these are learnable weights for the student model; The first matrix is ​​the "attention score"; after applying softmax to it, we get the attention map, which is the second attention matrix, reflecting the correlation strength between features at different locations.

[0054] Third, calculate the L2 loss between the first attention matrix and the second attention matrix to obtain the attention distillation loss.

[0055] In this embodiment, the attention distillation loss can be calculated using the following formula: ,in, This is the first attention matrix. This is the second attention matrix.

[0056] It should be noted that, in addition to L2 loss, attention distillation loss can also be calculated using KL divergence or cosine similarity.

[0057] In an optional implementation, to further improve the erasure effect of handwriting edges without accidentally erasing the background, this embodiment also introduces an optimization region. The optimization loss calculated based on the optimization region, along with the cross-entropy loss, feature distillation loss, attention distillation loss, and label distribution distillation loss, are used to train the student model. One specific implementation is as follows: First, calculate the cross-entropy loss based on the actual labels and soft labels; In this embodiment, cross-entropy loss is used to quantify the difference between soft labels and actual labels, directly driving the student model to approximate the true label distribution globally, ensuring that it has basic discrimination accuracy.

[0058] Secondly, the optimization region in the sample image is determined. The optimization region is either the handwriting boundary region or the handwriting region and the non-handwriting region. In this embodiment, the optimized region does not cover all pixels of the entire image, but rather a key subset predefined according to task requirements. When set as a handwriting boundary region, it refers to the transition zone where the handwriting meets the background, which typically has high spatial complexity and is difficult to repair; when set as a handwriting region and a non-handwriting region, it corresponds to the area where the written content is located and the remaining paper background area, respectively.

[0059] Third, calculate the optimization loss based on the optimization region; In this embodiment, if the optimization region is a handwriting boundary region, the edge restoration quality is quantified by comparing the consistency between the predicted boundary and the true boundary; if the optimization region is a handwriting region and a non-handwriting region, the prediction bias in the two types of regions is calculated separately, and a region weight factor is introduced for imbalance correction. This optimization loss, as a highly targeted auxiliary objective, can effectively alleviate the optimization shortcomings of traditional loss functions in cases of uneven class distribution or blurred boundaries, and improve the robustness of the model in key areas.

[0060] Fourth, calculate the total loss based on the cross-entropy loss, feature distillation loss, attention distillation loss, label distribution distillation loss, and optimization loss; In this embodiment, cross-entropy loss ensures basic classification accuracy, label distribution distillation loss promotes alignment of output probability distributions, feature distillation loss strengthens consistency of intermediate representations, attention distillation loss transfers spatial attention patterns, and optimization loss focuses on fine-tuning specific regions. These losses are balanced using learnable or preset weight coefficients to form a unified scalar objective function, which serves as the basis for parameter updates. This composite loss structure enables the student model to collaboratively optimize under multi-dimensional constraints, balancing overall performance with local details.

[0061] Fifth, adjust the parameters of the student model based on the total loss until the preset conditions are met.

[0062] In this embodiment, the preset termination condition can be a condition such as the loss value stabilizing, the validation set performance no longer improving, or the completion of a specified number of training rounds.

[0063] It should be noted that, depending on the actual accuracy and efficiency requirements and the computational cost of various losses, one or more of the following can be chosen to calculate the total loss: cross-entropy loss, feature distillation loss, attention distillation loss, label distribution distillation loss, and optimization loss. For example, when extremely high accuracy is required, the total loss can be calculated using cross-entropy loss, feature distillation loss, attention distillation loss, label distribution distillation loss, and optimization loss. When extremely high efficiency is required, one or more of the less computationally intensive methods can be chosen to calculate the total loss.

[0064] In an optional implementation, to further improve the erasure effect of handwriting edges, this embodiment provides an optimization loss calculation method for the optimization area being the handwriting boundary region: First, obtain the boundary region mask of the handwriting boundary region; In this embodiment, the boundary region mask is used to identify the set of pixels in the sample image that belong to the boundary between the handwriting and the surrounding non-handwriting regions. Edge detection algorithms such as the Sobel operator can be used to identify a narrow band region of a certain width around the handwriting outline and mark it as "1", while the remaining regions are marked as "0", thereby forming a boundary region mask with clear spatial positioning function.

[0065] Secondly, the predicted handwriting boundary region is determined based on the soft label; In this embodiment, the distribution of handwriting boundaries currently recognized by the model can be obtained by calculating the gradient magnitude of the predicted probability map or by tracing contour lines. This is the predicted handwriting boundary region, which reflects the student model's understanding of the handwriting outline under the current parameter configuration.

[0066] Third, the optimization loss is calculated based on the preset boundary weight coefficient, the boundary region mask, and the predicted handwriting boundary region. The preset boundary weight coefficient is greater than the preset value.

[0067] In this embodiment, the preset boundary weight coefficient assigns a higher contribution weight only to pixels marked as boundaries in the mask, while other areas maintain a lower or unit weight. The preset boundary weight coefficient is a positive number greater than a preset value (such as 1), used to further amplify the loss response intensity of the boundary areas.

[0068] When the optimization region is the handwriting boundary region, one way to calculate the optimization loss is as follows: ,in, This refers to the boundary-aware loss, specifically the optimization loss when the optimization region is the boundary region of the handwriting. For the preset boundary weight coefficients, This indicates that errors in the boundary region will be "punished more severely." For the boundary region mask, To predict the handwriting boundary region, CE( ) represents the cross-entropy loss.

[0069] In an optional implementation, to avoid confusing handwritten and non-handwritten areas and causing accidental erasure of non-handwritten areas, this embodiment calculates the optimization loss as follows when the optimization area is both a handwritten and non-handwritten area: First, obtain the handwriting area mask of the handwriting area and the background area mask of the non-handwriting area; Secondly, the handwriting prediction area and background prediction area are determined based on the soft label; Third, the optimization loss is calculated based on the preset handwriting region weight, preset background region weight, handwriting region mask, background region mask, handwriting prediction region, and background prediction region, with the preset handwriting region weight being greater than the preset background region weight.

[0070] In this embodiment, the handwriting region and the non-handwriting region can be divided into sample images during training. The weight of the handwriting region is preset to be greater than the weight of the background region. That is, the handwriting region loss is given a higher weight to ensure that the model can completely remove the handwriting. The non-handwriting region (i.e., background and printed text) is given a lower weight to reduce the risk of erasing the text.

[0071] When the optimization region is both a handwriting region and a non-handwriting region, one way to calculate the optimization loss is as follows: ,in, The optimization loss is calculated when the optimization region is both a handwriting region and a non-handwriting region. and These are the preset handwriting area weights and the preset background area weights, respectively. This indicates that higher training attention is being given to the handwriting area. and These are the handwriting region mask and the handwriting prediction region, respectively. and These are the background region mask and the background prediction region, respectively.

[0072] It should be noted that in practical applications, one of the two methods for calculating optimization loss can be chosen, or both can be used simultaneously.

[0073] In this embodiment, to more clearly demonstrate the student model training process, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating the process of training a student model to obtain a handwriting erasure model, as provided in this embodiment. Figure 2 In this model, the teacher model is SegFormer-B5, the student model is SegFormer-B0, and the sample images are handwritten document images. The sample images and actual labels are input into the teacher and student models respectively. The teacher model obtains the first distribution, the first feature map, and the first attention matrix, while the student model obtains the second distribution, the second feature map, the second attention matrix, and soft labels. Cross-entropy loss is calculated based on the soft labels and actual labels. Label distribution distillation loss is calculated based on the first and second distributions. Feature distillation loss is calculated based on the first and second feature maps. Attention distillation loss is calculated based on the first and second attention matrices. Optimization loss is calculated based on the optimization region, including boundary-aware loss and region-weighted loss. The total loss is calculated based on all the above losses. One formula for calculating the total loss is:

[0074] in, The weights are used to calculate the loss for the true labels, i.e., the cross-entropy loss. loss weights, Distillation loss for label distribution The weight can be set from 0.5 to 2.0, and can be coordinated with T, which is the preset distillation temperature coefficient, and can be set from 2 to 6, with 2 to 4 being commonly used. The weight for the characteristic distillation loss is set to 0.5-2.0 when there are multiple intermediate layers. The characteristic distillation loss of the i-th intermediate layer, The characteristic distillation loss of the i-th intermediate layer The weight, To represent the total number of intermediate feature layers used for feature layer distillation, for example, if we select the 2nd, 4th, and 6th layers of the teacher and student encoders for aligned distillation, then... , The weights for the attention distillation loss are set to 0.1-1.0. The attention distillation loss for the j-th attention layer The weight, This refers to the total number of attention layers used for attention distillation. For example, if three multi-head attention layers are selected in the teacher model and aligned with the corresponding attention layers in the student model, then... , To optimize the boundary-aware loss in the loss function The weights are set to 1.0-5.0. When it is necessary to increase the weights of the boundary awareness loss, they can be set higher. To optimize the region-weighted loss in the loss calculation The weight can be set to 1.0. It can be further subdivided into preset handwriting area weights. Preset background area weight , The value is usually set between 2 and 10. It can be increased if the handwriting is very sparse. .

[0075] To perform the corresponding steps in the above embodiments and various possible implementations, an implementation of the handwriting erasure device 100 is given below. Please refer to... Figure 3 , Figure 3This is a block diagram of the handwriting erasure device provided in this embodiment. It should be noted that the handwriting erasure device 100 provided by the present invention has the same basic principle and technical effect as the corresponding embodiment described above. For the sake of brevity, this embodiment does not mention or point out some of these aspects.

[0076] The handwriting erasure device 100 includes an acquisition module 110 and a processing module 120.

[0077] The acquisition module 110 is used to acquire the image to be processed from which the writing needs to be erased; The processing module 120 is used to input the image to be processed into a pre-trained handwriting erasure model to obtain an image after erasing the handwriting. The handwriting erasure model is obtained by using a pre-trained teacher model and knowledge distillation training on a pre-constructed student model based on a preset distillation strategy. The preset distillation strategy includes label distribution distillation for adjusting the label distribution of the student model according to the label distribution of the teacher model, feature distillation for aligning and constraining the intermediate layer feature maps of the student model according to the intermediate layer feature maps of the teacher model, and attention distillation for adjusting the attention weights of the student model according to the attention weights of the teacher model.

[0078] In an optional implementation, the handwriting erasure device 100 further includes a training module 130.

[0079] The training module 130 is used for: acquiring sample images and actual labels, where the actual label is the actual category of each pixel in the sample image; inputting the sample image into the teacher model to obtain a first distribution and a first feature map extracted by the intermediate layer of the teacher model, where the first distribution is the probability distribution of the predicted categories of all pixels in the sample image output by the teacher model; inputting the sample image into the student model to obtain soft labels, a second distribution, and a second feature map extracted by the intermediate layer of the student model, where the soft labels include the predicted categories of each pixel in the sample image output by the student model, and the second distribution is the probability distribution of all predicted categories in the soft labels; calculating the label distribution distillation loss based on the first and second distributions; calculating the feature distillation loss based on the first and second feature maps; calculating the attention distillation loss based on the attention weights of the teacher model and the student model; training the student model based on the actual labels, soft labels, label distribution distillation loss, feature distillation loss, and attention distillation loss, and using the trained student model as a handwriting erasure model.

[0080] In an optional implementation, the training module 130 is specifically used for: Using a preset distillation temperature coefficient, the first distribution and the second distribution are smoothed to obtain the processed first distribution and the processed second distribution. The KL divergence between the processed first distribution and the processed second distribution is calculated to obtain the label distribution distillation loss.

[0081] In an optional implementation, the training module 130 is further configured to: Perform a convolution operation on the second feature map so that the dimensions of the convolutioned second feature map are the same as those of the first feature map; Calculate the L2 loss between the first feature map and the convolved second feature map to obtain the feature distillation loss, and use the feature distillation loss to constrain the second feature map.

[0082] In an optional implementation, the training module 130 is further configured to: Extract the first attention matrix, where each element represents the attention weight of the corresponding pixel in the teacher model; The second attention matrix of the student model is calculated based on the attention weight of each pixel and the corresponding input feature map of the attention layer. Calculate the L2 loss between the first attention matrix and the second attention matrix to obtain the attention distillation loss.

[0083] In an optional implementation, the training module 130 is further configured to: Calculate the cross-entropy loss based on the actual labels and soft labels; Determine the optimization region in the sample image. The optimization region can be either the handwriting boundary region or the handwriting region and the non-handwriting region. Calculate the optimization loss based on the optimization region; The total loss is calculated based on the cross-entropy loss, feature distillation loss, attention distillation loss, label distribution distillation loss, and optimization loss. The parameters of the student model are adjusted based on the total loss until the preset conditions are met.

[0084] In an optional implementation, the optimization region is the handwriting boundary region, and the training module 130 is specifically used to calculate the optimization loss based on the optimization region as follows: Obtain the boundary region mask of the handwriting boundary region; Determine the predicted handwriting boundary region based on soft tags; The optimization loss is calculated based on the preset boundary weight coefficient, the boundary region mask, and the predicted handwriting boundary region. The preset boundary weight coefficient is greater than the preset value.

[0085] In an optional implementation, the optimization region is a handwriting region and a non-handwriting region, and the training module 130 is specifically used to calculate the optimization loss based on the optimization region as follows: Obtain the mask of the handwriting area and the mask of the background area (non-handwriting area); Determine the handwriting prediction area and background prediction area based on the soft label; The optimization loss is calculated based on the preset handwriting region weight, preset background region weight, handwriting region mask, background region mask, handwriting prediction region, and background prediction region. The preset handwriting region weight is greater than the preset background region weight.

[0086] This invention also provides a block diagram of an electronic device 10, which implements the handwriting erasure method of this embodiment. Please refer to... Figure 4 , Figure 4 This is a block diagram of the electronic device 10 provided in this embodiment. The electronic device 10 includes a processor 11, a memory 12 and a bus 13. The processor 11 and the memory 12 are connected through the bus 13.

[0087] The processor 11 can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the handwriting erasure method described above can be completed by the integrated logic circuitry in the hardware of the processor 11 or by software instructions. The processor 11 can be a general-purpose processor, including a CPU (Central Processing Unit), an NP (Network Processor), etc.; it can also be a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Logic Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0088] The memory 12 is used to store a program that implements the handwriting erasure method. The program can be a software function module stored in the memory 12 in the form of software or firmware or embedded in the OS (Operating System) of the electronic device 10.

[0089] After receiving the execution instruction, the processor 11 executes the program to implement the handwriting erasure method of the aforementioned embodiment.

[0090] In summary, embodiments of the present invention provide a handwriting erasure method, apparatus, electronic device, and computer storage medium. The method includes: acquiring an image to be processed from which handwriting needs to be erased; inputting the image to be processed into a pre-trained handwriting erasure model to obtain an image after handwriting erasure. The handwriting erasure model is obtained by using a pre-trained teacher model and knowledge distillation training on a pre-constructed student model based on a preset distillation strategy. The preset distillation strategy includes label distribution distillation for adjusting the label distribution of the student model according to the label distribution of the teacher model, feature distillation for aligning and constraining the intermediate layer feature maps of the student model according to the intermediate layer feature maps of the teacher model, and attention distillation for adjusting the attention weights of the student model according to the attention weights of the teacher model. Compared with the prior art, this embodiment has at least the following advantages: (1) The preset distillation strategy used in knowledge distillation training can adjust the label distribution of the student model according to the label distribution of the teacher model, align and constrain the intermediate layer feature map of the student model according to the intermediate layer feature map of the teacher model, and adjust the attention weight of the student model according to the attention weight of the teacher model. Through this multi-level knowledge distillation strategy, the lightweight student model can maintain the erasure accuracy and robustness close to that of the large teacher model while significantly reducing the computational complexity, thus achieving a balance between lightweight and high accuracy, and finally achieving a high-precision handwriting erasure effect using a lightweight handwriting erasure model; (2) The preset distillation temperature coefficient is used to smooth the output label distribution of the teacher model, thereby conveying more relative relationship information between categories and reducing the learning difficulty of the student model; (3) (3) By using the feature maps of the intermediate layer of the teacher model and the student model to obtain feature distillation, the student model can obtain richer semantic representation capabilities and improve the student model's ability to identify small handwriting and complex boundaries; (4) By using attention distillation, the student model can improve its ability to capture subtle differences between handwriting and background, so that the student model can inherit the teacher's attention pattern of "separation of handwriting and background areas", thereby removing handwriting traces more accurately in complex documents; (5) By using the handwriting edge area as the optimization area, the optimization loss is calculated to ensure that the student model's prediction at the boundary position is more accurate and to reduce "handwriting residue" or "text mis-erasing"; (6) By using the handwriting area and non-handwriting area as the optimization area, the optimization loss is calculated. The handwriting area loss is given a higher weight to ensure that the model can completely remove handwriting. The non-handwriting area (i.e., background and printed text) is given a lower weight to reduce the risk of erasing the main text.

[0091] The above descriptions are merely various embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for erasing handwriting, characterized in that, The method includes: Obtain the image to be processed from which the writing needs to be erased; The image to be processed is input into a pre-trained handwriting erasure model to obtain an image after handwriting erasure. The handwriting erasure model is obtained by using a pre-trained teacher model and a pre-constructed student model for knowledge distillation training based on a preset distillation strategy. The preset distillation strategy includes label distribution distillation for adjusting the label distribution of the student model according to the label distribution of the teacher model, feature distillation for aligning and constraining the intermediate layer feature maps of the student model according to the intermediate layer feature maps of the teacher model, and attention distillation for adjusting the attention weights of the student model according to the attention weights of the teacher model.

2. The method according to claim 1, characterized in that, The training steps for the handwriting erasure model include: Obtain sample images and actual labels, where the actual label is the actual category of each pixel in the sample image; The sample image is input into the teacher model to obtain a first distribution and a first feature map extracted by the intermediate layer of the teacher model. The first distribution is the probability distribution of the predicted category of all pixels in the sample image output by the teacher model. The sample image is input into the student model to obtain soft labels, a second distribution, and a second feature map extracted by the intermediate layer of the student model. The soft labels include the predicted category of each pixel in the sample image output by the student model, and the second distribution is the probability distribution of all predicted categories in the soft labels. Calculate the label distribution distillation loss based on the first distribution and the second distribution; Calculate the characteristic distillation loss based on the first feature map and the second feature map; Calculate the attention distillation loss based on the attention weights of the teacher model and the student model; The student model is trained based on the actual labels, the soft labels, the label distribution distillation loss, the feature distillation loss, and the attention distillation loss, and the trained student model is used as the handwriting erasure model.

3. The method according to claim 2, characterized in that, The step of calculating the label distribution distillation loss based on the first distribution and the second distribution includes: Using a preset distillation temperature coefficient, the first distribution and the second distribution are smoothed to obtain the processed first distribution and the processed second distribution. Calculate the KL divergence between the processed first distribution and the processed second distribution to obtain the distillation loss of the label distribution.

4. The method according to claim 2, characterized in that, The step of calculating the characteristic distillation loss based on the first feature map and the second feature map includes: Perform a convolution operation on the second feature map so that the convolutioned second feature map has the same dimension as the first feature map; The L2 loss between the first feature map and the convolved second feature map is calculated to obtain the feature distillation loss, which is then used to constrain the second feature map.

5. The method according to claim 2, characterized in that, The step of calculating the attention distillation loss based on the attention weights of the teacher model and the attention weights of the student model includes: Extract the first attention matrix, where each element in the first attention matrix represents the attention weight of the corresponding pixel in the teacher model; The second attention matrix of the student model is calculated based on the attention weight of each pixel and the corresponding input feature map of the attention layer. The attention distillation loss is obtained by calculating the L2 loss between the first attention matrix and the second attention matrix.

6. The method according to claim 2, characterized in that, The step of training the student model based on the actual labels, the soft labels, the label distribution distillation loss, the feature distillation loss, and the attention distillation loss includes: Calculate the cross-entropy loss based on the actual label and the soft label; Determine the optimized region in the sample image, wherein the optimized region is a handwriting boundary region or a handwriting region and a non-handwriting region; Calculate the optimization loss based on the optimization region; The total loss is calculated based on the cross-entropy loss, the feature distillation loss, the attention distillation loss, the label distribution distillation loss, and the optimization loss; The parameters of the student model are adjusted based on the total loss until the preset conditions are met.

7. The method according to claim 6, characterized in that, The optimization region is the handwriting boundary region, and the step of calculating the optimization loss based on the optimization region is as follows: Obtain the boundary region mask of the handwriting boundary region; The predicted handwriting boundary region is determined based on the soft label; The optimization loss is calculated based on the preset boundary weight coefficient, the boundary region mask, and the predicted handwriting boundary region, wherein the preset boundary weight coefficient is greater than a preset value.

8. The method according to claim 6, characterized in that, The optimization region is a handwriting region and a non-handwriting region. The step of calculating the optimization loss based on the optimization region is as follows: Obtain the handwriting region mask of the handwriting region and the background region mask of the non-handwriting region; The handwriting prediction region and background prediction region are determined based on the soft label; The optimization loss is calculated based on the preset handwriting region weight, the preset background region weight, the handwriting region mask, the background region mask, the handwriting prediction region, and the background prediction region, wherein the preset handwriting region weight is greater than the preset background region weight.

9. A pen-erasing device, characterized in that, The device includes: The acquisition module is used to acquire the image to be processed from which the writing needs to be erased; The processing module is used to input the image to be processed into a pre-trained handwriting erasure model to obtain an image after erasing the handwriting. The handwriting erasure model is obtained by using a pre-trained teacher model and knowledge distillation training on a pre-constructed student model based on a preset distillation strategy. The preset distillation strategy includes label distribution distillation for adjusting the label distribution of the student model according to the label distribution of the teacher model, feature distillation for aligning and constraining the intermediate layer feature maps of the student model according to the intermediate layer feature maps of the teacher model, and attention distillation for adjusting the attention weights of the student model according to the attention weights of the teacher model.

10. An electronic device, characterized in that, It includes a controller and a memory, the memory being used to store a program, and the controller being used to implement the handwriting erasure method as described in any one of claims 1-8 when executing the program.

11. A computer storage medium, characterized in that, It stores a computer program that, when executed by the controller, implements the handwriting erasure method as described in any one of claims 1-8.

Images