Paper handwriting restoration method and device, equipment, medium and product

Abstract

The invention discloses a paper handwriting restoration method and device, equipment, a medium and a product. The method comprises the following steps: preprocessing a color image and a depth image of target paper; generating a handwriting depth feature map and a handwriting thickness feature map of the target paper according to the preprocessed color image and depth image, and fusing the two types of feature maps to obtain a handwriting feature map of the target paper; and fusing the handwriting feature map and the handwriting segmentation image, filling missing handwriting in the fused image, and generating a handwriting restoration image of the target paper. According to the method, the faded and disappeared handwriting can be repaired according to the handwriting dent depth information on the paper, the repair direction is calibrated based on the physical dent information, misjudgment caused by only depending on pixel information is avoided, the handwriting dent serves as the physical retention information and is not influenced by paper fading and surface stains, the missing handwriting structure information of the original image can be supplemented, and the accuracy of the original image is improved. And the accuracy of paper handwriting restoration is improved.

Description

Technical Field

[0001] This invention relates to the field of handwriting restoration technology, and more particularly to a method, apparatus, equipment, medium, and product for restoring handwriting on paper. Background Technology

[0002] Old paper (such as antique books, manuscripts, archives, and company expense receipts) carries important information from historical, cultural, and commercial fields. However, due to the passage of time, writing tools, and storage environment, handwriting on paper often becomes blurred, faded, discontinuous, or even disappears, requiring handwriting restoration and reconstruction. Existing methods for paper handwriting restoration and reconstruction mainly include optical methods, chemical methods, and digital image processing technology. However, these methods have certain limitations in practical applications: Optical methods and digital image processing techniques have high requirements for paper quality and preservation. If the paper has stains, creases, discoloration, etc., optical methods will be unable to accurately capture the handwriting features due to severe interference (for example, stains may be misjudged as handwriting, and creases may cause image distortion). Digital image processing techniques also struggle to effectively remove complex interference, resulting in a significant decrease in restoration effectiveness. For blurry or discontinuous handwriting, both methods lack effective means to connect the broken parts, making it impossible to accurately restore the handwriting shape.

[0003] Because the effects of chemical reagents on paper and handwriting are unpredictable, improper reagent selection or reaction control errors can lead to the complete disappearance of handwriting and irreparable damage. Therefore, chemical methods are generally risky, difficult to operate, and inefficient. Furthermore, chemical methods require professional technicians to accurately select chemical reagents and control reaction conditions based on paper material and handwriting type, which raises the operational threshold. At the same time, chemical reactions require a certain amount of time to proceed fully, which cannot meet the needs of urgent restoration scenarios.

[0004] Therefore, existing paper handwriting restoration and reconstruction methods rely solely on pixel information, optical contrast, or chemical reactions, without considering key features such as the depth and physical morphology of handwriting indentations, which easily leads to misjudgment of handwriting morphology. When the original paper handwriting has low resolution, is severely blurred, or has a lot of noise, it is difficult to effectively identify and restore handwriting due to the lack of supplementary information. Summary of the Invention

[0005] This invention provides a method, apparatus, device, medium, and product for paper handwriting restoration. By obtaining the depth information of handwriting indentations on the paper based on the color and depth images of the paper, the handwriting indentation information is used to restore faded or disappeared handwriting. The restoration direction can be calibrated based on physical indentation information, avoiding misjudgments caused by relying solely on pixel information. Furthermore, as physical retention information, the handwriting indentations are not affected by paper fading or surface stains and can supplement the missing handwriting structure information in the original image, thereby improving the accuracy of paper handwriting restoration.

[0006] To achieve the above objectives, embodiments of the present invention provide a paper handwriting restoration method, comprising: Acquire color and depth images of the target paper, and preprocess the color and depth images; Based on the preprocessed color image and depth image, generate a feature map of the depth and thickness of the handwriting on the target paper. The handwriting depth feature map and the handwriting thickness feature map are fused together to obtain the handwriting feature map of the target paper; The handwriting feature map and the handwriting segmentation image are fused together to fill in the missing handwriting in the fused image, thereby generating a handwriting restoration image of the target paper.

[0007] As an improvement to the above scheme, the step of generating a handwriting depth feature map and a handwriting thickness feature map of the target paper based on the preprocessed color image and depth image includes: The depth value corresponding to the original handwriting region is extracted from the preprocessed depth image based on the original handwriting region of the preprocessed color image. Linear mapping is performed on the depth values ​​to generate a feature map of the pen stroke depth on the target paper; The preprocessed depth image is subjected to mean calculation, morphological dilation, and binarization to generate a feature map of the stroke thickness on the target paper.

[0008] As an improvement to the above scheme, the step of performing mean calculation, morphological dilation, and binarization on the preprocessed depth image to generate a handwriting thickness feature map of the target paper includes: Calculate the local mean of each pixel in the preprocessed depth image to obtain the mean feature map; A morphological dilation operation is performed on the mean feature map, and the grayscale histogram of the dilated mean feature map is binarized to obtain the handwriting thickness feature map of the target paper.

[0009] As an improvement to the above solution, the step of fusing the handwriting depth feature map and the handwriting thickness feature map to obtain the handwriting feature map of the target paper includes: The handwriting region of the handwriting thickness feature map is determined based on the pixel values ​​of all pixels in the handwriting thickness feature map. Traverse the target coordinates of each pixel in the handwriting region, and calculate the local mean of the gray values ​​of the corresponding target coordinates in the handwriting depth feature map; The calculated mean gray value is used as the gray value of the corresponding target coordinate in the handwriting depth feature map to obtain the handwriting feature map of the target paper.

[0010] As an improvement to the above scheme, the step of traversing the target coordinates of each pixel in the handwriting region and calculating the local mean of the gray values ​​corresponding to the target coordinates in the handwriting depth feature map includes: For the target coordinates of each pixel in the handwriting region, a local calculation window of fixed size is defined in the handwriting depth feature map, centered on the target coordinates. The mean grayscale value of the target coordinates is obtained by calculating the mean grayscale value of all pixels within the local calculation window.

[0011] As an improvement to the above scheme, the step of using the calculated grayscale mean as the grayscale value of the corresponding target coordinate in the handwriting depth feature map to obtain the handwriting feature map of the target paper includes: The calculated mean gray value is used as the gray value of the corresponding target coordinate in the handwriting depth feature map; Keeping the remaining gray values ​​in the handwriting depth feature map unchanged, the handwriting feature map of the target paper is obtained.

[0012] As an improvement to the above solution, the step of fusing the handwriting feature map and the handwriting segmentation image to fill in the missing handwriting in the fused image and generate a handwriting restoration image of the target paper includes: The handwriting image of the target paper is obtained by performing handwriting image segmentation on the processed color image; The handwriting feature map and the handwriting segmentation image are fused through logical operations; Fill in the missing strokes in the fused image to generate a stroke restoration image of the target paper.

[0013] As an improvement to the above scheme, the step of fusing the handwriting feature map and the handwriting segmentation image through logical operations includes: Perform an OR logical operation on the pixel values ​​of all corresponding pixels in the handwriting feature map and the handwriting segmentation image; The OR operation results of all corresponding pixels are combined to form a binary image as the fused image.

[0014] As an improvement to the above scheme, the step of performing an OR logical operation on the pixel values ​​of all corresponding pixels in the handwriting feature map and the handwriting segmentation image includes: An OR operation is performed on the pixels in the handwriting segmentation image whose pixel value corresponds to the handwriting pixel threshold and the pixels in the handwriting feature image whose pixel value does not correspond to the baseline pixel threshold. Wherein, the handwriting pixel threshold is the pixel value of the pixel corresponding to the original handwriting region in the handwriting segmentation image; the reference pixel threshold is the pixel value of the pixel corresponding to the non-handwriting region in the handwriting feature map.

[0015] As an improvement to the above solution, the step of filling in the missing handwriting in the fused image to generate a handwriting restoration image of the target paper includes: The handwriting segmentation image is subjected to handwriting edge extraction, and an edge mask is created based on the handwriting edges; Define the diffusion width of the edge mask, and create a diffusion region mask based on the edge mask and the diffusion width; A handwriting restoration image of the target paper is generated based on the missing handwriting in the merged image using the handwriting edges, the diffusion width, and the diffusion area mask.

[0016] As an improvement to the above solution, the step of generating a handwriting restoration image of the target paper based on the missing handwriting in the blended image using the handwriting edge, the diffusion width, and the diffusion area mask filling method includes: The missing handwriting area in the fused image is determined based on the diffusion area mask; Calculate the pixel distance from each pixel in the missing handwriting region to the edge of the handwriting; The fill pixel value of each pixel in the missing handwriting area is calculated using a linear proportional formula based on the pixel distance and the diffusion width. A handwriting restoration image of the target paper is generated by filling the missing handwriting area with pixel values ​​based on the diffusion area mask and the filling pixel values.

[0017] To achieve the above objectives, embodiments of the present invention provide a paper handwriting restoration device, comprising: The paper image processing module is used to acquire color and depth images of the target paper and to preprocess the color and depth images. The handwriting feature generation module is used to generate a handwriting depth feature map and a handwriting thickness feature map of the target paper based on the preprocessed color image and depth image. The handwriting feature fusion module is used to fuse the handwriting depth feature map and the handwriting thickness feature map to obtain the handwriting feature map of the target paper; The handwriting image restoration module is used to fuse the handwriting feature map and the handwriting segmentation image, fill in the missing handwriting in the fused image, and generate the handwriting restoration image of the target paper.

[0018] To achieve the above objectives, embodiments of the present invention provide a paper handwriting restoration device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the above-described paper handwriting restoration method.

[0019] To achieve the above objectives, embodiments of the present invention also provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the above-described paper handwriting repair method.

[0020] To achieve the above objectives, embodiments of the present invention also provide a computer program product, which is stored in a storage medium and executed by at least one processor to implement the steps of the paper handwriting restoration method described above.

[0021] Compared with existing technologies, the present invention discloses a paper handwriting restoration method, apparatus, device, medium, and product. This method involves acquiring a color image and a depth image of a target paper, preprocessing the color and depth images, generating a handwriting depth feature map and a handwriting thickness feature map of the target paper based on the preprocessed color and depth images, fusing the handwriting depth and thickness feature maps to obtain a handwriting feature map of the target paper, and fusing the handwriting feature map with a handwriting segmentation image to fill in missing handwriting in the fused image, thereby generating a restored handwriting image of the target paper. This method can obtain the depth information of handwriting indentations (handwriting depth and thickness feature maps) on the paper based on the color and depth images, and use this indentation information to restore faded or disappeared handwriting. It achieves restoration direction calibration based on physical indentation information, avoiding misjudgments caused by relying solely on pixel information. Furthermore, the handwriting indentations, as physically preserved information, are unaffected by paper fading or surface stains and can supplement the missing handwriting structure information in the original image, improving the accuracy of paper handwriting restoration. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a paper handwriting repair method provided in an embodiment of the present invention; Figure 2 This is a flowchart of a paper handwriting repair method provided in an embodiment of the present invention; Figure 3 This is a handwriting thickness feature map provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a paper handwriting repair device provided in an embodiment of the present invention; Figure 5 This is a structural block diagram of a paper handwriting repair device provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] It should be noted that the terms "comprising" and "specific" in this invention, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0025] Please see Figure 1 , Figure 1 This is a flowchart illustrating a paper handwriting repair method provided in an embodiment of the present invention. The paper handwriting repair method includes: S1, acquire the color image and depth image of the target paper, and preprocess the color image and depth image; S2, Generate a feature map of pen stroke depth and a feature map of pen stroke thickness on the target paper based on the preprocessed color image and depth image; S3, the handwriting depth feature map and the handwriting thickness feature map are fused to obtain the handwriting feature map of the target paper; S4, the handwriting feature map and the handwriting segmentation image are fused together to fill in the missing handwriting in the fused image and generate the handwriting restoration image of the target paper.

[0026] For example, the paper handwriting restoration method is implemented by a restoration server, which is capable of interacting with the target user. Figure 2 As shown, Figure 2This is a flowchart of a paper handwriting restoration method provided by an embodiment of the present invention. The restoration server acquires a color image and a depth image of the target paper (the paper is laid flat on a stable table, and a high-precision depth camera is used to simultaneously capture an RGB format color image and a depth image containing pixel depth data). The color image records the color and handwriting appearance information of the paper surface, and the depth image records the depth value of each pixel in the image, reflecting the depth of the handwriting indentations on the paper surface. Morphological operations are performed on the handwriting in the color image, and filtering and planar calibration are performed on the depth image to obtain preprocessed color and depth images. Preprocessing is then performed to extract the preprocessed handwriting data. The processed depth image reflects the physical characteristics of handwriting (depth of indentations, thickness of strokes), generating a handwriting depth feature map and a handwriting thickness feature map for the target paper. These two feature maps are then fused to obtain the handwriting feature map for the target paper. A deep learning image segmentation algorithm is used to segment the preprocessed color image into pure handwriting regions, converting them to grayscale. Non-handwriting regions are filled with blank pixel values ​​to eliminate background interference, resulting in a handwriting segmentation map. The handwriting feature map and the segmented handwriting image are then fused. Missing handwriting elements in the fused image are filled in to generate a handwriting restoration image for the target paper. Combining the depth of handwriting indentations with physical characteristics (such as writing pressure reflecting stroke thickness and physical characteristics determining edge contours) makes the restoration result more consistent with actual writing conditions, improving the accuracy of handwriting restoration. When the original handwriting quality is poor (low resolution, blurry, noisy), indentation information can compensate for insufficient pixel information, providing clues such as handwriting direction and stroke order, reducing reliance on the quality of the original image.

[0027] It's worth noting that the preprocessing of the paper depth map includes two operations: filtering and calibration. First, depth filtering techniques (Gaussian filtering, bilateral filtering) are applied to smooth the depth data. Because pen pressure is generally continuous, the measurement error caused by discontinuous and minute unevenness on the paper surface can be reduced by designing filtering parameters. Second, a plane calibration method is used for calibration. For paper, its surface can be considered an ideal plane; depth correction is performed by comparing the actual depth data with the ideal plane.

[0028] Preprocessing of the paper color image (RGB image): mainly involves morphological operations on the handwriting. First, a dilation method is used to thicken the handwriting on the paper, and then an erosion operation is used to remove small noise in the paper image and maintain the continuity of the handwriting.

[0029] Specifically, step S2 includes: S21, extract the depth value corresponding to the original handwriting region from the preprocessed depth image based on the original handwriting region of the preprocessed color image; S22, perform linear mapping on the depth value to generate a handwriting depth feature map of the target paper; S23, perform mean calculation, morphological dilation and binarization on the preprocessed depth image to generate a feature map of the stroke thickness of the target paper.

[0030] For example, based on the processed RGB image and depth image, depth information mapping is performed on the processed depth image, and handwriting image segmentation is performed on the processed RGB image. This segmentation serves as feature data for fusing the depth information of the handwriting and pen marks. In the processed depth image, when the pen marks create indentations on the paper surface (i.e., the deeper the pen marks), the depth value of the pen marks is negative. Therefore, depth information can represent the darkness and thickness of the pen marks. Using a linear mapping method, a grayscale image is created, mapping the depth values ​​to the darkness of the pen marks; areas with smaller depth values ​​are darker, indicating darker pen marks. Assume... The depth value in the depth image. For the corresponding grayscale value, This is the minimum depth value in the depth image (usually the deepest part of the handwriting). This represents the maximum depth value in the depth image (the depth of the paper surface). The maximum value of the grayscale value is 255. The following is an algorithm design for a linear mapping to convert depth values ​​to grayscale values.

[0031] , Specifically, the preprocessed RGB image (color image) has been optimized for handwriting outlines through dilation (connecting discontinuous strokes) and erosion (removing small noises), clearly distinguishing the original handwriting area from the background area (such as paper stains or blank areas). The RGB image is then pixel-level aligned with the preprocessed depth image. Since the depth camera captures images simultaneously, the pixel positions of the two correspond one-to-one, clearly defining the pixel range corresponding to the original handwriting area in the RGB image in the depth image. The depth value within this range is extracted. In the preprocessed depth image, the deeper the indentation (the greater the writing pressure), the smaller the corresponding pixel depth value. The depth value is correlated with the grayscale value using a linear formula, so that the smaller the depth value (the deeper the indentation), the smaller the grayscale value (the darker the color in the feature map), intuitively reflecting the depth of the handwriting. The calculated grayscale value of each pixel in the original handwriting area is filled into the corresponding position on a blank canvas of the same size as the RGB image. The background area (the part of the RGB image without handwriting) is uniformly filled with a grayscale value of 255 (white), ultimately generating a single-channel grayscale image, i.e., the handwriting depth feature map. The embodiments of the present invention eliminate the interference of background region depth values, which can ensure that subsequent depth value mapping is only for real handwriting and avoid invalid depth data of the background region (such as minor unevenness on the paper surface) from affecting the accuracy of the feature map.

[0032] More specifically, step S23 includes: S231, Calculate the local mean of each pixel in the preprocessed depth image to obtain the mean feature map; S232, perform morphological dilation on the mean feature map, and binarize the grayscale histogram of the dilated mean feature map to obtain the handwriting thickness feature map of the target paper.

[0033] For example, local mean calculation: The local mean of each pixel is calculated using a Gaussian weighted average method to obtain a mean feature map. Gaussian weighting can better take into account the depth information around the pixel, thus obtaining more accurate handwriting thickness features.

[0034] Morphological dilation: Apply morphological dilation to the mean feature map; dilation can connect broken stroke parts and expand the stroke area, making the stroke thickness features more obvious; choose a small square matrix as the structuring element to determine the range and shape of the dilation operation.

[0035] Binarization: The Otsu method is used to binarize the dilated mean feature map. A histogram is calculated on the dilated mean feature map. The optimal threshold is automatically selected based on the statistical characteristics of the image content to distinguish between handwriting and background, resulting in a binarized handwriting thickness feature map. Each pixel value can be set to 0 or 255 to represent the thickness of the handwriting.

[0036] Specifically, step S3 includes: S31, determine the handwriting region of the handwriting thickness feature map based on the pixel values ​​of all pixels in the handwriting thickness feature map; S32, traverse the target coordinates of each pixel in the handwriting region, and calculate the local mean of the gray values ​​of the corresponding target coordinates in the handwriting depth feature map; S33, the calculated average gray value is used as the gray value of the corresponding target coordinate in the handwriting depth feature map to obtain the handwriting feature map of the target paper.

[0037] For example, calculating the local mean of a handwriting depth feature map: using a mask to supplement pixel values, such as... Figure 3 As shown, Figure 3 This invention provides a handwriting thickness feature map. First, the handwriting thickness feature map is used as a mask image. In the image, the handwriting area is 255 (white), and the non-handwriting area is 0 (dark). Features are then traversed. Figure 1 In the (handwriting thickness feature map), for each pixel, when a pixel with a value of 255 is encountered, its position in the feature map is determined. Figure 2 The corresponding position in the (handwriting depth feature map). In the feature... Figure 2 In this context, a local region (e.g., a 5x5 window) is defined around the given location, and the average grayscale value of all pixels within that local region is calculated.

[0038] Fill the mask area with local mean: Assign the calculated grayscale mean to the feature. Figure 2 The pixels (i.e., features) corresponding to the middle map region Figure 1 (The pixel position with a median value of 255). Finally, the fused feature map (handwriting feature map) is obtained, which combines the features of handwriting depth and the features of the thickness of the handwriting area.

[0039] It's worth noting that the handwriting thickness feature map is a binary image (pixel values ​​are only 0 or 255). A pixel value of 255 indicates that the location belongs to the handwriting area, which is the core area for subsequent depth information calculation; a pixel value of 0 indicates that the location is a non-handwriting area and does not participate in the depth information fusion calculation. The handwriting depth feature map is a grayscale image (pixel value range 0-255). The smaller the pixel value, the smaller the corresponding depth value (the deeper the handwriting indentation), which visually appears as a deeper handwriting; the larger the pixel value, the larger the corresponding depth value (the shallower the handwriting indentation or no indentation), which visually appears as a shallower handwriting or no handwriting. Both types of feature maps are generated based on the same paper photographed by the same high-precision depth camera, and image calibration ensures that the pixel positions correspond perfectly (i.e., feature maps are not directly related to the image data). Figure 1 A certain coordinate pixels, and features Figure 2 median coordinate The pixels correspond to the same physical location on the paper.

[0040] More specifically, step S32 includes: S321, For the target coordinates of each pixel in the handwriting region, a local calculation window of fixed size is defined in the handwriting depth feature map with the target coordinates as the center; S322, calculate the average gray value of all pixels within the local calculation window to obtain the average gray value of the target coordinates.

[0041] For example, the target coordinates of each pixel in the handwriting region In the handwriting depth feature map, the target coordinates Centered on the target, a fixed-size local calculation window is defined (e.g., 5×5, which can be adjusted according to the handwriting density; a window that is too small is easily affected by noise, while a window that is too large blurs details). The grayscale values ​​(i.e., light and dark feature values) of all pixels within this window are counted, and the arithmetic mean (grayscale mean) is calculated. This mean reflects the target coordinates. The average depth of the surrounding strokes is considered to avoid the impact of outliers in a single pixel (such as depth measurement errors) on the fusion result.

[0042] More specifically, step S33 includes: S331, the calculated average gray value is used as the gray value of the corresponding target coordinate in the handwriting depth feature map; S332, keeping the remaining gray values ​​in the handwriting depth feature map unchanged, to obtain the handwriting feature map of the target paper.

[0043] For example, the calculated grayscale mean is assigned to the target coordinates in the handwriting density feature map. The pixels are replaced with the original single depth value (original grayscale value) at that location; the coordinates of all pixels in the handwriting region are processed one by one to update the grayscale value of all effective fusion areas (handwriting regions); for the coordinates of the pixel value of 0 in the handwriting thickness feature map (non-handwriting region), the original grayscale value of that location in the handwriting depth feature map remains unchanged, thus obtaining the handwriting feature map of the target paper, ensuring that the information of the non-handwriting region is not interfered with; the handwriting feature map clearly identifies the handwriting region as handwriting (region 255 of the handwriting thickness feature map) and clearly identifies the depth of the handwriting region (local mean of the handwriting depth feature map). Through local mean calculation, small noises in the handwriting depth feature map (such as depth errors caused by small unevenness of the paper surface) are smoothed out, and at the same time, the interference of non-handwriting regions is eliminated through mask filtering, improving anti-interference.

[0044] Specifically, step S4 includes: S41, perform handwriting image segmentation on the processed color image to obtain the handwriting segmentation image of the target paper; S42, the handwriting feature map and the handwriting segmentation image are fused through logical operations; S43, fill in the missing handwriting in the fused image to generate a handwriting restoration image of the target paper.

[0045] For example, a deep learning-based image segmentation method is used to segment the handwriting portion of the original RGB image of paper. For instance, a handwriting dataset is pre-collected to construct and train a handwriting segmentation algorithm model. The handwriting segmentation algorithm model performs handwriting image segmentation on the processed color image to obtain an image containing only handwriting, which is then converted to grayscale. The non-handwriting portions are filled with blank pixel values ​​to eliminate other background interference factors, resulting in a handwriting segmentation image. Logical fusion and weighted fusion operations are used to fuse the handwriting feature map and the handwriting segmentation image. For example, logical operations are performed on the handwriting feature map and the handwriting segmentation image to ensure that the handwriting portions in the handwriting segmentation image remain unchanged, preserving the original handwriting edges and the interior of the handwriting. A linear scaling diffusion method is used to fill in the original handwriting edges up to the edges of the handwriting feature map to generate a handwriting restoration image of the target paper.

[0046] More specifically, step S42 includes: S421, Perform an OR logical operation on the pixel values ​​of all corresponding pixels in the handwriting feature map and the handwriting segmentation image; S422 combines the OR operation results of all corresponding pixels to form a binary image as the fused image.

[0047] For example, an OR logical operation is performed on the handwriting feature map and the handwriting segmentation image to retain the handwriting region in the handwriting segmentation image and add the region covered by the handwriting feature map. For instance, an OR logical operation is performed on the pixel values ​​of all corresponding coordinates in the two types of images (note: a pixel value of 0 indicates no information / background, i.e., a non-handwriting region, and a pixel value of 255 indicates valid information / handwriting, i.e., a logical mapping of the handwriting region). The OR operation results of all corresponding pixels are combined to form a binary image (pixel value 0 or 255) as the fused image. This fused image clearly identifies the original handwriting region that needs to be retained and the potential handwriting region that needs to be filled with features, excludes irrelevant background regions, and can more comprehensively reflect the true state of the handwriting, improving the accuracy of handwriting restoration.

[0048] More specifically, the step of performing an OR logical operation on the pixel values ​​of all corresponding pixels in the handwriting feature map and the handwriting segmentation image includes: An OR operation is performed on the pixels in the handwriting segmentation image whose pixel value corresponds to the handwriting pixel threshold and the pixels in the handwriting feature image whose pixel value does not correspond to the baseline pixel threshold. Wherein, the handwriting pixel threshold is the pixel value of the pixel corresponding to the original handwriting region in the handwriting segmentation image; the reference pixel threshold is the pixel value of the pixel corresponding to the non-handwriting region in the handwriting feature map.

[0049] For example, the handwriting pixel threshold can be set to 255, and the baseline pixel threshold can be set to 0. If a certain coordinate in the handwriting segmentation image... The pixel value is 255 (original handwriting area), regardless of the coordinate in the handwriting feature map. Regardless of the pixel value, the result of the coordinate calculation is always 255 (handwriting region); if the handwriting segmentation image contains... The pixel value is 0 (originally no handwriting), but this location in the handwriting feature map... If the pixel value is not 0 (a potential handwriting area with indentation features), the calculated result of this coordinate is 255 (included in the handwriting area as the range of missing handwriting to be filled); if in both types of images The pixel values ​​are all 0 (no original handwriting and no indentation features), and the result of the calculation at this coordinate is 0 (background area, not involved in subsequent filling).

[0050] More specifically, S43 includes: S431, Extract the handwriting edges from the handwriting segmentation image and create an edge mask based on the handwriting edges; S432, Define the diffusion width of the edge mask, and create a diffusion region mask based on the edge mask and the diffusion width; S433, Generate a handwriting restoration image of the target paper based on the missing handwriting in the image after the diffusion area mask is filled and fused, according to the handwriting edge, the diffusion width, and the diffusion area mask.

[0051] For example, the Canny edge detection algorithm is used to obtain the handwriting edges in the handwriting segmentation image to create an edge mask, where edge pixel values ​​are 1 and non-edge pixel values ​​are 0. A diffusion width for the edge mask is defined (e.g., 5-10 pixels, adjusted according to the thickness of the handwriting). This diffusion width determines the range of pixel values ​​that will be proportionally filled into the blended image starting from the handwriting edge. Based on the edge mask and diffusion width, a diffusion region mask is created, which smoothly transitions from 1 to 0 near the edge. Using the edge mask as a base, the diffusion region is expanded outwards from the edge pixels (the original area without handwriting but with features), forming a diffusion region. Pixel values ​​within the diffusion region smoothly transition from 1 (near the edge) to 0 (far from the edge), generating an edge diffusion mask. The closer a pixel value is to 1 within the diffusion region, the closer the location is to the original handwriting, requiring more preservation of the original handwriting features; the closer a pixel value is to 0, the farther the location is from the original handwriting, requiring more addition of indentation features. For each pixel within the diffusion region, its distance to the handwriting edge is calculated, and the filling ratio is determined based on this distance. The pixel values ​​of the diffusion region are updated using the diffusion region mask and the calculated pixel values. For the non-diffusion region (i.e., the region inside the original handwriting), the pixel values ​​of the original handwriting segmentation map are kept unchanged to obtain the handwriting restoration image of the target paper.

[0052] More specifically, step S433 includes: S4331, Determine the missing handwriting area of ​​the fused image based on the diffusion area mask; S4332, Calculate the pixel distance from each pixel in the missing handwriting region to the edge of the handwriting; S4333, calculate the fill pixel value of each pixel in the missing handwriting area using a linear proportional formula based on the pixel distance and the diffusion width; S4334, Based on the diffusion area mask and the filling pixel values, fill in the pixel values ​​of the missing handwriting area to generate the handwriting restoration image of the target paper.

[0053] For example, the missing handwriting region (diffusion region) of the fused image is determined according to the diffusion region mask, the pixel distance from each pixel in the missing handwriting region to the edge of the handwriting is calculated, the filling pixel value of each pixel in the missing handwriting region is calculated using a linear ratio formula based on the pixel distance and the diffusion width, and the pixel values ​​of the missing handwriting region are filled according to the diffusion region mask and the filling pixel values ​​to generate the handwriting restoration image of the target paper. For example, for the original handwriting interior area (where the pixel value in the edge diffusion mask is 0 and the pixel value in the handwriting segmentation image is 255), the original pixel values ​​of the handwriting segmentation image are directly retained without filling (to avoid destroying the clear original handwriting); for the edge diffusion area (where the pixel value in the edge diffusion mask is between 0 and 1), the filling pixel value of each pixel in the missing handwriting area is calculated using a linear scaling formula, and the pixel value of the missing handwriting area is updated to achieve a smooth transition from handwriting edge to feature supplementation; for the background area (where the pixel value in the fused image is 0), the pixel value is kept at 0, no processing is performed, and the pixel values ​​of all areas are integrated to form a complete handwriting restoration image, ensuring that the original handwriting is clearly preserved, the missing handwriting is accurately filled, and the transition area is natural and without discontinuity. For example, the original pixel values ​​of the inner area of ​​the original handwriting, the calculated pixel values ​​of the diffusion area, and the blank pixel values ​​of the background area are integrated one by one according to their coordinates to form a complete grayscale image. The integrated image is then slightly grayscale equalized to adjust the overall contrast, ensuring that the restored handwriting has consistent depth and clear edges, and avoiding uneven local brightness caused by feature fusion. Finally, a handwriting restoration image of the target paper is obtained, which not only preserves the true shape of the original handwriting, but also fills in the faded and disappeared missing handwriting through the indentation feature, realizing the restoration and reconstruction of handwriting on old paper.

[0054] The linear proportion formula is as follows: , In the formula, The fill pixel value for each pixel in the missing handwriting area; The ratio of the pixel distance to the diffusion width; Segment the edge pixel values ​​of the handwriting image. This represents the pixel value at the corresponding coordinates of the merged image.

[0055] This invention uses an edge mask to divide the area and fills in only the missing areas at the edges of the original handwriting, completely preserving the original clear handwriting and avoiding information distortion caused by restoration. Linear proportional filling ensures that the gray values ​​of the filled area and the edges of the original handwriting gradually change without obvious discontinuities, resulting in strong visual continuity of the restored handwriting. Filling is based on the indentation features (depth and thickness) obtained by a depth camera, and the filled content conforms to the physical traces of the original writing (such as the changes in depth caused by writing pressure), improving the accuracy of restoration.

[0056] This invention discloses a paper handwriting restoration method. The method involves acquiring a color image and a depth image of a target paper, preprocessing the color and depth images, generating a handwriting depth feature map and a handwriting thickness feature map based on the preprocessed color and depth images, fusing the handwriting depth and thickness feature maps to obtain a handwriting feature map of the target paper, and fusing the handwriting feature map with a handwriting segmentation image to fill in missing handwriting in the fused image, thereby generating a restored handwriting image of the target paper. This method can obtain the depth information of handwriting indentations (handwriting depth and thickness feature maps) on the paper based on the color and depth images, and use this indentation information to restore faded or disappeared handwriting. It achieves restoration direction calibration based on physical indentation information, avoiding misjudgments caused by relying solely on pixel information. Furthermore, the handwriting indentations, as physically preserved information, are unaffected by paper fading or surface stains and can supplement the missing handwriting structure information in the original image, improving the accuracy of paper handwriting restoration.

[0057] See Figure 4 , Figure 4 This is a schematic diagram of the structure of a paper handwriting repair device 10 provided in an embodiment of the present invention. The paper handwriting repair device 10 includes: The paper image processing module 11 is used to acquire color images and depth images of the target paper and to preprocess the color images and depth images. The handwriting feature generation module 12 is used to generate a handwriting depth feature map and a handwriting thickness feature map of the target paper based on the preprocessed color image and depth image. The handwriting feature fusion module 13 is used to fuse the handwriting depth feature map and the handwriting thickness feature map to obtain the handwriting feature map of the target paper; The handwriting image restoration module 14 is used to fuse the handwriting feature map and the handwriting segmentation image, fill in the missing handwriting in the fused image, and generate the handwriting restoration image of the target paper.

[0058] The paper handwriting repair device 10 provided in this embodiment of the invention can realize all the processes of the paper handwriting repair method of the above embodiment. The functions and technical effects of each module in the device are the same as the functions and technical effects of the paper handwriting repair method of the above embodiment, and will not be repeated here.

[0059] See Figure 5 , Figure 5This is a schematic diagram of the structure of a paper handwriting repair device 20 provided in an embodiment of the present invention. The paper handwriting repair device 20 of this embodiment includes: a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the above-described paper handwriting repair method embodiment. Alternatively, when the processor 21 executes the computer program, it implements the functions of each module in the above-described paper handwriting repair device embodiment.

[0060] For example, the computer program may be divided into one or more modules, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the paper handwriting restoration device 20.

[0061] The paper handwriting restoration device 20 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The paper handwriting restoration device 20 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the paper handwriting restoration device 20 and does not constitute a limitation on the device. It may include more or fewer components than shown, or combine certain components, or use different components. For example, the paper handwriting restoration device 20 may also include input / output devices, network access devices, buses, etc.

[0062] The processor 21 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The processor 21 is the control center of the paper handwriting restoration device 20, connecting all parts of the device via various interfaces and lines.

[0063] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the paper handwriting restoration device 20 by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0064] If the integrated module of the paper handwriting restoration device 20 is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable medium may be appropriately added to or subtracted from the content as required by the legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium may not include electrical carrier signals and telecommunication signals.

[0065] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0066] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the paper handwriting repair method as described above.

[0067] Furthermore, embodiments of the present invention also provide a computer program product, which is stored in a storage medium and executed by at least one processor to implement the steps of the paper handwriting restoration method described above.

[0068] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for restoring handwriting on paper, characterized in that, include: Acquire color and depth images of the target paper, and preprocess the color and depth images; Based on the preprocessed color image and depth image, generate a feature map of the depth and thickness of the handwriting on the target paper. The handwriting depth feature map and the handwriting thickness feature map are fused together to obtain the handwriting feature map of the target paper; The handwriting feature map and the handwriting segmentation image are fused together to fill in the missing handwriting in the fused image, thereby generating a handwriting restoration image of the target paper.

2. The paper handwriting restoration method as described in claim 1, characterized in that, The step of generating a handwriting depth feature map and a handwriting thickness feature map of the target paper based on the preprocessed color image and depth image includes: The depth value corresponding to the original handwriting region is extracted from the preprocessed depth image based on the original handwriting region of the preprocessed color image. Linear mapping is performed on the depth values ​​to generate a feature map of the pen stroke depth on the target paper; The preprocessed depth image is subjected to mean calculation, morphological dilation, and binarization to generate a feature map of the stroke thickness on the target paper.

3. The paper handwriting restoration method as described in claim 2, characterized in that, The step of performing mean calculation, morphological dilation, and binarization on the preprocessed depth image to generate a handwriting thickness feature map of the target paper includes: Calculate the local mean of each pixel in the preprocessed depth image to obtain the mean feature map; A morphological dilation operation is performed on the mean feature map, and the grayscale histogram of the dilated mean feature map is binarized to obtain the handwriting thickness feature map of the target paper.

4. The paper handwriting restoration method as described in claim 1, characterized in that, The step of fusing the handwriting depth feature map and the handwriting thickness feature map to obtain the handwriting feature map of the target paper includes: The handwriting region of the handwriting thickness feature map is determined based on the pixel values ​​of all pixels in the handwriting thickness feature map. Traverse the target coordinates of each pixel in the handwriting region, and calculate the local mean of the gray values ​​of the corresponding target coordinates in the handwriting depth feature map; The calculated mean gray value is used as the gray value of the corresponding target coordinate in the handwriting depth feature map to obtain the handwriting feature map of the target paper.

5. The paper handwriting restoration method as described in claim 4, characterized in that, The step of traversing the target coordinates of each pixel in the handwriting region and calculating the local mean of the gray values ​​corresponding to the target coordinates in the handwriting depth feature map includes: For the target coordinates of each pixel in the handwriting region, a local calculation window of fixed size is defined in the handwriting depth feature map, centered on the target coordinates. The mean grayscale value of the target coordinates is obtained by calculating the mean grayscale value of all pixels within the local calculation window.

6. The paper handwriting restoration method as described in claim 4, characterized in that, The step of using the calculated grayscale mean as the grayscale value of the corresponding target coordinate in the handwriting depth feature map to obtain the handwriting feature map of the target paper includes: The calculated mean gray value is used as the gray value of the corresponding target coordinate in the handwriting depth feature map; Keeping the remaining gray values ​​in the handwriting depth feature map unchanged, the handwriting feature map of the target paper is obtained.

7. The paper handwriting restoration method as described in claim 1, characterized in that, The step of fusing the handwriting feature map and the handwriting segmentation image, filling in the missing handwriting in the fused image, and generating the handwriting restoration image of the target paper includes: The handwriting image of the target paper is obtained by performing handwriting image segmentation on the processed color image; The handwriting feature map and the handwriting segmentation image are fused through logical operations; Fill in the missing strokes in the fused image to generate a stroke restoration image of the target paper.

8. The paper handwriting restoration method as described in claim 7, characterized in that, The step of fusing the handwriting feature map and the handwriting segmentation image through logical operations includes: Perform an OR logical operation on the pixel values ​​of all corresponding pixels in the handwriting feature map and the handwriting segmentation image; The OR operation results of all corresponding pixels are combined to form a binary image as the fused image.

9. The paper handwriting restoration method as described in claim 8, characterized in that, The step of performing an OR logical operation on the pixel values ​​of all corresponding pixels in the handwriting feature map and the handwriting segmentation image includes: An OR operation is performed on the pixels in the handwriting segmentation image whose pixel value corresponds to the handwriting pixel threshold and the pixels in the handwriting feature image whose pixel value does not correspond to the baseline pixel threshold. Wherein, the handwriting pixel threshold is the pixel value of the pixel corresponding to the original handwriting region in the handwriting segmentation image; the reference pixel threshold is the pixel value of the pixel corresponding to the non-handwriting region in the handwriting feature map.

10. The paper handwriting restoration method as described in claim 7, characterized in that, The process of filling in the missing handwriting in the fused image to generate a handwriting restoration image of the target paper includes: The handwriting segmentation image is subjected to handwriting edge extraction, and an edge mask is created based on the handwriting edges; Define the diffusion width of the edge mask, and create a diffusion region mask based on the edge mask and the diffusion width; A handwriting restoration image of the target paper is generated based on the missing handwriting in the merged image using the handwriting edges, the diffusion width, and the diffusion area mask.

11. The paper handwriting restoration method as described in claim 10, characterized in that, The step of generating a handwriting restoration image for the target paper by filling in missing handwriting in the blended image based on the handwriting edges, the diffusion width, and the diffusion area mask includes: The missing handwriting area in the fused image is determined based on the diffusion area mask; Calculate the pixel distance from each pixel in the missing handwriting region to the edge of the handwriting; The fill pixel value of each pixel in the missing handwriting area is calculated using a linear proportional formula based on the pixel distance and the diffusion width. A handwriting restoration image of the target paper is generated by filling the missing handwriting area with pixel values ​​based on the diffusion area mask and the filling pixel values.

12. A paper handwriting restoration device, characterized in that, include: The paper image processing module is used to acquire color and depth images of the target paper and to preprocess the color and depth images. The handwriting feature generation module is used to generate a handwriting depth feature map and a handwriting thickness feature map of the target paper based on the preprocessed color image and depth image. The handwriting feature fusion module is used to fuse the handwriting depth feature map and the handwriting thickness feature map to obtain the handwriting feature map of the target paper; The handwriting image restoration module is used to fuse the handwriting feature map and the handwriting segmentation image, fill in the missing handwriting in the fused image, and generate the handwriting restoration image of the target paper.

13. A paper handwriting restoration device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the paper handwriting restoration method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform the paper handwriting restoration method as described in any one of claims 1-11.

15. A computer program product, characterized in that, The computer program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the paper handwriting restoration method as described in any one of claims 1-11.

Images