Handwritten document reconstruction and seal completion method and system based on multi-modal fusion

Abstract

The application provides a handwriting document reconstruction and seal completion method and system based on multi-modal fusion, relates to the technical field of multi-modal fusion, and extracts a primary state physical layer, a primary state structure layer and a primary state semantic layer through layering; constructs a structure boundary constraint graph and a semantic shape constraint graph; introduces the structure boundary constraint graph and the semantic shape constraint graph to repair a damaged area of the primary state physical layer based on the primary state physical layer, and generates a refined physical layer; reversely extracts new geometric elements by using the refined physical layer, optimizes the primary state structure layer, and generates an optimized structure layer; uses residual positioning rules to mark residual areas based on the primary state semantic layer and the refined physical layer; performs local image enhancement and a three-level cascading recognition strategy, and generates a corrected semantic layer; obtains a visual enhancement base map based on the refined physical layer; and generates an enhanced handwriting document image in combination with the optimized structure layer and the corrected semantic layer, thereby improving the integrity, accuracy and visual consistency of handwriting document reconstruction and seal completion.

Description

Technical Field

[0001] This invention relates to the field of multimodal fusion technology, specifically to a method and system for handwritten document reconstruction and stamp completion based on multimodal fusion. Background Technology

[0002] Based on the hierarchical extraction concept, this method decomposes handwritten documents into physical, structural, and semantic layers, respectively processing pixel-level material classification, geometric vector elements, and textual content information. A bidirectional interaction and constraint guidance mechanism is established between layers to achieve unified optimization of damage repair and content reconstruction. Currently, research on improving the quality of handwritten document reconstruction mainly focuses on two independent directions: low-level image restoration and high-level character recognition. One emphasizes pixel-level image enhancement and texture synthesis, while the other focuses on optical character recognition and semantic understanding optimization. However, practical applications face the following challenges: Existing methods for processing handwritten documents mix various information such as text, seals, paper textures, and damage marks in the same image space for unified processing, leading to mutual interference and difficulty in separating different information. Furthermore, the serial, pipeline-like unidirectional processing flow amplifies minor errors in earlier steps and makes backtracking correction impossible. Throughout the process, there is a lack of bidirectional collaboration between high-level semantic information and low-level pixel information; high-level information such as text semantics and seal content cannot effectively guide low-level image restoration, and low-level restoration results cannot provide feedback to correct high-level inferences. In addition, a fixed, uniform processing strategy is used for all regions, failing to adaptively adjust based on the characteristics of different regions such as text areas, seal areas, and background areas. Ultimately, this results in information loss, error accumulation, logical contradictions, and inconsistent effects in the reconstruction results, making it difficult to balance completeness and accuracy. Summary of the Invention

[0003] To achieve the above objectives, the present invention provides the following technical solution: a method for handwritten document reconstruction and stamp completion based on multimodal fusion, the method comprising: Step S1: Obtain the handwritten document image; extract the initial physical layer, initial structural layer, and initial semantic layer through layering; and construct the structural boundary constraint graph and semantic shape constraint graph. Step S2: Based on the initial physical layer, introduce structural boundary constraint diagram and semantic shape constraint diagram to repair the damaged area of ​​the initial physical layer and generate the refined physical layer; Step S3: Extract new geometric elements by reverse engineering the refined physical layer, and optimize the initial structure layer through consistency comparison and fusion strategies to generate the optimized structure layer; Step S4: Based on the initial semantic layer and the refined physical layer, residual regions are marked using residual localization rules; local image enhancement and a three-level cascaded recognition strategy are applied to the residual regions to generate a corrected semantic layer. Step S5: Obtain a visually enhanced base map based on the refined physical layer; and combine it with the optimized structural layer and the corrected semantic layer to generate an enhanced handwritten document image.

[0004] Furthermore, the step of extracting the initial state physical layer, initial state structural layer, and initial state semantic layer through layering includes: Based on the handwritten document image, a preliminary color classification map is generated, and the pixel color purity and edge sharpness are calculated. For the preliminary color classification map, the probability distribution of pixels belonging to various substances is obtained, and layered masks for various substances are generated. Combining the probability distribution of pixels belonging to various substances, color purity, and edge sharpness, the initial confidence of pixels is generated, and the physical layer confidence map is obtained. The initial physical layer is constructed. For handwritten document images, geometric elements are extracted and geometric vector coordinates are generated; the initial geometric confidence of the geometric elements is calculated to obtain the structural layer confidence map; and the initial state structural layer is constructed. For handwritten document images, the system locates text regions and outputs text border coordinates; for text regions, it obtains text sequences, text units, and the recognition probability of text units; for text border coordinates, it obtains paragraph structure information; for specific elements in the text border coordinates, it obtains specific text sequences and specific text confidence scores; based on the text sequences, the recognition probabilities of text units, and the paragraph structure information, it obtains the confidence score of text units; according to the text unit confidence scores, it divides text units into high-confidence regions, low-confidence regions, and regions to be completed; and it constructs an initial semantic layer.

[0005] Furthermore, the construction of the structural boundary constraint graph and semantic shape constraint graph includes: The geometric vector data is converted into a geometric pixel-level constraint mask, and the pixels within the mask are marked. Based on the geometric pixel-level constraint mask and geometric confidence, the pixel constraint strength is constructed according to the spatial relationship between the pixels and geometric elements. The constructed pixel constraint strength is stored in the geometric channel according to the geometric elements to form a multi-channel structural boundary constraint map. For the initial semantic layer, glyph templates are extracted; the extracted glyph templates are converted into pixel-level constraint maps; and constraint strength is constructed based on the confidence of character units. A semantic shape constraint map is generated based on pixel-level constraint maps and constraint strengths.

[0006] Furthermore, based on the initial physical layer, structural boundary constraint graphs and semantic shape constraint graphs are introduced to repair the damaged areas of the initial physical layer, generating a refined physical layer, including: For the damaged areas in the initial physical layer, the mask identifies the areas to be repaired. For the pixels in the areas to be repaired, the structural boundary constraint map and semantic shape constraint map are called. The constraint strength is used to repair the pixels in the areas to be repaired, and a refined material layering mask for the initial physical layer is generated. Based on the repaired pixels and their matching degree with the structural boundary constraint map and semantic shape constraint map, the confidence map of the refined initial physical layer is calculated to obtain the confidence map of the refined physical layer; and the refined physical layer is constructed.

[0007] Furthermore, the step of extracting new geometric elements by reverse engineering the refined physical layer and optimizing the initial structure layer through a consistency comparison and fusion strategy to generate an optimized structure layer includes: Based on the refined initial physical layer of various material layer masks, new geometric elements are extracted; The new geometric elements are compared with the geometric elements in the initial structure layer to obtain the comparison results. Based on the comparison results, a fusion strategy is executed to output the optimized geometric vector data of the initial structure layer. The confidence of the geometric elements in the initial structure layer is updated by comprehensively considering three factors: fitting quality, consistency degree, and physical support. The updated confidence map of the initial structure layer is then output. The optimized structure layer is composed of the optimized geometric vector data of the initial structure layer and the updated confidence map of the initial structure layer.

[0008] Furthermore, the fusion strategy includes: Existence check: If the new geometric element has a corresponding term in the initial state structure layer, then proceed to the deviation degree check; If a new geometric element does not have a corresponding term in the initial state structure layer, it is determined to be a newly added geometric element, which is then added directly and assigned an initial confidence level based on the fitting quality, and marked as newly added. If a geometric element in the initial state structure layer has no corresponding item in the new geometric element, it is determined that a geometric element is missing. The geometric element is retained, the initial confidence of the geometry in the initial state structure layer is adjusted, and it is marked as needing to be reviewed. Deviation degree judgment: Compare the deviation of the key parameters in the new geometric element and the initial state structure layer geometric element with the preset deviation threshold. If the deviation of the key parameters in the new geometric element and the geometric element is less than the preset deviation threshold, it is judged to be completely consistent, the geometric element is retained, the initial confidence of the geometry in the initial state structure layer is adjusted, and it is marked as consistent with the old and new. If the deviation is greater than or equal to the preset deviation threshold but less than the maximum allowable threshold, it is considered an acceptable deviation. The fitting quality is then compared further. If the fitting quality of the new geometric element is better than that of the geometric element, the new geometric element replaces the original geometric element. The confidence level is set based on the fitting quality of the new geometric element, and the result is marked as optimized. If the fitting quality of the geometric element is better than or equal to that of the new geometric element, the geometric element is retained. The initial confidence level of the geometry in the initial state structure layer is adjusted based on the fitting quality of the geometric element, and the result is marked as verified. If the value is greater than or equal to the maximum allowable threshold, it is judged as a serious conflict. At the same time, the new geometric element and the geometric element are retained as candidates, the initial confidence of geometry in the initial state structure layer is adjusted, and it is marked as pending confirmation.

[0009] Furthermore, based on the initial semantic layer and the refined physical layer, residual region labeling is performed using residual localization rules; local image enhancement and a three-level cascaded recognition strategy are applied to the residual regions to generate a corrected semantic layer, including: Based on the text unit sequence, text unit confidence, and region type in the initial semantic layer, as well as the text ink mask and confidence map in the refined physical layer, residual localization rules are used to identify text units that need to be re-inferred and mark them as residual regions. For the labeled residual regions, local image enhancement is performed using the text ink mask in the refined physical layer and the confidence map of the refined physical layer. A three-level cascaded recognition strategy is then used to re-recognize the enhanced local image to obtain the preliminary updated text recognition results. The geometric vector data in the optimized structure layer is used to correct the initial updated text recognition results, resulting in structurally aligned text results, and the structural support of each text unit is obtained. Based on the updated character unit recognition probability and structural support in the preliminary updated character recognition results, the final character unit confidence is calculated by weighting; and a corrected semantic layer is constructed.

[0010] Furthermore, the residual positioning rules include: Condition 1: If the confidence level of a text unit is lower than the second reference threshold, it is marked as a region requiring mandatory re-identification. Condition 2: If the confidence level of a text unit is between the second reference threshold and the first reference threshold, it is marked as a confidence level verification area. Condition 3: If the coverage ratio between the refined physical layer text ink mask and the initial semantic layer text bounding box is lower than the standard coverage ratio, it is marked as a region with insufficient spatial matching. Condition 4: If the mean confidence level of the refined physical layer corresponding to the Chinese character region mask in the refined physical layer is lower than the standard mean, it is marked as a region with low physical reliability. A text unit that meets at least one condition is marked as a residual region; a text unit that does not meet any condition is marked as a high-confidence reserved region.

[0011] Furthermore, the visually enhanced base map is obtained based on the refined physical layer; and combined with the optimized structural layer and the corrected semantic layer, an enhanced handwritten document image is generated, including: Based on various material masks in the refined physical layer, pixel-level fusion reconstruction is performed to obtain a visually enhanced base map; the optimized geometric elements of the structural layer are superimposed on the visually enhanced base map, and multi-dimensional consistency verification is performed to obtain the verified structural layer geometric elements. Based on the modified semantic layer, and combined with the visually enhanced base map and verified structural layer geometric elements, an enhanced handwritten document image is obtained.

[0012] A handwritten document reconstruction and stamp completion system based on multimodal fusion, the system comprising: Multimodal decoupling dual-constraint construction engine: acquires handwritten document images; extracts initial state physical layer, initial state structural layer and initial state semantic layer through hierarchical extraction; and constructs structural boundary constraint map and semantic shape constraint map; Constraint-guided intelligent refinement unit for physical layer: Based on the initial physical layer, structural boundary constraint graphs and semantic shape constraint graphs are introduced to repair the damaged areas of the initial physical layer and generate a refined physical layer; Reverse geometric fusion structure optimization unit: New geometric elements are extracted by reversing the physical layer, and the initial structure layer is optimized by consistency comparison and fusion strategy to generate the optimized structure layer; The residual localization semantic depth correction unit: Based on the initial semantic layer and the refined physical layer, residual regions are marked using residual localization rules; local image enhancement and a three-level cascaded recognition strategy are applied to the residual regions to generate the corrected semantic layer; Multidimensional fusion visual enhancement and reconstruction unit: Based on the refined physical layer, a visual enhancement base map is obtained; and combined with the optimized structural layer and the corrected semantic layer, an enhanced handwritten document image is generated.

[0013] This invention provides a method and system for handwritten document reconstruction and stamp completion based on multimodal fusion. It has the following beneficial effects: This invention, through layered processing and refinement at the physical, structural, and semantic levels, can accurately reconstruct the paper texture, ink marks, seal ink, and damaged areas of handwritten documents, significantly improving image readability and visual integrity. Simultaneously, combined with local enhancement technology, it can recover incomplete text and seal information in complex scenes, reducing misidentification rates and improving the accuracy and reliability of text and seal information. This provides a solid technical guarantee for applications such as document digitization, archival preservation, and judicial and cultural relic restoration. By fully leveraging the capabilities of multimodal information fusion, physical texture, geometric structure, and semantic information are mutually constrained and corrected to achieve high-precision restoration of document content. Multi-dimensional consistency verification and structure-semantic feedback mechanisms ensure the consistency of reconstruction results at the pixel, shape, and textual logic levels, improving the robustness of restoration and recognition. Meanwhile, dynamic weight updates based on confidence and fitting quality enable the method to adapt to different types of handwritten documents, different seal styles, and degrees of damage, avoiding the limitations of single-modal restoration methods and enhancing the overall versatility and reliability of the method. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the steps of the handwritten document reconstruction and stamp completion method based on multimodal fusion of the present invention. Figure 2 This is a flowchart of the reverse geometry fusion structure optimization unit of the present invention; Figure 3 This is an architecture diagram of the handwritten document reconstruction and seal completion system based on multimodal fusion of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] like Figures 1 to 2 As shown, a method for handwritten document reconstruction and stamp completion based on multimodal fusion is presented, which includes: Step S1: Obtain the handwritten document image; extract the initial physical layer, initial structural layer, and initial semantic layer through layering; and construct the structural boundary constraint graph and semantic shape constraint graph. Step S2: Based on the initial physical layer, introduce structural boundary constraint diagram and semantic shape constraint diagram to repair the damaged area of ​​the initial physical layer and generate the refined physical layer; Step S3: Extract new geometric elements by reverse engineering the refined physical layer, and optimize the initial structure layer through consistency comparison and fusion strategies to generate the optimized structure layer; Step S4: Based on the initial semantic layer and the refined physical layer, residual regions are marked using residual localization rules; local image enhancement and a three-level cascaded recognition strategy are applied to the residual regions to generate a corrected semantic layer. Step S5: Obtain a visually enhanced base map based on the refined physical layer; and combine it with the optimized structural layer and the corrected semantic layer to generate an enhanced handwritten document image.

[0017] Specific methods for obtaining images of handwritten documents include: Acquire original handwritten document images, including single images and multispectral images; among them, single images are in common formats such as PNG, TIFF, and JPEG, with a resolution of 300 DPI (print resolution unit), acquired by scanning with a flatbed scanner or taking pictures with a high-speed document camera, and the color mode is RGB or grayscale, with a bit depth of 8 bits or more. Multispectral images are multi-channel images containing visible light, infrared, and ultraviolet spectral bands, with three or more spectral bands, acquired through a multispectral scanner or special acquisition equipment; The original handwritten document image file includes EXIF ​​information, which contains the acquisition time and device parameters; The original handwritten document images are preprocessed, including denoising, color correction, and geometric correction, to obtain the final handwritten document image. Specifically, non-local mean denoising or bilateral filtering is used to eliminate noise generated during the acquisition process. White balance and color space standardization are performed according to the acquisition device parameters. Perspective distortion correction is performed to ensure that the page boundaries remain parallel to the boundaries of the acquired handwritten document image (eliminating distortion caused by incorrect scanning or shooting angles or tilted paper placement). By standardizing image acquisition parameters and preprocessing procedures, we ensure that input images have uniform high resolution, standardized colors, and geometric correction, providing a reliable data foundation for subsequent layered extraction and accurate restoration.

[0018] The initial state physical layer, initial state structural layer, and initial state semantic layer are extracted hierarchically, including: Extraction of the initial physical layer: Based on the handwritten document image (RGB image), a color space conversion is performed to convert the RGB color space to Lab or HSV color space, resulting in the converted handwritten document image. Based on the converted handwritten document image, cluster analysis (such as using K-means clustering) is performed on the pixel colors to generate a preliminary color classification map. The number of clusters in the cluster analysis is determined according to the specific handwritten document image, and is generally greater than or equal to 2, because the handwritten document image includes at least the ink color and the paper background color. The specific process of K-means clustering is as follows: After converting the handwritten document image into a color space, the color features of the pixel points are extracted (such as the saturation component in the HSV color space, or the a* and b* components in the Lab color space). The cluster centers are initialized through the K-means++ algorithm, and the optimal number of clusters K (usually K≥2) is determined before clustering based on prior knowledge or silhouette coefficients. The pixels are iteratively assigned to the nearest centroid and the centroid position is updated until the change in centroid is less than the threshold or the maximum number of iterations is reached. Finally, the pixels are mapped to K color clusters and a preliminary classification map reflecting the distribution of color regions is generated. The initial color classification image and the handwritten document image are stitched together (using a stitching function, such as the torch.cat function in PyTorch), and then input into a deep learning segmentation network (such as U-Net, DeepLab, or SegFormer, which are existing technologies). The output is the probability distribution of pixels belonging to various substances; among them, various substances include paper substrate, text ink, stamp ink, damaged areas, and other elements. Based on the probability distribution of pixels belonging to various substances, layered masks for various substances are generated; among them, layered masks for various substances include paper substrate masks, text ink masks, stamp ink masks, damaged area masks, and masks for other elements. For the converted handwritten document image, the color purity of each pixel is calculated. Specifically, the saturation value is calculated based on the saturation component in the HSV color space, or the ratio of chromaticity value to luminance value is calculated based on the a* and b* components in the Lab color space. The saturation value and the ratio of chromaticity value to luminance value are used as color purity indicators. Based on handwritten document images, edge detection algorithms are used to calculate the edge intensity of pixels, which is used as the pixel edge sharpness. For each pixel, the initial confidence level of each pixel is obtained by weighting and combining its probability distribution of belonging to various substances, color purity, and edge sharpness, thus generating an initial physical layer confidence map. The initial physical layer consists of various material layered masks and an initial physical layer confidence map; The confidence map is essentially a two-dimensional matrix, with each position storing the initial confidence score of the corresponding pixel. Therefore, after calculating the initial confidence scores, these values ​​are directly filled into the two-dimensional matrix to obtain the initial physical layer confidence map; each pixel has an initial confidence score. Among them, the paper substrate mask is used to record the paper texture, background color and aging areas; the text ink mask is used to record the ink distribution of handwritten and printed text; the stamp ink mask is used to record the distribution of red or blue stamp ink; the damaged area mask is used to record stains, tears and missing areas; and the other element mask is used to record other elements such as table lines and binding holes; all are stored in the form of eight-bit grayscale images.

[0019] Extraction of the initial state structure layer: For handwritten document images, extract the edges of the handwritten document images (using the Cannibal operator or the Sobel operator); based on the extracted handwritten document image edges, use a contour finding algorithm to generate a preliminary contour set; for the preliminary contour set, use a multi-source geometric strategy to extract geometric elements; The specific method for extracting geometric elements using a multi-source geometric strategy is as follows: For the initial contour set, the text baseline is fitted by combining projection profile analysis with a random sampling consistency algorithm to output the text baseline line segment; Hough transform is used to identify structural elements, including the seal outline, page boundary, and table border; document distribution algorithm is used to identify layout structures such as paragraphs and tables to generate character bounding boxes; the text baseline line segment, seal outline, page boundary, table border, and character bounding box are the geometric elements; The extracted geometric elements are vectorized (using polygon approximation, curve fitting, or least squares methods) to generate geometric vector data (including geometric type, coordinate parameters, and fitting parameters). For each geometric element, the edge sharpness is determined by the mean edge intensity of the pixels in the region. The fitting error is calculated based on the residual standard deviation between the pixels and the fitted curve (the fitted curve is an ideal geometric model generated by a fitting algorithm; for each pixel, the straight-line distance from it to the fitted curve is calculated, and this distance is called the residual; the standard deviation is calculated from the residuals of all pixels; for example, for the geometric element of a text baseline segment, there are 10 pixels on the text baseline segment. Based on these 10 pixels, a straight-line fitting algorithm (such as the least squares method) is used to obtain the fitted curve. The fitted curve is not equal to any single pixel, but rather an idealized baseline). The edge sharpness and fitting error of the geometric element are weighted and combined (each with a weight of 0.5) to calculate the geometric confidence of each geometric element and generate the initial structural layer confidence map. The initial state structure layer is composed of geometric vector data and an initial structure layer confidence map; wherein, the geometric confidence of each geometric element corresponds one-to-one with the geometric vector data. Text baseline segments are stored as line segments, recording the starting point coordinates, ending point coordinates, and confidence level; stamp outlines are stored as circles, ellipses, or polygons, recording the center point, radius, or vertex sequence, and confidence level; page boundaries are stored as rectangles, recording the coordinates of the four corners and confidence level; table borders are stored as sets of line segments, recording multiple line segments and confidence levels; character bounding boxes are stored as bounding boxes, recording the coordinates of the top left corner, width, height, and confidence level.

[0020] Extraction of the initial state semantic layer: For handwritten document images, a text detection algorithm is used to locate the text regions in the handwritten document image and output the coordinates of the text bounding boxes; for the text regions, an optical character recognition model (such as CRNN or SVTR) is used to recognize the text and output the text unit sequence and the recognition probability of each text unit; Based on the text bounding box coordinates, the line structure is determined by y-coordinate clustering (texts with similar y-coordinates are grouped into the same line), and the paragraph division is determined by line spacing and indentation features (lines with larger line spacing or first-line indentation are used as paragraph boundaries). The paragraph structure information is output (including the total number of paragraphs, the line sequence of each paragraph, and the sequence of text units in each line sequence). Specifically, for specific elements within the text bounding box coordinates, recognition algorithms are employed for processing. For example, for the seal area, a seal detection algorithm is used to locate the seal position, and a seal text recognition algorithm (such as an encoder-decoder architecture based on an attention mechanism) is used to extract the seal content, outputting the seal text unit sequence and seal confidence score. For the table area, a table detection algorithm is used to locate the table position, and a table recognition algorithm is used to extract the table content, outputting the table's row and column structure and table text unit sequence. The algorithms used above are all existing technologies and will not be elaborated upon here. For each text unit sequence, a comprehensive evaluation is performed based on the recognition probability of each text unit and its contextual consistency in the paragraph structure information to calculate the text unit confidence score. The contextual consistency in the paragraph structure information is represented by the inverse of the perplexity of the text unit sequence in the paragraph information calculated by the language model. The lower the perplexity, the stronger the semantic coherence and the higher the contextual consistency. The language model can be an autoregressive language model based on Transformer. The sequence of text units in the paragraph structure is taken as input and fed into the language model (autoregressive language model based on Transformer). The model calculates the conditional probability of each text unit in an autoregressive manner and finally calculates the perplexity of the text unit sequence. Based on the confidence level of the text units, the text units are divided into region types, including high confidence regions, low confidence regions, and regions to be completed. Among them, the high confidence region is the region where the confidence of the text unit is higher than or equal to the first threshold, and the recognition probability is directly adopted; The low-confidence region is the region where the confidence of a character unit is lower than the first threshold but higher than or equal to the second threshold. The top H candidate character units and their corresponding recognition probabilities are retained (the optical character recognition model outputs a list of candidate characters and the probability of each candidate character). Here, H represents the length of the candidate character sequence retained for each character unit. According to empirical methods, the value of H is usually set between 3 and 7. The region to be completed is the region where the confidence level of the text unit is lower than or equal to the second threshold. It is left blank and marked as to be completed. The first threshold and the second threshold can be obtained by empirical method, and the first threshold is greater than the second threshold. The initial semantic layer is constructed from the sequence of text units, the recognition probability of text units, paragraph structure information, text unit confidence, and region type. Each text unit corresponds to a text unit confidence level.

[0021] By decoupling handwritten document images into an initial physical layer, an initial structural layer, and an initial semantic layer, information separation and independent representation are achieved, providing a precise data foundation for targeted repair and collaborative optimization in subsequent steps.

[0022] The steps for constructing a structural boundary constraint diagram are as follows: The geometric vector data is converted into a geometric pixel-level constraint mask, and the pixels within the mask are marked. Specifically, for each geometric element, a corresponding pixel-level constraint mask (with pixel values ​​being non-negative integers) is generated in the image space based on its geometric type and coordinate parameters. For the text baseline line segment, a set of pixels that coincide with the line segment is generated using a straight-line rasterization algorithm and marked as the text baseline constraint mask (pixel is marked as 1). For the seal outline stage, a set of pixels of the outline line and its internal area is generated according to its outline type (circle, ellipse, rectangle), and marked as the seal outline constraint mask (internal pixels are marked as 1, and pixels on the outline are marked as 2). For the page boundary, generate a set of pixels on the rectangular boundary and mark it as the page boundary constraint mask (pixel is marked as 1). For table borders, generate a set of pixels on the horizontal and vertical lines, and mark them as table border constraint masks (pixel marks are 1). Based on the geometric pixel-level constraint mask and geometric confidence, the constraint strength of each pixel is constructed according to the spatial relationship between the pixel and the geometric element. Among them, for linear geometric elements (text baseline segments, page boundaries, table borders), the constraint strength of a pixel is negatively correlated with its distance from the geometric element (the closer the distance, the higher the constraint strength; the farther the distance, the lower the constraint strength). For regional geometric elements (within the outline of a stamp), the constraint strength of the internal pixels is uniformly based on the constraint strength. The geometric confidence level of a geometric element is used as the basic constraint strength (the higher the geometric confidence level, the higher the constraint strength of the geometric element; the lower the geometric confidence level, the lower the constraint strength). The constructed pixel constraint strengths are stored in the geometric channels according to the geometric elements, forming a multi-channel structural boundary constraint map, including text baseline line segment constraint channel, seal outline constraint channel, page boundary constraint channel and table border constraint channel; Here, a constraint strength of zero indicates no constraint, a constraint strength of non-zero indicates the existence of a constraint, and a higher constraint strength indicates a tighter constraint.

[0023] The steps to construct a semantic shape constraint graph are as follows: For text units or seal characters in the initial semantic layer, a font matching algorithm is used to extract glyph templates from the standard font library (which is an existing library). The purpose of glyph template extraction is to obtain shape references for each text unit and seal character that can be used for constraint repair. The standard font library includes a text library and a seal font library. The standard font library includes common printed fonts (such as Song, Kai, and Fangsong) as well as a specific handwritten font sample library for handwritten documents. Specifically, for text units, a glyph template extraction strategy is set based on the confidence level of the text unit. The specific content is as follows: for high confidence areas, the text unit with the highest recognition probability is directly used, and the corresponding glyph template is called from the text library. For low-confidence regions, the glyph templates corresponding to the top H candidate character units are selected. Based on the recognition probability of each candidate character unit, the glyph template corresponding to the character unit with the highest recognition probability is directly selected as a constraint. For seal text, a layout mapping strategy is used to extract character templates. Specifically, based on the geometric type of the seal (circle, ellipse, rectangle) and the seal content, the corresponding character template for the seal text unit is retrieved from the seal font library. The character template is then spatially transformed (translated, rotated, scaled) according to the geometric parameters of the seal outline (center, radius, etc.) to align it with the seal outline. The rotation is based on the center of the seal. The extracted glyph template is converted into a pixel-level constraint map (used to record whether each pixel is subject to shape constraints and the type of constraint; rasterization can be used for this). In the pixel-level constraint map, a pixel with a value of 1 indicates that the pixel is subject to shape constraints; a pixel with a value of 0 indicates that the pixel is not subject to shape constraints. Constraint strength is constructed based on the confidence level of text units; for high-confidence regions, the pixel constraint strength is equal to the text unit confidence level; for low-confidence regions, the pixel constraint strength is the product of the text unit confidence level and the pixel-level fusion weight; for regions to be filled, the pixel constraint strength is 0. A semantic shape constraint map is generated based on the product of the pixel-level constraint map and the constraint strength; where the pixel-level constraint map records whether a pixel is constrained, and the constraint strength is a specific numerical value.

[0024] By converting geometric structural information and textual semantic information into pixel-level constraint maps, precise positional constraints and shape guidance are provided for subsequent physical layer repair.

[0025] Based on the initial physical layer, structural boundary constraint graphs and semantic shape constraint graphs are introduced to repair the damaged areas of the initial physical layer. The specific method for generating the refined physical layer is as follows: Identify the area to be repaired by masking the damaged area in the initial physical layer; among them, the damaged area includes stains, tears, and missing areas, and the pixels with pixel values greater than the standard threshold in the mask are marked as the area to be repaired; the standard threshold can be set according to the empirical method; For the pixels in the area to be repaired, call the structural boundary constraint graph and the semantic shape constraint graph, and use the constraint strength to repair the pixels in the area to be repaired, generating various material layer masks of the refined initial physical layer, including the refined paper substrate mask, text ink mask, seal ink mask, damaged area mask, and other element masks, all stored in the form of 8-bit grayscale images; Example: For the pixels in the damaged area that belong to the seal ink mask, only the pixels located in the seal contour constraint channel in the structural boundary constraint graph and with the constraint strength of this channel greater than 0 are repaired; during repair, the constraint strength corresponding to the seal text in the semantic shape constraint graph is used as the shape guide to make the repair result consistent with the glyph template; the texture source is taken from the most complete area with the highest confidence in the seal ink mask in the initial physical layer (extracted through connected component analysis); for example: due to paper folding, there is a tear mark in the official seal area, resulting in partial loss of the official seal contour, and the bottom right stroke of the character "limit" inside the official seal is also missing; locate the pixels in the damaged area that belong to the "seal ink mask", and check whether they are located inside the seal contour constraint channel in the structural boundary constraint graph; because the seal contour constraint channel has marked the area inside the circular official seal and the constraint strength is 1.0, these pixels meet the repair conditions (constraint strength > 0); in the seal ink mask of the initial physical layer, find the undamaged complete official seal area (such as the upper left corner of the official seal) through connected component analysis and use it as the texture sample; during repair, read the glyph template of the character "limit" in the semantic shape constraint graph (the constraint strength of each pixel in this channel is equal to the confidence of the text unit, for example, 0.9); for the missing stroke area of the character "limit", the shape of the glyph template will be referred to, and the ink in the texture sample will be preferentially filled in according to the contour of the glyph template instead of filling randomly; the official seal contour is completed and the gap is repaired; the missing strokes of the character "limit" are restored according to the shape of the glyph template, and the repaired strokes are consistent with the surrounding seal ink texture; For pixels belonging to the text ink mask within the damaged area, the constraint strength of the text baseline constraint channel in the structural boundary constraint map is used as the repair weight, and an image inpainting algorithm based on the orientation field is employed to restore ink continuity. During repair, the constraint strength of the corresponding text unit in the semantic shape constraint map is considered, prioritizing the preservation of the character structure in areas with higher constraint strength. After repair, for areas where the pixel value is inconsistent with the text baseline direction in the repair result, morphological closing operations are used for smoothing. For example, in the handwritten amount "¥12,345.67", the "3" is partially covered by ink stains. For pixels belonging to the "text ink mask" within the damaged area, the text baseline constraint channel in the structural boundary constraint map is invoked. In this channel, along the baseline direction of the amount number (assuming it is horizontal)... The system uses a series of pixels with a constraint strength of 1.0, while other areas have a constraint strength of 0. An image inpainting algorithm based on the orientation field is employed. The algorithm determines the direction of ink extension (along the baseline) based on the constraint strength of the text baseline constraint channel, ensuring that the repaired ink remains horizontal and not skewed. Simultaneously, it reads the character template channel of the letter "3" in the semantic shape constraint map (constraint strength is the character unit confidence level, e.g., 0.85). In areas covered by smudges, the system prioritizes filling the ink according to the shape of the "3" (two semicircles), rather than simply copying from the surrounding texture. After repair, the system checks whether the pixels in the repaired area are consistent with the baseline direction. If individual pixels deviate (e.g., bulge upwards), morphological closing operations (dilation followed by erosion) are used to smooth the overall shape, making it regular. For pixels in the damaged area that belong to the paper substrate mask or other element masks, texture synthesis or background filling is used for repair. No structural boundary constraint map or semantic shape constraint map is used in the repair process. If the stained area also involves the paper substrate (for example, the stain covers the blank area), then for pixels that belong to the "paper substrate mask", the system directly uses texture synthesis or background filling to sample textures from the surrounding clean paper area for filling, without using any structural or semantic constraints. Based on the degree of matching between the repaired pixels and the structural boundary constraint map and semantic shape constraint map, the initial physical layer confidence map after refinement is calculated, and the refined physical layer confidence map is obtained. The refined physical layer is composed of various material layer masks and the confidence map of the refined physical layer after the initial state physical layer is refined.

[0026] The specific methods for calculating the confidence map of the refined physical layer based on the matching degree between the repaired pixels and the structural boundary constraint map and semantic shape constraint map include: For the repaired pixels, calculate the distance deviation between the pixel position and the nearest constraint boundary (such as text baseline line segment, seal outline, page boundary) in the structural boundary constraint diagram. Based on the distance deviation of the nearest constraint boundary and the standard deviation parameter of the distance deviation, calculate the degree of structural constraint matching. For the repaired pixel, a local image patch with a fixed neighborhood range centered on the pixel is extracted as the repair region. The similarity between the repair region and the corresponding glyph template in the semantic shape constraint map is calculated, and the similarity is used as the degree of semantic constraint matching of the pixel. For the repaired pixel, the confidence level of the repaired pixel is calculated by weighted fusion, combining its confidence level in the initial physical layer confidence map, the degree of structural constraint matching, and the degree of semantic constraint matching. The same method is used to calculate the confidence level of other pixels to be repaired, and finally the confidence map of the refined physical layer is obtained.

[0027] The specific methods for generating the optimized structure layer include: extracting new geometric elements by reversing the process of refining the physical layer, and optimizing the initial structure layer through a consistency comparison and fusion strategy; and generating the optimized structure layer through: Based on the refined initial physical layer of various material layer masks, new geometric elements are extracted; The new geometric elements are compared with the geometric elements in the initial structure layer to obtain the comparison results. Based on the comparison results, a fusion strategy is executed to output the optimized geometric vector data of the initial structure layer. The confidence of the geometric elements in the initial structure layer is updated by comprehensively considering three factors: fitting quality, consistency degree, and physical support. The updated confidence map of the initial structure layer is then output. The optimized structure layer is composed of the optimized geometric vector data of the initial structure layer and the updated confidence map of the initial structure layer.

[0028] Based on the refined initial physical layer mask for various material layers, the specific steps for extracting new geometric elements are as follows: Using the seal ink mask as input, morphological closing operation is used to fill the micro-fractures that exist after repair. The outer contour point set is extracted through edge detection. After identifying the seal geometry type, the appropriate fitting algorithm is selected. For circular or elliptical seals, the least squares method or RANSAC algorithm is used to fit the center coordinates, radius, and rotation angle. For rectangular seals, the vertex sequence is fitted. The fitting residual is calculated as a quality index and the new geometric elements of the seal are output. Using the text ink mask as input, horizontal projection analysis is performed to determine the approximate position of each line of text, including line height and line spacing. For each line of text, the centroid point set of the text pixels is extracted. The RANSAC algorithm is used to fit a straight line and remove outliers. The standard deviation of the distance from the point to the fitted line is calculated as the fitting residual. The coordinates of the start and end points of the baseline line segment, the slope, the intercept, and the fitting residual are output. These are all new geometric elements of the text baseline line segment. Using the paper substrate mask as input, the maximum outer contour of the paper area is extracted, the minimum bounding rectangle is calculated to obtain the coordinates of the four corners of the page, the fit between the contour and the bounding rectangle is calculated as a quality index, and the coordinates of the top left, top right, bottom right, and bottom left corners of the page and the page rotation angle are output. At the same time, the fit index is also output. These are all new geometric elements of the page boundary. Using other element masks as input, Hough transform is used to detect horizontal and vertical line segments, and line segments are merged and extended to form a complete set of borders. The continuity and integrity of line segments are calculated as quality indicators, and the horizontal line set, vertical line set, cell grid structure and continuity index are output, which are other new geometric elements. The extraction of the above-mentioned new geometric elements can be performed in parallel without strict sequential dependencies. During actual execution, the specific elements to be extracted are dynamically selected based on the content. For example, for images without stamps, the extraction of new geometric elements of stamp outlines is skipped, and for images without tables, the extraction of new geometric elements of table borders is skipped. The specific methods of the integration strategy include: Existence check: If the new geometric element has a corresponding term in the initial state structure layer, then proceed to the deviation degree check; If a new geometric element does not have a corresponding term in the initial state structure layer, it is determined to be a newly added geometric element, which is then added directly and assigned an initial confidence level based on the fitting quality, and marked as newly added. If a geometric element in the initial state structure layer has no corresponding item in the new geometric element, it is determined that a geometric element is missing. The geometric element is retained, and the initial confidence of the geometry in the initial state structure layer is reduced (e.g., reduced to half of the confidence), and it is marked as needing to be reviewed. Deviation degree judgment: Compare the deviation of the key parameters in the new geometric element and the initial state structure layer geometric element with the preset deviation threshold. If the deviation of the key parameters in the new geometric element and the geometric element is less than the preset deviation threshold, it is judged to be completely consistent, the geometric element is retained, and the initial confidence of the geometry in the initial state structure layer is increased (e.g., increased to 1.2 times the confidence, but the upper limit is not more than 1), and it is marked as consistent with the old (here, the old refers to the confidence of the geometry in the initial state structure layer). If the deviation is greater than or equal to the preset deviation threshold but less than the maximum allowable threshold, it is considered an acceptable deviation. The fitting quality is then compared further. If the fitting quality of the new geometric element is better than that of the geometric element, the new geometric element replaces the original geometric element. The confidence level is set based on the fitting quality of the new geometric element, and the element is marked as optimized. If the fitting quality of the geometric element is better than or equal to that of the new geometric element, the geometric element is retained. The initial confidence level of the geometry in the initial state structure layer is set based on the fitting quality of the geometric element, and the element is marked as verified. If the value is greater than or equal to the maximum allowable threshold, it is judged as a serious conflict. At the same time, the new geometric element and the geometric element are retained as candidates, and the initial confidence of geometry in the initial state structure layer is adjusted to a low value (such as setting all to 0.3), and marked as pending confirmation. The preset deviation threshold and the maximum allowable threshold can be obtained from relevant materials.

[0029] The method for updating the confidence level of geometric elements and outputting the updated structural layer confidence map by comprehensively considering three factors—fit quality, consistency, and physical support—includes the following: Among them, the fitting quality is obtained by normalizing the residual index of the geometric element fitting algorithm, and the smaller the residual, the higher the confidence level; the consistency degree is determined by the consistency degree of the key parameters of the new and old geometric elements, with complete consistency being 1.0, acceptable deviation being 0.6-0.9, and serious conflict being 0.2-0.4; physical support is taken as the average pixel confidence level of the region where the geometric element is located in the refined physical layer confidence map.

[0030] The fit quality is obtained by normalizing the fit residuals of the extracted geometric elements (or new geometric elements). It is the standard deviation of the distance from the point to the fitted curve (in pixels); set a maximum allowable residual. ,when When the confidence level of the fit is 0, The confidence level is 1; the maximum permissible residual is usually... Based on the image resolution setting, then according to The fitting quality is calculated, where, For fitting quality; The degree of consistency is determined by the deviation between the key parameters of the new geometric element and the key parameters of the original geometric element. For the seal outline, key parameters include the deviation of the center distance. and radius deviation Set a threshold for perfect consistency in the distance between the centers of the circles. Maximum allowable threshold for center distance deviation Threshold for perfectly consistent radius deviation and the maximum allowable threshold for radius deviation (Set according to image resolution); and according to , Calculate the consistency component of the center distance deviation Consistency component of radius deviation The consistency degree is obtained by weighting and combining the consistency components of the center distance deviation and the radius deviation. The center distance deviation is the Euclidean distance between the center coordinates of the seal in the newly extracted seal outline and the center coordinates of the seal in the initial state structure layer. The radius deviation is the absolute difference between the seal radius in the newly extracted seal outline and the seal radius in the initial state structure layer. Physical support is the average confidence of all pixels in the confidence map of the refined physical layer corresponding to the region where the new geometric element is located in the refined initial physical layer. For example, if the seal outline is re-extracted in the refined physical layer, the region where the new geometric element is located refers to the set of pixels covered by this newly extracted seal. Each pixel has a confidence, so the physical support is the average confidence of the pixels in the region covered by this newly extracted seal. A correction factor is generated by weighted fusion of fitting quality, consistency, and physical support. Based on the correction factor and the confidence of geometric elements, the updated confidence of geometric elements is obtained through weighted fusion. The weights of fitting quality, consistency, and physical support can be set to 1 / 3, and the weights of correction factor and confidence of geometric elements can be set to 1 / 2. Based on the initial semantic layer and the refined physical layer, residual regions are labeled using residual localization rules. Local image enhancement and a three-level cascaded recognition strategy are applied to the residual regions. The specific methods for generating the corrected semantic layer include: Based on the text unit sequence, text unit confidence, and region type in the initial semantic layer, as well as the text ink mask and confidence map in the refined physical layer, residual localization rules are used to identify text units that need to be re-inferred and mark them as residual regions. For the labeled residual regions (i.e. the text units that need to be re-inferred), local image enhancement is performed using the text ink mask and the confidence map of the refined physical layer. A three-level cascaded recognition strategy is then used to re-recognize the enhanced local images to obtain the preliminary updated text recognition results (including the updated text unit recognition probability and text unit confidence). The specific steps for local image enhancement using the text ink mask and the confidence map of the refined physical layer are as follows: First, for the marked residual region, the original local image block of the region is cropped from the original handwritten document image; at the same time, local text ink mask blocks and local confidence map blocks with the same coordinate range are cropped from the text ink mask and the confidence map of the refined physical layer. Secondly, the foreground of the original local image block is extracted using a local text ink mask block. The pixels corresponding to the non-text ink areas in the local text ink mask block are set to the background color (usually the paper background color) to obtain the extracted local image. Next, for pixels with a confidence level greater than or equal to 0.7 in the local confidence patch, the minimum and maximum gray values ​​of these pixels in the extracted local image are calculated, and then the gray values ​​of the corresponding pixels are proportionally ( , The new grayscale value is obtained by linear stretching (where g is the grayscale value of the corresponding pixel). For example, the minimum grayscale value is 80 and the maximum grayscale value is 120, but a standard grayscale value range is 0 to 255. Grayscale values ​​between 80 and 120 are mapped proportionally to the range of 0 to 255. Thus, a grayish stroke with low contrast (80 to 120) will become a stroke with obvious differences between pure black and pure white. For pixels with a confidence level less than or equal to 0.3 in a local confidence patch, take that pixel as the center and the corresponding pixel in the extracted local image as the center, calculate the median gray value of all pixels in its 3*3 neighborhood, and replace the gray value of that pixel with the median gray value. For pixels with a confidence level greater than 0.3 and less than 0.7 in the local confidence patch, the grayscale value of the corresponding pixel in the extracted local image remains unchanged; finally, the enhanced local image is generated. The geometric vector data in the optimized structure layer is used to correct the initial updated text recognition results, resulting in structurally aligned text results (including rearranging the text order, correcting the paragraph layout, and adjusting the seal text arrangement), and the structural support of each text unit is obtained (the normalized value of the distance deviation from the text unit to the nearest baseline segment is the structural support). The process involves using geometric vector data from the optimized structure layer to provide feedback correction to the initially updated character recognition results, resulting in structurally aligned text. Specifically, this includes extracting baseline segments from the structure optimization layer, associating each character unit in the initially updated character recognition results with the nearest baseline segment (e.g., calculating the vertical distance from the center point of each character unit to each baseline segment, selecting the nearest baseline segment as the assigned baseline for that character unit), sorting the characters within the same baseline according to their projected coordinates (i.e., the projected distance from the center point of the character unit to the starting point of the baseline) from smallest to largest to obtain the correct inline text order, retaining character units not associated with any baseline segment in their original position and marking them as awaiting review, and finally outputting the rearranged text order. Based on the updated character unit recognition probability and structural support in the initial updated character recognition results, the final character unit confidence is calculated by weighting; the structural pair and the final character unit confidence are combined to form a modified semantic layer.

[0031] The specific steps of the residual location rule are as follows: Condition 1: If the confidence level of a text unit is lower than the second reference threshold, it is marked as a region requiring mandatory re-identification. Condition 2: Text units whose confidence levels fall between the second reference threshold and the first reference threshold (inclusive) are marked as confidence levels to be verified; the first and second reference thresholds are obtained from relevant literature. Condition 3: The coverage ratio of the refined physical layer text ink mask and the initial state semantic layer text bounding box is lower than the standard coverage ratio (e.g., 0.7), and it is marked as an area with insufficient spatial matching; among them, the coverage ratio is the ratio of the number of intersection pixels of the refined physical layer text ink mask and the initial state semantic layer text bounding box to the number of pixels of this text bounding box; Condition 4: The average value of the refined physical layer confidence corresponding to the text area mask in the refined physical layer is lower than the standard average value (0.6), and it is marked as an area with low physical reliability; If a text unit meets at least one condition, it is marked as a residual area; a text unit that does not meet any condition is marked as a high-confidence retention area, and the original text recognition result is preferentially retained in the subsequent fusion step.

[0032] Among them, the specific method of the three-level cascade recognition strategy includes: The first-level candidate screening: Use a lightweight convolutional neural network to recognize the enhanced local image, and generate a candidate text set and the initial confidence of the text unit; among them, the enhanced local image is segmented (such as using connected component analysis or the text bounding box coordinates in the initial state structure layer) into enhanced local image blocks of single text units; and input into the network. The front end of the network uses depthwise separable convolution to extract multi-scale feature maps, and the back end is connected to a global average pooling layer, a fully connected layer, and a Softmax classifier, and outputs the probability values of the text unit belonging to each character in the predefined character set (such as if the input text is an enhanced local image block of an uncertain character '月', the final output is '月(0.7), 日(0.15), 用(0.08), 目(0.03), etc.'); Set a probability threshold (such as 0.1), and screen out the top N candidate characters with higher probabilities (usually take 2 to 7) to form a candidate text set, and at the same time record the initial confidence of the candidate characters in each candidate text set (that is, the probability value output by Softmax); At the second - level fine - discrimination stage, for the candidate character set, the attention mechanism model is used to output the fine - recognition sequence and recognition probability. For the candidate character set generated at the first - level (such as "month", "day", "use", "eye", etc.), the enhanced local image patches of individual character units are input into the attention mechanism model (such as the encoder - decoder model based on the attention mechanism). The encoder uses a deep convolutional network (such as ResNet) to extract a multi - scale feature map sequence of the enhanced local image patches. The decoder uses a recurrent neural network (such as LSTM or GRU) combined with the attention mechanism to dynamically calculate the region weights in the feature map that are most relevant to the current character at each decoding time step, generate a context vector, and predict the probability distribution of the current character based on this vector and the hidden state at the previous moment. Since the candidate character set has significantly reduced the search space, the attention mechanism can more precisely focus on stroke details, thereby effectively distinguishing between similar - shaped characters such as "month", "day", "use", "eye", etc. The decoder finally outputs the fine - recognition probabilities of the character unit belonging to each character in the candidate character set (e.g., finally output "month" (0.92), "day" (0.05), "use" (0.02), "eye" (0.01)), and selects the character with the highest probability as the final fine - recognition sequence and recognition probability (e.g., "month" (0.92)); At the third - level rule verification stage, for the fine - recognition sequence, the stroke - structure rule library is used for verification. The character unit that passes the verification and its recognition probability are weighted - fused with the physical support degree to obtain the character - unit confidence. Specifically, taking "month" in the fine - recognition sequence as the key, retrieve the standard stroke information (such as the number of strokes, stroke order) of this character in the stroke - structure rule library. For the enhanced local image patch of the corresponding individual character unit, extract the skeleton features of the actual written strokes (such as the positions of stroke endpoints, intersection points, stroke - direction angles, etc.), and compare them item - by - item with the standard stroke structure in the rule library to calculate the structure matching degree. A standard matching degree can be set, and if it is greater than or equal to the standard matching degree, it means passing the verification. If it passes the verification, the recognition probability of this character (character unit) is averaged and weighted - fused with the physical support degree to obtain the confidence of this character unit (if it does not pass the verification, in the first - level, reduce the preliminary confidence, such as reducing it by half, and repeat the operation until the verification passes). Here, both the character and the character unit represent the same meaning (a single character), just with different names in different contexts; Among them, the stroke - structure rule library calls existing libraries, such as the MakeMeAHanzi project and the GB2312 standard font library template, which include thousands of Chinese characters, including structured information such as the stroke order, number of strokes, median line of strokes, pinyin, interpretation, and radical decomposition of each Chinese character; Based on the refined physical layer, a visually enhanced base map is obtained; and the specific methods for generating an enhanced handwritten document image by combining the optimized structure layer and the corrected semantic layer include: Based on the various material masks in the refined physical layer, pixel-level fusion reconstruction is performed (the various material masks in the refined physical layer are superimposed according to priority: the stamp ink mask has the highest priority, followed by the text ink mask, then other element masks, and the paper substrate mask is used as the background; for pixels in overlapping areas, the pixel value corresponding to the highest priority mask is taken (e.g., when stamp ink covers text ink, stamp ink is displayed first); for non-overlapping areas, the pixel value of the corresponding mask is taken directly), generating the final visual base map; the final visual base map is then processed and enhanced (including contrast, noise suppression, and sharpening operations) to obtain the visually enhanced base map; The optimized geometric elements of the structural layer are overlaid on the visually enhanced base image, and multi-dimensional consistency verification is performed to obtain the qualified geometric elements of the structural layer (e.g., the consistency verification of the stamp area is: comparing the pixel position of the stamp ink in the visually enhanced base image with the coverage (proportion of the stamp ink within the outline) and geometric deviation (average distance between the outline and the edge of the ink) of the stamp outline in the optimized structural layer; if the coverage is lower than a set threshold (e.g., 0.8) or the geometric deviation is greater than a set threshold (e.g., 3 pixels), it is marked as needing adjustment; the consistency verification of the text baseline segment is: comparing the centroid position of the text ink in the visually enhanced base image with the vertical distance of the text baseline in the optimized structural layer; if the average distance is greater than a set threshold (e.g., 2 pixels), it is marked as needing adjustment; the consistency verification of the page boundary is: comparing the positional deviation of the actual page edge (extracted through edge detection) in the visually enhanced base image with the page boundary of the optimized structural layer; if the corner deviation is greater than a threshold (e.g., 5 pixels), it is marked as needing adjustment). For geometric elements marked as needing adjustment in the multi-dimensional consistency verification, iterative optimization adjustments are performed (e.g., using the edge point set of the stamp ink in the visually enhanced base image as a reference, the stamp outline (circle / ellipse) is refitted using the least squares method, updating the center coordinates, radius, and rotation angle; after adjustment, the coverage and geometric deviation are recalculated, and if they are still lower than the set threshold, the confidence of the stamp outline is reduced (e.g., by 0.8 times the original confidence); using the centroid point set of the text ink in the visually enhanced base image as a reference, the text baseline is refitted using the RANSAC algorithm, updating the baseline start and end coordinates, slope, and intercept; after adjustment, the average distance from the text centroid to the baseline is recalculated, and if it is still greater than the set threshold, the baseline confidence is reduced (e.g., by 0.8 times the original confidence); using the actual page edge in the visually enhanced base image as a reference, the minimum bounding rectangle is recalculated, updating the page corner coordinates and rotation angle; after adjustment, the corner deviation is recalculated, and if it is still greater than the set threshold, the page boundary confidence is reduced (e.g., by 0.8 times the original confidence)). The enhanced handwritten document image is obtained by fusing the corrected semantic layer, the visually enhanced base map, and the validated structural layer geometric elements. Based on the text content, seal text, and paragraph layout of the semantic layer, and combined with the visual enhancement base map and verified structural layer geometric elements (such as overlaying verified structural layer geometric elements on the visual enhancement base map and embedding structured metadata files), an enhanced handwritten document image is obtained.

[0033] By fusing multi-source information and verifying consistency, enhanced handwritten document images are generated that balance visual realism, geometric accuracy, and content correctness.

[0034] like Figure 3 As shown, a handwritten document reconstruction and stamp completion system based on multimodal fusion includes: Multimodal decoupling dual-constraint construction engine: acquires handwritten document images; extracts initial state physical layer, initial state structural layer and initial state semantic layer through hierarchical extraction; and constructs structural boundary constraint map and semantic shape constraint map; Constraint-guided intelligent refinement unit for physical layer: Based on the initial physical layer, structural boundary constraint graphs and semantic shape constraint graphs are introduced to repair the damaged areas of the initial physical layer and generate a refined physical layer; Reverse geometric fusion structure optimization unit: New geometric elements are extracted by reversing the physical layer, and the initial structure layer is optimized by consistency comparison and fusion strategy to generate the optimized structure layer; The residual localization semantic depth correction unit: Based on the initial semantic layer and the refined physical layer, residual regions are marked using residual localization rules; local image enhancement and a three-level cascaded recognition strategy are applied to the residual regions to generate the corrected semantic layer; Multidimensional fusion visual enhancement and reconstruction unit: Based on the refined physical layer, a visual enhancement base map is obtained; and combined with the optimized structural layer and the corrected semantic layer, an enhanced handwritten document image is generated.

[0035] In this embodiment, through layered processing and refinement at the physical, structural, and semantic layers, the paper texture, ink marks, seal ink, and damaged areas of handwritten documents can be accurately reconstructed, significantly improving the readability and visual integrity of the image. Simultaneously, combined with local enhancement technology, it can recover incomplete text and seal information in complex scenes, reducing the false recognition rate and improving the accuracy and reliability of text and seal information. This provides a solid technical guarantee for applications such as document digitization, archival preservation, and judicial and cultural relic restoration. By fully leveraging the capabilities of multimodal information fusion, physical texture, geometric structure, and semantic information are mutually constrained and corrected to achieve high-precision restoration of document content. Multi-dimensional consistency verification and structure-semantic feedback mechanisms ensure the consistency of reconstruction results at the pixel, shape, and textual logic levels, improving the robustness of restoration and recognition. Meanwhile, dynamic weight updates based on confidence and fitting quality enable the method to adapt to different types of handwritten documents, different seal styles, and degrees of damage, avoiding the limitations of single-modal restoration methods and enhancing the overall versatility and reliability of the method.

[0036] This application also provides an electronic device. The electronic device may include one or more processors and one or more memories. The memories store computer-readable code, which, when executed by the one or more processors, can perform the handwritten document reconstruction and stamp completion method and system based on multimodal fusion as described above.

[0037] The methods and systems according to the embodiments of this application can also be implemented using the architecture of the electronic device shown in this application. The electronic device may include a bus, one or more CPUs, ROM, RAM, a communication port connected to a network, input / output, a hard disk, etc. The storage device in the electronic device, such as a ROM or hard disk, may store the handwritten document reconstruction and stamp completion method and system based on multimodal fusion provided in this application. Furthermore, the electronic device may also include a user interface. Of course, the architecture shown in this application is merely exemplary; when implementing different devices, one or more components in the electronic device shown in this application may be omitted according to actual needs.

[0038] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a reference structure" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0039] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for handwritten document reconstruction and stamp completion based on multimodal fusion, characterized in that, The method includes: Step S1: Obtain the handwritten document image; extract the initial physical layer, initial structural layer, and initial semantic layer through layering; and construct the structural boundary constraint graph and semantic shape constraint graph. Step S2: Based on the initial physical layer, introduce structural boundary constraint diagram and semantic shape constraint diagram to repair the damaged area of ​​the initial physical layer and generate the refined physical layer; Step S3: Extract new geometric elements by reverse engineering the refined physical layer, and optimize the initial structure layer through consistency comparison and fusion strategies to generate the optimized structure layer; Step S4: Based on the initial semantic layer and the refined physical layer, residual regions are marked using residual localization rules; local image enhancement and a three-level cascaded recognition strategy are applied to the residual regions to generate a corrected semantic layer. Step S5: Obtain a visually enhanced base map based on the refined physical layer; and combine it with the optimized structural layer and the corrected semantic layer to generate an enhanced handwritten document image.

2. The method for handwritten document reconstruction and seal completion based on multimodal fusion according to claim 1, characterized in that, The process of extracting the initial physical layer, initial structural layer, and initial semantic layer through layering includes: Based on the handwritten document image, a preliminary color classification map is generated, and the pixel color purity and edge sharpness are calculated; for the preliminary color classification map, layered masks for various materials are generated; and the initial confidence of the pixels is obtained to obtain the physical layer confidence map; and the initial physical layer is constructed. For handwritten document images, geometric elements are extracted and geometric vector coordinates are generated; the initial geometric confidence of the geometric elements is calculated to obtain the structural layer confidence map; and the initial state structural layer is constructed. For handwritten document images, the system locates text regions and outputs text border coordinates; for text regions, it obtains text sequences, text units, and the recognition probability of text units; for text border coordinates, it obtains paragraph structure information; for specific elements in the text border coordinates, it obtains specific text sequences and specific text confidence scores; based on the text sequences, the recognition probabilities of text units, and the paragraph structure information, it obtains the confidence score of text units; according to the text unit confidence scores, it divides text units into high-confidence regions, low-confidence regions, and regions to be completed; and it constructs an initial semantic layer.

3. The method for handwritten document reconstruction and seal completion based on multimodal fusion according to claim 2, characterized in that, The construction of the structural boundary constraint graph and semantic shape constraint graph includes: The geometric vector data is converted into a geometric pixel-level constraint mask, and the pixels within the mask are marked. Based on the geometric pixel-level constraint mask and geometric confidence, the pixel constraint strength is constructed according to the spatial relationship between the pixels and geometric elements. The constructed pixel constraint strength is stored in the geometric channel according to the geometric elements to form a multi-channel structural boundary constraint map. For the initial semantic layer, glyph templates are extracted; the extracted glyph templates are converted into pixel-level constraint maps; and constraint strength is constructed based on the confidence of character units. A semantic shape constraint map is generated based on pixel-level constraint maps and constraint strengths.

4. The method for handwritten document reconstruction and seal completion based on multimodal fusion according to claim 3, characterized in that, The process involves using the initial physical layer as a foundation, introducing structural boundary constraint maps and semantic shape constraint maps to repair damaged areas of the initial physical layer, and generating a refined physical layer, including: For the damaged areas in the initial physical layer, the mask identifies the areas to be repaired. For the pixels in the areas to be repaired, the structural boundary constraint map and semantic shape constraint map are called. The constraint strength is used to repair the pixels in the areas to be repaired, and a refined material layering mask for the initial physical layer is generated. Based on the repaired pixels and their matching degree with the structural boundary constraint map and semantic shape constraint map, the confidence map of the refined initial physical layer is calculated to obtain the confidence map of the refined physical layer; and the refined physical layer is constructed.

5. The method for handwritten document reconstruction and seal completion based on multimodal fusion according to claim 4, characterized in that, The process of extracting new geometric elements from a refined physical layer and optimizing the initial structure layer through a consistency comparison and fusion strategy to generate an optimized structure layer includes: Based on the refined initial physical layer of various material layer masks, new geometric elements are extracted; The new geometric elements are compared with the geometric elements in the initial structure layer to obtain the comparison results. Based on the comparison results, a fusion strategy is executed to output the optimized geometric vector data of the initial structure layer. The confidence of the geometric elements in the initial structure layer is updated by comprehensively considering three factors: fitting quality, consistency degree, and physical support. The updated confidence map of the initial structure layer is then output. The optimized structure layer is composed of the optimized geometric vector data of the initial structure layer and the updated confidence map of the initial structure layer.

6. The method for handwritten document reconstruction and stamp completion based on multimodal fusion according to claim 5, characterized in that, The fusion strategy includes: Existence check: If a new geometric element has a corresponding item in the initial state structure layer, then proceed to the deviation level check; if there is no corresponding item, then it is a newly added geometric element, which is added and given an initial confidence level, and marked as newly added; if there is no corresponding item for the geometric element in the initial state structure layer, then the geometric element is retained, the initial geometric confidence level is adjusted, and it is marked as needing to be reviewed. Degree of deviation judgment: If the key parameters of the new geometric elements and the initial structure layer geometric elements deviate less than the preset deviation threshold, they are completely consistent. The geometric elements are retained, the initial confidence of the geometry is adjusted, and they are marked as consistent with the old and new. If the deviation is greater than or equal to the preset deviation threshold but less than the maximum allowable threshold, it is considered an acceptable deviation. A comparison is then made. If the fitting quality of the new geometric element is better than that of the geometric element, the new geometric element is used to replace the original geometric element, and the confidence level of the new geometric element is set, marking it as optimized. If the fitting quality of the geometric element is better than or equal to that of the new geometric element, the geometric element is retained, and the initial confidence level of the geometry is adjusted, marking it as verified. If the value is greater than or equal to the maximum allowable threshold, it is considered a serious conflict. The new geometric element and the original geometric element are retained, the initial geometric confidence is adjusted, and it is marked as pending confirmation.

7. The method for handwritten document reconstruction and stamp completion based on multimodal fusion according to claim 6, characterized in that, The residual region is marked using residual localization rules based on the initial state semantic layer and the refined physical layer. A local image enhancement and three-level cascaded recognition strategy is applied to the residual region to generate a corrected semantic layer, including: Based on the text unit sequence, text unit confidence, and region type in the initial semantic layer, as well as the text ink mask and confidence map in the refined physical layer, residual localization rules are used to identify text units that need to be re-inferred and mark them as residual regions. For the labeled residual regions, local image enhancement is performed using the text ink mask in the refined physical layer and the confidence map of the refined physical layer. A three-level cascaded recognition strategy is then used to re-recognize the enhanced local image to obtain the preliminary updated text recognition results. The geometric vector data in the optimized structure layer is used to correct the initial updated text recognition results, resulting in structurally aligned text results, and the structural support of each text unit is obtained. Based on the updated character unit recognition probability and structural support in the preliminary updated character recognition results, the final character unit confidence is calculated by weighting; and a corrected semantic layer is constructed.

8. The method for handwritten document reconstruction and stamp completion based on multimodal fusion according to claim 7, characterized in that, The residual location rules include: Condition 1: If the confidence level of a text unit is lower than the second reference threshold, it is marked as a region requiring mandatory re-identification. Condition 2: If the confidence level of a text unit is between the second reference threshold and the first reference threshold, it is marked as a confidence level verification area. Condition 3: If the coverage ratio between the refined physical layer text ink mask and the initial semantic layer text bounding box is lower than the standard coverage ratio, it is marked as a region with insufficient spatial matching. Condition 4: If the mean confidence level of the refined physical layer corresponding to the Chinese character region mask in the refined physical layer is lower than the standard mean, it is marked as a region with low physical reliability. A text unit that meets at least one condition is marked as a residual region; a text unit that does not meet any condition is marked as a high-confidence reserved region.

9. The method for handwritten document reconstruction and stamp completion based on multimodal fusion according to claim 8, characterized in that, The visually enhanced basemap is obtained based on the refined physical layer; By combining an optimized structural layer and a corrected semantic layer, enhanced handwritten document images are generated, including: Based on various material masks in the refined physical layer, pixel-level fusion reconstruction is performed to obtain a visually enhanced base map; the optimized geometric elements of the structural layer are superimposed on the visually enhanced base map, and multi-dimensional consistency verification is performed to obtain the verified structural layer geometric elements. Based on the modified semantic layer, and combined with the visually enhanced base map and verified structural layer geometric elements, an enhanced handwritten document image is obtained.

10. A handwritten document reconstruction and seal completion system based on multimodal fusion, characterized in that, The system includes: Multimodal decoupling dual-constraint construction engine: acquires handwritten document images; extracts initial state physical layer, initial state structural layer and initial state semantic layer through hierarchical extraction; and constructs structural boundary constraint map and semantic shape constraint map; Constraint-guided intelligent refinement unit for physical layer: Based on the initial physical layer, structural boundary constraint graphs and semantic shape constraint graphs are introduced to repair the damaged areas of the initial physical layer and generate a refined physical layer; Reverse geometric fusion structure optimization unit: New geometric elements are extracted by reversing the physical layer, and the initial structure layer is optimized by consistency comparison and fusion strategy to generate the optimized structure layer; The residual localization semantic depth correction unit: Based on the initial semantic layer and the refined physical layer, residual regions are marked using residual localization rules; local image enhancement and a three-level cascaded recognition strategy are applied to the residual regions to generate the corrected semantic layer; Multidimensional fusion visual enhancement and reconstruction unit: Based on the refined physical layer, a visual enhancement base map is obtained; and combined with the optimized structural layer and the corrected semantic layer, an enhanced handwritten document image is generated.

Images