Digital restoration method, system and equipment for scanned image of defective ancient book and medium

Abstract

The invention discloses a defective ancient book scanning image digital restoration method, system and device and a medium, and relates to the technical field of image processing, and the method comprises the steps: importing a to-be-restored ancient book scanning image; performing OCR processing on the image to obtain an OCR result; segmenting a defect area of the image to obtain a binary mask; constructing structured data based on an OCR result, inputting the structured data into the position enhancement inference model, and predicting complemented characters at the defect and a predicted position bounding box of the complemented characters; according to the defect proportion in the predicted position surrounding frame and the recognition result information entropy of the corresponding character, character restoration is carried out; after character pixels are separated from the ancient book scanning image, an ancient book background restoration model is called based on the binary mask to generate background pixels of the defect position, and restored characters and restored backgrounds are fused again; and storing and outputting a final restored image. The method assists the user in reasoning and proofreading in the whole process, and the repairing efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to methods, systems, equipment and media for the digital restoration of scanned images of damaged ancient books. Background Technology

[0002] Many ancient books and classics exist, but due to age and preservation issues, a significant portion have suffered damage. Currently, there are two main approaches to restoring the value of ancient book resources: physical restoration and digital restoration. Physical restoration focuses on the physical book itself, aiming to extend its lifespan as much as possible. With the development of computer technology and the digitization of cultural relics, digital restoration has also become a mainstream method. Because it removes the physical attributes of ancient books and restores them from scanned images, it is more conducive to resource utilization and cultural dissemination.

[0003] Digital restoration comprises two parts: verification of missing content and restoration of ancient book images (corresponding to documentary value and artistic value, respectively). These two parts are typically conducted separately. However, current industry research reveals two problems: First, verifying missing content, i.e., restoring documentary value, relies heavily on the knowledge base of authoritative experts in the field (history, archaeology, linguistics, and culture), resulting in a substantial workload. Second, image restoration also relies on manual image manipulation; while some basic image tools are available, the workload remains significant. These factors have led to a high demand in the industry for intelligent restoration methods. Research on commercial ancient book digitization systems reveals that digital restoration is usually a functional module within a digital resource processing system, offering only basic image processing techniques such as brightness and contrast adjustment, image correction, background removal, and morphological damage filling. Meanwhile, various cutting-edge AI restoration technologies have not yet been practically implemented. Summary of the Invention

[0004] Based on the technical problems existing in the background technology, this invention proposes a method, system, equipment and medium for digital restoration of scanned images of damaged ancient books, which assists users in reasoning and proofreading throughout the entire process and improves the restoration efficiency.

[0005] The method for digitally restoring damaged ancient books from scanned images proposed in this invention includes: Import the scanned image of the ancient book to be repaired; The image is processed by OCR to obtain OCR results including the initial bounding box of each character, the recognition result and the confidence score, and a manual proofreading interface is provided. Segment the missing regions in the scanned images of ancient books to obtain a binary mask; Missing Information Reasoning: Based on the OCR results, extract and structure the context paragraphs of the page text, insert variable-length mask markers at the missing parts, and input the structured data containing text, candidate words, initial position bounding boxes and variable-length mask markers into the position enhancement reasoning model to predict the complete text at the missing parts and its predicted position bounding boxes. Text restoration: Based on the proportion of missing text within the predicted bounding box and the information entropy of the corresponding text recognition result, select character library matching and replacement or call the AI ​​text restoration model to restore the text, thereby obtaining the restored text; Background restoration: After separating the text pixels from the scanned image of the ancient book, the background pixels of the missing area are generated by calling the ancient book background restoration model based on the binary mask, and the restored text is then re-fused with the restored background. Save and output the final repaired image.

[0006] Furthermore, the extraction and structuring of the contextual paragraphs of the page text specifically involves: Clustering is performed based on the x-coordinate of the center point of the initial position bounding box of each character, dividing it into initial character column blocks; Within each initial text column block, clustering is performed based on the vertical distance between texts to divide the text into multiple text columns; Clustering is performed based on the average size of the text in each text column to distinguish text columns with different font types; Merge adjacent text columns that belong to the same font type and whose fitted straight-line distance is within the threshold. If there are missing areas in the merged interval, insert variable-length mask markers. Based on the page layout, columns of text of the same type that conform to the logical relationship between the top and bottom will be merged into paragraphs.

[0007] Furthermore, the location-enhanced inference model is a multi-task model based on an improved RoBERTa architecture, including an embedding layer, a Roberta encoder backbone, a decoder, and four task heads; The vector projected from the candidate character is directly inserted into the position of the candidate character in the text embedding to construct a new text embedding, which includes the character and a variable-length mask marker. The embedding layer integrates new text embedding, position embedding, and initial position bounding box embedding, and converts them into vectors of a set dimension respectively; The new text embedding and the initial position bounding box embedding are transformed into input variables using a gating fusion mechanism, and then combined with the position embedding and input into the Roberta encoder backbone. The output of the Roberta encoder trunk is used to predict candidate words by passing the candidate word prediction task header and the candidate word location box task header respectively. The output of the Roberta encoder trunk is decoded by the decoder and then input into the mask interval text generation task head and the mask interval text position regression prediction task head, respectively, to generate variable-length text sequences at variable-length mask markers.

[0008] Furthermore, the new text embedding vector and the initial position bounding box embedding vector are transformed into input variables using a gating fusion mechanism. Combined with the position embedding, the fused embedding is obtained and input into the Roberta encoder backbone. The fusion embedding The formula is as follows: ; ; in, For the first New text embedded at each position For the first The initial position bounding box is embedded at each location. For the first Position embedding corresponding to each position For the first Type embedding corresponding to each position For the first The gating coefficient at each position, For activation function, Here is the weight matrix of the gated linear layer. For the bias term of the gated linear layer, This is a vector concatenation operation. This indicates element-wise multiplication.

[0009] Furthermore, the location-enhanced inference model is trained by jointly optimizing the loss functions of the following four tasks: candidate character classification loss, candidate character bounding box regression loss, masked interval text generation loss, and masked interval text bounding box regression loss.

[0010] Furthermore, based on the proportion of missing text within the predicted bounding box and the information entropy of the corresponding character recognition result, the method of selecting character library matching and replacement or calling an AI character restoration model for character restoration is as follows: When the defect ratio is higher than the first threshold and the information entropy is higher than the second threshold, the method of matching and replacing the characters from the pre-built ancient book font library is adopted to repair the defect. Otherwise, the AI ​​text restoration model is invoked to generate and fill in the missing pixels based on the remaining stroke pixels.

[0011] Furthermore, the ancient book background restoration model is an image restoration network based on a generative pre-trained model, trained using a database of ancient book page background images.

[0012] A digital restoration system for scanned images of damaged ancient books, including: The interaction layer provides a user interface for importing images, displaying results, and receiving manual proofreading instructions; The service layer encapsulates: The OCR module is used to detect and recognize text in images, and outputs the initial bounding box of each character, the recognition result, and the confidence score. The AI ​​inference module for missing information is used to segment the missing area to obtain a binary mask, extract the context paragraphs of the page text based on the OCR results, and call the position enhancement inference model to predict the complete text at the missing location and its predicted position bounding box. The text restoration module is used to select a character library for matching and replacement or call an AI text restoration model to restore the text based on the proportion of missing parts within the predicted bounding box and the information entropy of the corresponding text recognition result, so as to obtain the restored text. The paper background restoration module is used to separate the text pixels from the scanned image of the ancient book, and then call the ancient book background restoration model based on the binary mask to generate the background pixels of the missing area, and then re-integrate the restored text with the restored background. The model layer stores the weight parameters of the OCR model, the location-enhanced inference model, the AI ​​text restoration model, and the ancient book background restoration model. The data layer stores ancient book font libraries, ancient book page background libraries, and ancient book corpora.

[0013] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described above.

[0014] A computer-readable storage medium storing a plurality of classification programs, the plurality of classification programs being invoked by a processor to execute the method described above.

[0015] The advantages of the method, system, equipment, and medium for digital restoration of scanned images of damaged ancient books provided by this invention are as follows: It assists ancient book digitization engineers in completing resource digitization and image restoration work. Its key feature is that it divides the digital restoration process into multiple key steps, seamlessly integrating it into the existing ancient book resource digitization workflow. This makes it easy for industry professionals to use, and it integrates multiple tools such as OCR, text reasoning, text restoration, and background restoration, providing full-process assistance for users in reasoning and proofreading, thus improving restoration efficiency. Furthermore, this embodiment, specifically for tasks involving damaged text information, newly designs a position-enhanced reasoning model (ADR-Roberta) for damaged ancient book text. This model can embed multimodal information such as text, candidate characters, initial position bounding boxes, and variable-length mask markers output by OCR processing, and simultaneously infer the corresponding missing text and its position, greatly enhancing the ability to restore the original layout. Attached Figure Description

[0016] Figure 1 System architecture diagram of a system for digital restoration of scanned images of damaged ancient books; Figure 2 A flowchart of the entire system's workflow; Figure 3 This is a diagram of the system's physical architecture. Figure 4 A flowchart for performing contextual search on OCR results and constructing structured input text; Figure 5 This is a model architecture diagram of the location-enhanced reasoning model for the missing ancient texts used in this system. Figure 6 A model diagram of the text task header and bounding box task header for the location-enhanced inference model. Detailed Implementation

[0017] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0018] like Figures 1 to 6 As shown, the method for digitally restoring damaged ancient book scanned images proposed in this invention includes: Step 1: Import the scanned image of the ancient book to be restored; Step 2: Perform OCR processing on the image to obtain OCR results including the initial bounding boxes of each character, the recognition results, and the confidence scores, and provide a manual proofreading interface; Step 3: Segment the missing regions of the scanned image of the ancient book to obtain a binary mask; Step 4, Missing Information Inference: Based on the OCR results, extract and structure the context paragraphs of the page text, insert variable-length mask markers at the missing parts, and input the structured data containing text, candidate words, initial position bounding boxes and variable-length mask markers into the position enhancement inference model to predict the complete text at the missing parts and its predicted position bounding box. Step 5: Text Repair: Based on the proportion of missing text within the predicted bounding box and the information entropy of the corresponding text recognition result, select character library matching and replacement or call the AI ​​text repair model to repair the text, thereby obtaining the repaired text; Step 6: Background Restoration: After separating the text pixels from the scanned image of the ancient book, the background pixels of the missing area are generated by calling the ancient book background restoration model based on the binary mask, and the restored text is then re-fused with the restored background. Step 7: Save and output the final repaired image.

[0019] This embodiment assists ancient book digitization engineers in completing resource digitization and image restoration. Its key feature is the division of the digitization and restoration process into multiple key steps, seamlessly integrating it into the existing ancient book resource digitization workflow. This makes it easy for industry professionals to use, and it integrates multiple tools such as OCR, text reasoning, text restoration, and background restoration, providing end-to-end assistance for users in reasoning and proofreading, thus improving restoration efficiency. Furthermore, this embodiment proposes a novel enhanced location reasoning model (ADR-Roberta) for missing text information tasks. This model can embed multimodal information such as text, candidate characters, initial position bounding boxes, and variable-length mask markers output by OCR processing, simultaneously inferring the corresponding missing text and its position, greatly enhancing the ability to restore the original layout.

[0020] This embodiment provides a digital restoration system for scanned images of damaged ancient books to implement the above-mentioned restoration method, including: The interaction layer provides a user interface for importing images, displaying results, and receiving manual proofreading instructions; The service layer encapsulates: The OCR module is used to detect and recognize text in images, and outputs the initial bounding box of each character, the recognition result, and the confidence score. The AI ​​inference module for missing information is used to segment the missing area to obtain a binary mask, extract the context paragraphs of the page text based on the OCR results, and call the position enhancement inference model to predict the complete text at the missing location and its predicted position bounding box. The text restoration module is used to select a character library for matching and replacement or call an AI text restoration model to restore the text based on the proportion of missing parts within the predicted bounding box and the information entropy of the corresponding text recognition result, so as to obtain the restored text. The paper background restoration module is used to separate the text pixels from the scanned image of the ancient book, and then call the ancient book background restoration model based on the binary mask to generate the background pixels of the missing area, and then re-integrate the restored text with the restored background. The model layer stores the weight parameters of the location-enhanced inference model, the AI ​​text restoration model, and the ancient book background restoration model. The data layer stores ancient book font libraries, ancient book page background libraries, and ancient book corpora.

[0021] Specifically, The interaction layer primarily consists of web pages displayed to users through the front-end presentation module and JavaScript code embedded in the interactive function module for event interaction responses. The service layer encapsulates various functional modules, providing interfaces for system implementation, specifically including: a project resource management and configuration module, an OCR module, an AI inference module for missing information, a character shape restoration module, a paper background restoration module, and a user management and authentication module. The model layer contains the weights of machine learning models used for various AI inference services, including OCR models (including detection and recognition), an augmented reality (ADR-Roberta) model for missing text in ancient books, an AI model for character shape restoration in ancient books, and a model for background restoration in ancient books. The data layer comprises various resource databases used in the ancient book processing process, including ancient book font libraries, ancient book page background libraries, and ancient book corpora.

[0022] The ancient book font library, the ancient book page background library, and the ancient book corpus are all derivatives of the institution's process of digitizing ancient books.

[0023] Digitizing ancient books involves scanning traditional ancient books into images, performing OCR processing, labeling and manually calibrating the recognition results, and organizing the text into paragraphs according to the reading order.

[0024] For files that have already been digitally annotated, the individual text and background on the image can be further segmented into binary values. After the text is extracted, it can be stored as digital text after binarization and noise reduction. An ancient book font library can be built according to the text encoding and book source.

[0025] After all the text is segmented out, the remaining background can be directly filled to become a complete book page background, and a book page background library can be constructed according to the book source.

[0026] Manually organized text paragraphs constitute the corpus of ancient texts, which is the main task of digitizing ancient texts.

[0027] The project resource management and configuration module in the service layer is used to browse or edit completed and incomplete project files, and can access ancient book font libraries, ancient book page background libraries, ancient book corpora, etc., archived during project processing. It also includes setting commonly used parameters in the processing flow, such as model APIs and model parameters.

[0028] The user management and authentication module in the service layer is used to create multiple users for project resource management. Users log in to the system through password authentication, and special permissions between users can be specified by the super administrator, including access permissions to resource libraries and project libraries.

[0029] The system uses the Django development framework, including web components. Various AI inference services are implemented using FastAPI, and communication between the system and these services is via HTTP. This facilitates isolation between different runtime environments, as well as centralized computing power and cross-platform access. See the overall system architecture diagram below. Figure 1 See physical architecture Figure 3 As shown.

[0030] In one embodiment, step two involves performing OCR processing on the image to obtain OCR results including the initial bounding boxes of each character, the recognition results, and the confidence scores, and providing a manual verification interface. Specifically: In the service layer, the OCR module calls the OCR model in the model layer to provide character-level detection for the document, returning the initial bounding boxes of all detected characters on the page. The format is defined, and each character is identified separately. The corresponding identification results and confidence scores are returned. The confidence scores are sorted in descending order. For characters with the lowest top-1 confidence scores, multiple candidate characters are returned. These represent the horizontal center point, vertical center point, width, and height of the initial bounding box obtained after OCR processing.

[0031] The OCR model in this embodiment includes a text detection sub-model and a text recognition sub-model. The text detection sub-model is trained using the general detection model YOLO, and the text recognition sub-model is trained using the general image classification model ResNet.

[0032] In one embodiment, step four, missing information inference: Based on the OCR results, the context paragraphs of the page text are extracted and structured. Variable-length mask markers are inserted at the points crossing the missing text. The structured data containing the text, candidate characters, initial position bounding boxes, and variable-length mask markers is input into the position-enhanced inference model to predict the completed text and its position at the missing text. Specifically: The AI ​​inference module for missing information in the service layer includes sub-modules such as missing mask segmentation, contextual paragraph extraction, and AI candidate character / mask inference. Missing mask segmentation identifies the missing locations in the original scanned image of the ancient book (the missing pixels are assumed to be white during scanning, so only white pixels need to be extracted) and binarizes them to obtain a binary mask for missing location determination. Contextual paragraph extraction uses a combination of unsupervised clustering and prior rules. Clustering separates text columns of different font types (e.g., main text and annotation text), searches the text columns, and then merges text lines interrupted by missing information based on the linear distance between columns. Finally, text lines meeting the criteria are merged into paragraphs. The AI ​​candidate character / mask inference submodule structures all extracted paragraphs into a combination of characters, candidate characters, initial position bounding boxes, variable-length mask markers ([ULM]), and overall mask region borders required by the augmented inference model (ADR-Roberta) for the location of missing ancient texts. It then infers the complete text and its position at the missing location using the augmented inference model (ADR-Roberta) for the location of missing ancient texts in the model library. That is, the best candidate character, candidate character borders, mask region strings, and borders of each character in the mask region.

[0033] In this embodiment, the process of performing context search based on heuristic rules and unsupervised clustering for OCR results is described in [link to documentation]. Figure 4 Specifically, see (a1) to (a7): (a1) Organize the OCR results, manually remove detection errors or positions that are not considered, and fine-tune the initial position bounding box of regions with excessive detection errors.

[0034] (a2) Extract and standardize the x-coordinates of the center points of all initial position bounding boxes, convert them into a distribution with a mean of 0 and a standard deviation of 1, and then cluster them according to the x-coordinates (e.g., density-based spatial clustering (DBSCAN clustering)) to divide them into initial text column blocks.

[0035] For each initial text column block, agglomerative clustering is performed based on vertical distance. The criteria for two texts belonging to the same column are: ① The two texts are arranged vertically, and the vertical angle of the line connecting their centers is greater than t0; ② The distance between the two texts does not exceed one character height (based on the median of the global initial position bounding box); ③ The width error between the two texts is not greater than h0. After iterative agglomerative clustering, the initial text column will be divided into multiple text columns.

[0036] (a3) Text column clustering: Clustering is performed based on the average size of the characters in each text column to distinguish text columns with different font types.

[0037] For all text columns globally, calculate the average size of each text column, and then perform K-means clustering on all the obtained average sizes. The number of clusters K represents the font type (such as body text, annotation text, title text, etc. Different types of characters in ancient books generally use different font sizes). It can be specified by the user, or the minimum inter-class distance (the deviation between the average sizes of the two classes) can be set and automatically determined by the system.

[0038] (a5) Merge text columns: Merge adjacent text columns that belong to the same font type and whose fitted straight line distance is within the threshold. If there are missing areas in the merge interval, insert variable length mask markers.

[0039] Calculate the fitted line for the center points of the bounding boxes at all initial positions for each text column. Then, within each column block, merge the text columns using agglomeration. The criteria for merging two text columns are: ① The two text columns belong to the same category; ② The fitted line segments of the two text columns do not overlap vertically (y-axis); ③ The distance between the two fitted line segments is less than d0. If there are no other texts in the middle of the two text columns, they are merged directly. If there are missing areas, a [ULM] variable-length mask marker is added in the middle of the merge.

[0040] The advantage of setting two text columns as the fusion criterion is that it effectively utilizes the prior rules of the distribution of ancient texts, which can effectively expand the context and improve the accuracy of reasoning about missing information.

[0041] (a6) Paragraph context search: The global text columns are sorted from top to bottom and right to left according to their coordinates. Starting from the first text column, subsequent text columns that meet the requirements are merged into a paragraph. The criteria for determining whether two text columns belong to the same context are as follows: ① The two text columns belong to the same category; ② The first text column ends at the bottom edge of the page, or the area up to the bottom edge is a missing area, and the second text column starts at the top edge of the page, or the area up to the top edge is a missing area. The two text columns that meet the criteria are stored in a paragraph set. If there is a missing area in the middle of the combination, a [ULM] variable-length mask mark is also added.

[0042] (a7) Paragraph organization: Based on the page layout, merge columns of text of the same type that conform to the logical relationship between the top and bottom into paragraphs.

[0043] The text, initial position bounding boxes, candidate words, and variable-length mask markers of all text columns in the paragraph are organized according to a specified format to form structured data. This data is then input into the position-enhanced inference model for calculation.

[0044] This embodiment employs a five-stage (five-layer) clustering approach to enhance the effectiveness for analyzing text in ancient book layouts. The five stages correspond to steps (a2), (a3), (a4), (a5), and (a6). First, step (a2) uses DBSCAN clustering based on the common distribution patterns of ancient books to initially divide the text into multiple initial text column blocks, effectively reducing the difficulty of subsequent detailed column segmentation. Then, step (a3) ​​extracts text columns from the initial text column blocks using agglomerative methods, merging adjacent characters into the same column based on their relative positions. This method effectively utilizes the prior rules governing the distribution of the main text and interlineal annotations in ancient books, quickly distinguishing different text columns. Next, following step (a4), K-means clustering is performed on the average size of the characters in the extracted text columns across the entire page. Using the average initial position bounding box size of a single text column, rather than all initial position bounding boxes across the entire page, effectively addresses the issue of uneven character sizes in ancient books. K-means clustering is used because the number of font types on a single page of an ancient book is typically limited, usually 1-2, and K-means clustering is very effective for clustering with few categories. Next, step (a5) determines the interrupted text column based on the distance of the fitted line from the center of the text column (the interruption may be due to missing text or interruption by annotation content). This determination condition can effectively extract the broken text column and can adapt to the problem of text column tilt. Finally, step (a6) merges similar context columns according to the a priori rules of ancient books to form paragraphs, increasing the context length and improving the accuracy of inference about the missing location.

[0045] In this embodiment, the specific structure of the location-enhanced inference model is shown below. Figure 5 The position-enhanced inference model is a multi-task model based on an improved RoBERTa architecture, including an embedding layer, a Roberta encoder backbone, a decoder, and four task heads. The model directly inserts the vectors projected from candidate characters into the positions of candidate characters in the text embedding to construct new text embeddings, which include characters and variable-length mask markers. The embedding layer fuses the new text embeddings, position embeddings, candidate character embeddings, and initial position bounding box embeddings, transforming them into vectors of a set dimension. A gating fusion mechanism is used to transform the new text embeddings and initial position bounding box embeddings into input variables, which are then input into the Roberta encoder backbone. The output of the Roberta encoder backbone is used to predict candidate characters through the candidate character prediction task head and the candidate character position bounding box task head, respectively. The output of the Roberta encoder backbone is decoded by the decoder and input into the mask interval text generation task head and the mask interval text position regression prediction task head, respectively, to generate variable-length text sequences at variable-length mask markers. The decoder consists of six (optional) Transformer Decoder layers.

[0046] Specifically: (b1) In this embodiment, in the embedding layer of the position-enhanced inference model, text embedding and position embedding adopt the native Roberta embedding method, jointly embedding them into a 768 (configurable) dimensional vector. Candidate characters are first embedded into a 768 (configurable) dimensional vector using ordinary text embedding. Then, multiple vectors are concatenated and projected into a 768 (configurable) dimensional vector through a linear layer projection, replacing the position of the candidate characters in the text embedding to construct a new text embedding. The initial position bounding box embedding uses an MLP method to convert the normalized xywh format into a 768 (configurable) dimensional vector. It should be noted that the normalized xywh format is the result of a mathematical transformation that compresses the (x,y,w,h) format from pixel space to a fixed numerical range.

[0047] It should be noted that, finally, a gating fusion mechanism is used to automatically learn the weight values ​​and transform the new text embedding and the initial position bounding box embedding into 768-dimensional input variables, which are then input into the Roberta encoder backbone.

[0048] Gated blending is used, combining new text embedding and initial position bounding box embedding, to dynamically adjust the ratio of text information to layout weights, calculating the first... Gating coefficient at each position : ; in, For the first New text embedded at each position For the first The initial position bounding box is embedded at each location. For activation function, Here is the weight matrix of the gated linear layer. For the bias term of the gated linear layer, This is a vector concatenation operation.

[0049] Specifically, this embodiment will Initialize to a negative value to make the gating value close to 0 in the early stage of training, thereby ensuring that the position augmentation inference model learns the semantic features of the text and candidate characters first before introducing the initial position bounding box embedding.

[0050] Final Fusion embedding at each location The calculation is as follows: ; in, This indicates element-wise multiplication. For the first The position embeddings corresponding to each position are indexed sequentially from the original Roberta native absolute position encoding table. It can be found. For the first The type embedding corresponding to each position is obtained by looking up the original Roberta native paragraph type encoding table by paragraph identifier.

[0051] and It inherits the embedding part from the original Roberta. This fusion embedding generation method allows the model to focus more on text features when fixing semantics, and utilize layout features when fixing layout.

[0052] (b2) In this embodiment, the entire location-enhanced inference model architecture has two main output branches, whose output heads correspond to the candidate word branch and the variable-length mask branch, respectively. The candidate word branch follows the original Reberta pure encoder architecture, predicting the token probability at a single position (i.e., used to predict candidate words). The variable-length mask branch, however, employs an encoder-decoder seq2seq architecture to generate variable-length text sequences at variable-length mask markers. The output headers of the candidate word branch correspond to the candidate word prediction task header and the candidate word location bounding box task header, while the output headers of the variable-length mask branch correspond to the mask interval text generation task header and the mask interval text position regression prediction task header.

[0053] All four task heads can be implemented using MLP. The candidate character prediction task head outputs the probability distribution of the predicted character at each "candidate character position" (a probability distribution map the size of the vocabulary). The candidate character position bounding box task head outputs the predicted bounding box (xywh format) of a single character obtained from regression.

[0054] The masked interval text generation task head uses a greedy search algorithm to progressively infer the masked interval text sequence (the text at each position represents a predicted character probability distribution, returning the character with the highest probability at each position, and also returning the first position), until the [EOS] symbol is encountered. Similarly, the masked interval text position regression prediction task head infers the position sequence of the predicted position bounding box (xywh format) of the longest single character, and truncates the masked interval text sequence output by the masked interval text generation task head to the same length. Here, [EOS] is the termination position of the identifier string.

[0055] The dual-branch, four-task-head configuration combines multiple tasks, including prediction of easily confused candidate characters in ancient texts (character probability + predicted position bounding box) and prediction of consecutive missing strings (character probability sequence + predicted position bounding box sequence). Based on the full-text layout information of gating fusion text semantics and predicted position bounding boxes, it fully utilizes the contextual reasoning of the Roberta encoder and the variable-length generation capability of the decoder to achieve end-to-end prediction and predict missing layout information of ancient texts in one go.

[0056] The candidate character prediction task header and the masked region text generation task header have the same structure. For the text task header, see... Figure 6 In (a), the input sequentially passes through a linear layer (Linear), an activation function (GELU), a normalization layer (LayerNorm), and another linear layer (Linear) to obtain the output. The text task header in this embodiment uses a general MLP architecture, and the activation function chosen is GELU, which is consistent with the native Roberta encoder.

[0057] The candidate character location bounding box task header and the mask interval text position regression prediction task header have the same structure. See [link to task header]. Figure 6 In (b), the input sequentially passes through a linear layer, the GELU activation function, the Dropout regularization function, another linear layer, the GELU activation function, the Dropout regularization function, another linear layer, and the Tanh activation function to obtain the output. In this embodiment, the bounding box task head also adopts a general MLP architecture, and the output activation function is Tanh, which limits the output range to [-1, 1]. This avoids convergence difficulties caused by numerical explosion during bounding box size regression.

[0058] like Figure 6 As shown in (a), the text task header is the character classification probability in the vocab_size dimension (total number of words in the dictionary), as follows: Figure 6 As shown in (b), the bounding box task header outputs the relative parameters of the predicted bounding box, with the output dimension being 4-dimensional relative position coordinates, in the following format: ,in This is the x-coordinate offset of the center point. This is the offset of the center point's ordinate. The width is the logarithmic scaling factor. Using the height logarithmic scaling factor, for a given predicted location bounding box output by the location-enhanced inference model, it is transformed into... The absolute coordinates in the format, where The format is equivalent to the OCR model output. The format and conversion method are as follows: ; in, These are the center coordinates of the predicted bounding box of the candidate word or the initial bounding box of the variable-length mask marker. ) is the median of the width and height of the bounding box corresponding to the initial position of the entire input structured data.

[0059] (b3) Training process for location-enhanced inference models; Dataset Construction: This location-enhanced inference model task is unique, requiring character-level labeled ancient text documents as the foundation. Missing masks are then manually synthesized, and an OCR tool is used to construct the missing ancient text document dataset. The samples in the missing ancient text document dataset include: a structured input text sequence (input_text) containing text, candidate characters, and variable-length mask labels; OCR output and specially labeled initial location bounding boxes (input_box); the original text paragraphs (label_text); and the original text bounding boxes (label_box). The original text paragraphs (label_text) and original text bounding boxes (label_box) serve as text labels and location labels, respectively. Using label_text and label_box, target labels for each subtask can be generated, including: for the candidate character prediction branch, the true character label (Label_cand_char) and the true bounding box label (Label_cand_box); and for the continuous mask prediction branch, the true missing text label sequence (Label_span_text) and the true bounding box label sequence (Label_span_box).

[0060] Task Training: During the training process, the dataset of missing ancient books is used as the input sample. For samples with a small proportion of missing mask, it is necessary to randomly cover a part of the area or select a part of the text for OCR again for data augmentation.

[0061] Loss Calculation: The location-enhanced inference model is trained using a multi-task joint optimization strategy. The model consists of four sub-tasks: inference for the candidate character prediction task head and the candidate character bounding box task head (Tasks 1 and 2), and inference for the masked interval text generation task head and the masked interval text position regression prediction task head (Tasks 3 and 4). Total Loss Function Defined as the weighted sum of the losses of the four subtasks.

[0062] ; in, For candidate character classification loss, The regression loss is the candidate character location bounding box. For the text generation loss in the masked area, The regression loss is for the text location bounding box within the masked region. These are the loss weights configured by the user.

[0063] Typically, it is the cross-entropy loss function, used to measure the difference between the character probability distribution predicted by the position-enhanced inference model for each "candidate character position" and the real character label Label_cand_char; Typically, the cross-entropy loss function is used to measure the difference between the masked range text sequence generated by the decoder through the location-enhanced inference model and the true missing text label sequence Label_span_text.

[0064] The combined loss function measures the numerical and geometric differences between the predicted bounding boxes of individual characters (i.e., candidate character location box regressions) and the ground truth box labels (Label_cand_box) predicted by the location-enhanced inference model. Both are combined loss functions, measuring the overall difference between the sequence of multiple predicted location bounding boxes generated by the location-enhanced inference model for the entire text and the sequence of ground truth label boxes (Label_span_box).

[0065] In order to balance the numerical stability and geometric overlap of the regression, and A combined loss function was used: = ; That, Corresponding to or , The mean absolute error (MAE) (i.e., numerical regression loss) is calculated by determining the mean absolute error between the predicted bounding box and the ground truth bounding box coordinates. This helps in regression convergence. The full intersection-union loss (i.e., geometric similarity loss) is calculated by comparing the intersection-union loss of the predicted location bounding box and the ground truth bounding box labels. This approach better handles the alignment and aspect ratio of the predicted location bounding boxes and effectively addresses the consistency of the center point distance and aspect ratio of the bounding boxes. The weights for the mean absolute error loss, The weights for the complete intersection and union ratio loss.

[0066] This embodiment is illustrated by... To complete the text and ensure the accuracy of the document's content verification, through... By restoring the layout and ensuring the accuracy of the artistic layout, the two major values ​​of digital restoration are fully realized.

[0067] Understandably, the "content of the text" specifically refers to the textual information layer contained in ancient books, including the sequence of Chinese characters, words, phrases, and their semantic connotations. Restoration of this dimension aims to restore the reading information of ancient books, falling under the categories of textual criticism and philology for value preservation. The "artistic layout" specifically refers to the visual presentation of ancient books, including but not limited to: font style, ink characteristics, page layout, margins, paper texture, and signs of damage. Restoration of this dimension aims to restore the original appearance of ancient books, falling under the categories of bibliography and book history for value preservation.

[0068] This embodiment uses weight allocation of the total loss function. Applied to text generation tasks, Applying to the positional regression task, it achieves joint optimization and collaborative restoration of document content and artistic layout within the same model framework. Unlike traditional methods that process the two separately, it significantly improves the efficiency and accuracy of digital restoration.

[0069] In one embodiment, step five, text repair: Based on the proportion of missing text within the predicted bounding box and the information entropy of the corresponding text recognition result, a character library matching and replacement is selected, or an AI text repair model is invoked to repair the text, thereby obtaining the repaired text; specifically: The service layer's text restoration module includes two sub-modules: text replacement and AI restoration. The text replacement sub-module searches for similar fonts from the character databases (ancient book font database, ancient book page background database, and ancient book corpus) preserved during the digitization of ancient books, and replaces missing or abnormal text on the current page. The AI ​​restoration sub-module calls a GAN (Generative Adversarial Network) model to gradually infer the missing stroke pixels from the remaining text shapes. The GAN model undergoes transfer training between complete texts selected from the same ancient book and standard fonts.

[0070] The calculation of information entropy for OCR recognition results of missing characters involves characters where the predicted bounding box of the missing location intersects with the masked area, meaning that part of the text is covered by the masked area. ; in, For information entropy, The first one in the OCR results The probability of each candidate character H represents the total number of candidate characters. A value close to 0 indicates a high degree of certainty in the OCR result, while a larger value indicates a high degree of uncertainty, making the OCR recognition result unreliable.

[0071] When the defect ratio is higher than the first threshold and the information entropy is higher than the second threshold, the method of matching and replacing the characters from the pre-built ancient book font library is adopted to repair the defect. Otherwise, the AI ​​text restoration model is invoked to generate and fill in the missing pixels based on the remaining stroke pixels.

[0072] If the information entropy is low and the loss ratio is low, then the text quality is good. If the information entropy is high but the loss ratio is low, then the character may be an uncommon character. If the information entropy is low and the loss ratio is high, then the OCR may only recognize a radical. If the information entropy is high and the mask ratio is high, then it means that the loss has damaged the character's structure.

[0073] As a preferred embodiment, the AI ​​text restoration model employs CycleGAN, a style transfer model proposed by Zhu et al., used in image style transfer tasks, and subsequently applied by scholars to the generation of artistic text. The generation of ancient texts is similar to that of artistic texts, therefore it can also be used in the task of restoring ancient text fonts. Reference: Zhu JY, Park T, Isola P, et al. Unpaired Image-to-Image Translation using Cycle-ConsistentAdversarial Networks[J].IEEE, 2017.DOI:10.1109 / ICCV.2017.244.

[0074] In one embodiment, step six, background restoration: After separating the text pixels from the scanned image of the ancient book, the background pixels at the missing area are generated by calling the ancient book background restoration model based on the binary mask, and the restored text is then re-fused with the restored background; specifically: The paper background restoration module in the service layer separates the text and inputs the remaining ancient book background image into the ancient book background restoration model. The ancient book background restoration model infers the pixels at the location of the missing (or manually defined missing) areas through training features. The ancient book background restoration model is a generative AI pre-trained model, pre-trained using a book layout background library.

[0075] As a preferred embodiment, the model used for ancient book background restoration is RFRNet. RFRNet is a recurrent feature reasoning network, a model for image inpainting proposed by Li et al. It can provide the image and the mask of the region to be filled, which is similar to the task of ancient book background restoration and can be directly transferred and used. The literature source for RFRNet is: Li J, Wang N, Zhang L, et al. Recurrent Feature Reasoning for Image Inpainting[J].IEEE, 2020.DOI:10.1109 / CVPR42600.2020.00778.

[0076] Example 1 This embodiment proposes a method for digitally restoring scanned images of damaged ancient books, including the following steps: S1: Configure system basic functions, service interfaces and other parameters. Users log in to the digitization restoration system for scanned images of damaged ancient books using their work accounts (assuming they have already registered).

[0077] S2: Enter the main workbench, select New Project and import the scanned image of the ancient book to be restored; S3: OCR and Proofreading Stage: ① The OCR model is called to run character-level text detection, detecting the initial bounding boxes of all text on the page. Users can adjust the size and position of the detected results, delete, and redraw them. ② Then, the adjusted initial bounding boxes are recognized one by one, providing a confidence score and candidate characters for each, displayed as a confidence score heatmap. Users can also proofread the recognition results. For positions with uncertain results, candidate results can be retained and marked as confusing characters. In this stage, users can choose to segment the font file and store it in a font library, or export the OCR results.

[0078] S4: Missing Information Inference Stage: Includes missing mask segmentation, context paragraph extraction, and AI candidate character / mask inference: ① Perform missing mask segmentation, automatically detect missing locations in the original scanned image of the ancient book (usually requiring a pure white background during scanning), or the user can manually set the stained area as part of the missing mask. The system binarizes the entire stained template and uses the obtained binary mask as a reference image. ② Structured context extraction: Based on the initial bounding box obtained from OCR, the entire page is first divided into initial column blocks by agglomeration clustering according to the X coordinate. Each column block usually consists of one or more text columns. Then, a character-by-character vertical search is used to find individual text columns. At the same time, clustering is performed based on the average character size of each text column to distinguish between annotation text and main text. For text columns of the same type, it is determined whether the beginning and end of the two columns cross the missing part and whether the fitted straight line distance between the two text columns is within the threshold based on the reference image. The text columns are automatically merged (if they cross the missing location, a variable-length mask marker [ULM] is inserted). Finally, based on the upper and lower boundaries of the page and the distance between columns, adjacent columns in the X-axis direction are automatically merged into natural paragraphs. ③ Based on the extracted paragraphs and OCR results, structured data containing text, candidate characters, initial position bounding boxes, and variable-length mask markers is constructed. The Augmented Detailed Retrieval and Inference (ADR-Roberta) model for missing texts in ancient books is then invoked to infer the predicted complete text and its predicted position bounding box at the missing location. Users can proofread the output and selectively export the text corpus to a corpus. For continuous masked regions, the AI ​​text restoration model is invoked again after each Chinese character is identified, adjusting the probability of subsequent inferences.

[0079] S5: Text Restoration Stage: The system automatically performs image discrimination based on the predicted bounding box after information completion. The discrimination method includes calculating the proportion of missing characters within the predicted bounding box and the information entropy of the corresponding character recognition result. If the proportion of missing characters is greater than a certain threshold and the information entropy is higher than a certain threshold, a character library matching and replacement method is used. Otherwise, the system uses an AI text restoration model to complete the missing parts based on the existing characters. Users can manually select the restoration method or preview the results and generate multiple times.

[0080] S6: Background Restoration Stage: The system first binarizes and segments the character pixels, retaining only the background pixels. Then, it uses the binary mask generated in step S4 as the basis for inference. Users can also manually add missing areas to the scanned image using a drawing tool. Next, the ancient book background restoration model is called to generate pixels on the remaining background and cover them with the binary mask. Finally, the segmented text is re-integrated with the restored background to form the final restored image.

[0081] S7: Users can save the repaired images and specify corresponding groups and tags for later modification. Resources such as text, corpora, and page backgrounds generated during the repair process can be viewed, managed, and exported.

[0082] Based on the above description of the embodiments, those skilled in the art will understand that the method, system, device, and medium for digital restoration of scanned images of damaged ancient books described in this embodiment can be implemented purely in software or deployed and run on a general-purpose or dedicated computing hardware platform. Based on this essence, the technical solution of this embodiment can be specifically implemented in the form of a software product containing program instructions. This software product can be stored on various non-volatile storage media or directly deployed as a local or cloud service. The program instructions are used to cause computer devices with processing capabilities—including but not limited to personal computers, server clusters, mobile terminals, or other network devices—to execute the steps described in this embodiment.

[0083] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for digitally restoring damaged ancient books from scanned images, characterized in that, include: Import the scanned image of the ancient book to be repaired; The image is processed by OCR to obtain OCR results including the initial bounding box of each character, the recognition result and the confidence score, and a manual proofreading interface is provided. Segment the missing regions in the scanned images of ancient books to obtain a binary mask; Missing Information Reasoning: Based on the OCR results, extract and structure the context paragraphs of the page text, insert variable-length mask markers at the missing parts, and input the structured data containing text, candidate words, initial position bounding boxes and variable-length mask markers into the position enhancement reasoning model to predict the complete text at the missing parts and its predicted position bounding boxes. Text restoration: Based on the proportion of missing text within the predicted bounding box and the information entropy of the corresponding text recognition result, the character library is selected for matching and replacement or the AI ​​text restoration model is called to restore the text, thereby obtaining the restored text; Background restoration: After separating the text pixels from the scanned image of the ancient book, the background pixels of the missing area are generated by calling the ancient book background restoration model based on the binary mask, and the restored text is then re-fused with the restored background. Save and output the final repaired image.

2. The method according to claim 1, characterized in that, The extraction and structuring of the context paragraphs of the page text specifically involves: Clustering is performed based on the x-coordinate of the center point of the initial position bounding box of each character, dividing it into initial character column blocks; Within each initial text column block, clustering is performed based on the vertical distance between texts to divide the text into multiple text columns; Clustering is performed based on the average size of the text in each text column to distinguish text columns with different font types; Merge adjacent text columns that belong to the same font type and whose fitted straight-line distance is within the threshold. If there are missing areas in the merged interval, insert variable-length mask markers. Based on the page layout, columns of text of the same type that conform to the logical relationship between the top and bottom will be merged into paragraphs.

3. The method according to claim 1, characterized in that, The location-enhanced inference model is a multi-task model based on an improved RoBERTa architecture, including an embedding layer, a Roberta encoder backbone, a decoder, and four task heads; The vector projected from the candidate character is directly inserted into the position of the candidate character in the text embedding to construct a new text embedding, which includes the character and a variable-length mask marker. The embedding layer integrates new text embedding, position embedding, and initial position bounding box embedding, and converts them into vectors of a set dimension respectively; The new text embedding and the initial position bounding box embedding are transformed into input variables using a gating fusion mechanism, and then combined with the position embedding and input into the Roberta encoder backbone. The output of the Roberta encoder trunk is used to predict candidate words by passing the candidate word prediction task header and the candidate word location box task header respectively. The output of the Roberta encoder trunk is decoded by the decoder and then input into the mask interval text generation task head and the mask interval text position regression prediction task head, respectively, to generate variable-length text sequences at variable-length mask markers.

4. The method according to claim 3, characterized in that, The new text embedding vector and the initial position bounding box embedding vector are transformed into input variables using a gating fusion mechanism. Combined with the position embedding, the fused embedding is obtained and input into the Roberta encoder backbone. The fusion embedding The formula is as follows: ； ； in, For the first New text embedded at each position For the first The initial position bounding box is embedded at each location. For the first Position embedding corresponding to each position For the first Type embedding corresponding to each position For the first The gating coefficient at each position, For activation function, Here is the weight matrix of the gated linear layer. For the bias term of the gated linear layer, This is a vector concatenation operation. This indicates element-wise multiplication.

5. The method according to claim 1, characterized in that, The location-enhanced inference model is trained by jointly optimizing the loss functions of the following four tasks: candidate character classification loss, candidate character bounding box regression loss, masked interval text generation loss, and masked interval text bounding box regression loss.

6. The method according to claim 1, characterized in that, Based on the proportion of missing text within the predicted bounding box and the information entropy of the corresponding character recognition result, the text is repaired by selecting a character library matching and replacement or calling an AI text repair model. Specifically: When the defect rate is higher than the first threshold and the information entropy is higher than the second threshold, the method of matching and replacing the characters from the pre-built ancient book font library is adopted to repair the defect. Otherwise, the AI ​​text restoration model is invoked to generate and fill in the missing pixels based on the remaining stroke pixels.

7. The method according to claim 1, characterized in that, The ancient book background restoration model is an image restoration network based on a generative pre-trained model, trained using a database of ancient book page background images.

8. A system for digitally restoring scanned images of damaged ancient books, characterized in that, include: The interaction layer provides a user interface for importing images, displaying results, and receiving manual proofreading instructions; The service layer encapsulates: The OCR module is used to detect and recognize text in images, and outputs the initial bounding box of each character, the recognition result, and the confidence score. The AI ​​inference module for missing information is used to segment the missing area to obtain a binary mask, extract the context paragraphs of the page text based on the OCR results, and call the position enhancement inference model to predict the complete text at the missing location and its predicted position bounding box. The text restoration module is used to select a character library for matching and replacement or call an AI text restoration model to restore the text based on the proportion of missing parts within the predicted bounding box and the information entropy of the corresponding text recognition result, so as to obtain the restored text. The paper background restoration module is used to separate the text pixels from the scanned image of the ancient book, and then call the ancient book background restoration model based on the binary mask to generate the background pixels of the missing area, and then re-integrate the restored text with the restored background. The model layer stores the weight parameters of the OCR model, the location-enhanced inference model, the AI ​​text restoration model, and the ancient book background restoration model. The data layer stores ancient book font libraries, ancient book page background libraries, and ancient book corpora.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of classification programs, which are used by a processor to execute the method as described in any one of claims 1-7.

Images