Space-level Chinese ancient book image restoration method and related equipment

Abstract

The invention discloses a chapter-level Chinese ancient book image restoration method and related equipment, and belongs to the field of ancient book restoration. The method comprises the following steps: acquiring ancient book images, and screening out intact ancient book images and damaged ancient book images; for an intact ancient book image, simulating damage in a random mask mode, and constructing a training data pair; for the damaged ancient book image, marking the position of the incomplete character, and determining the specific content of the incomplete character; constructing a repair model based on the diffusion model, and training the repair model; and using the trained model to gradually realize the restoration of the chapter-level ancient book image in the form of an overlapped sliding window in an autoregression mode. According to the method, on the basis of the damaged positions and the damaged text content of the ancient books marked by human experts, the diffusion model technology is adopted, an autoregressive restoration method is combined, automatic restoration of the chapter-level damaged ancient book image is achieved, and the overall consistency of the font style and the background is effectively kept in the restoration process.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and ancient book restoration, and in particular to a text-level ancient Chinese book image restoration method and related equipment. Background Art

[0002] In the current information age, the widespread application of digital technology has greatly improved the efficiency of document processing, storage, analysis and dissemination.

[0003] In recent years, methods and paradigms in the fields of computer science and artificial intelligence have played an important role in digital humanities research, promoting the development of ancient book digitization, digital heritage protection, information discovery and other fields. Although these technologies have achieved remarkable results, they still face many problems and challenges in processing damaged images of ancient Chinese books. (1) Existing methods mainly repair individual Chinese characters in ancient books and directly paste the repaired character images back into the original images. This processing method easily leads to inconsistencies between the repaired area and the original image background. (2) Existing methods can usually only repair small image blocks (such as 512×512 pixels), which makes it difficult to cope with the needs of repairing large-area damaged or entire page, entire volume and other ancient book images. Summary of the Invention

[0004] In order to at least solve one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide an autoregressive text-level Chinese ancient book image restoration method based on a diffusion model and related equipment.

[0005] The first technical solution adopted by the present invention is:

[0006] A method for restoring an image of an ancient Chinese book at a paragraph level, comprising the following steps:

[0007] Data preparation: Obtain images of ancient books and filter out intact and damaged ones;

[0008] Data preprocessing: For intact ancient book images, damage is simulated by random masking to construct training data pairs; for damaged ancient book images, the location of the incomplete text is marked and the specific content of the incomplete text is determined;

[0009] Incomplete text generation: Build a restoration model based on the diffusion model and use the constructed training data to train the restoration model, so that the model can learn the writing styles of different ancient books;

[0010] Autoregressive page-level restoration: Using the trained model, high-quality restoration of ancient book images at the page level is gradually achieved through an autoregressive approach using overlapping sliding windows.

[0011] Furthermore, the data preprocessing includes:

[0012] For intact ancient book images, the images are cut into image blocks of a preset size, and the text areas of the image blocks are randomly destroyed by random masking to form "intact-damaged" data pairs;

[0013] For damaged ancient book images, the position of the damaged text is accurately marked by marking character boxes, and the marking results are recorded in a unified format. The recording format is (x 左上角 ,y 左上角 ,x 右下角 ,y 右下角 );

[0014] For the marked incomplete character positions, the content of each incomplete character is restored to obtain an image of the incomplete character repair.

[0015] Furthermore, the restoration model takes the damaged image, the damaged position mask image and the damaged character content image as input to generate an ancient book image restoration result that is highly consistent in character style and background;

[0016] The repair model includes a text position and content decoder and a U-shaped backbone network;

[0017] The text position and content decoder is used to generate a corresponding damaged text position mask map and a damaged text content map according to the marked damaged text position and content;

[0018] The input of the U-shaped backbone network consists of the damaged original image, the damaged text content image, the damaged text position mask image and a noise image, and the output is a repaired image to achieve high-quality restoration of the ancient book page.

[0019] Furthermore, the character supervision mechanism is used to train the repair model, and the loss function is:

[0020]

[0021] In the formula, VGG() represents the VGG network, x 修复图 To repair the image, x 目标图 is the target image, x 修复区域掩码 is the repair area mask map.

[0022] Furthermore, the autoregressive length-level repair includes:

[0023] Get the image of the ancient book to be restored, set the size and sliding step of the image block, initialize the set of character frames to be restored, and record the area where each character to be restored is located;

[0024] Enter the autoregressive loop to determine whether there are still unrepaired characters; if all characters have been repaired, terminate the loop; if there are still unrepaired characters, continue to execute the subsequent steps;

[0025] Count the minimum horizontal and vertical coordinates, as well as the maximum horizontal and vertical coordinates, of all currently unrepaired character boxes to form the minimum bounding rectangle of the unrepaired character boxes. Use the four corner positions of the bounding rectangle as the starting points of the candidate image blocks and calculate the number of unrepaired character boxes that are completely covered by the image block corresponding to each starting point.

[0026] Determine the starting point, move the window on the image according to the set sliding step size and direction, and generate image blocks in sequence; whenever a character box is completely covered by an image block, it is marked as repaired; if the character box is only partially covered, it is not marked as repaired;

[0027] For each generated image block, a damaged text content map and a damaged text position mask map are generated respectively;

[0028] Conduct connected domain analysis and morphological processing on the damaged text position mask map;

[0029] Save each image block, damaged text content map, damaged text position mask map and its position information to the repaired image block set until all character boxes are completely repaired and covered;

[0030] For each generated image block, the size of its damaged text content map and damaged text position mask map is uniformly adjusted to the preset standard size to adapt to the input requirements of the subsequent restoration model;

[0031] Input the image block, damaged text content map, and damaged text position mask map into the trained restoration model for inference to obtain the restored image block;

[0032] Paste the repaired image blocks back into the original image according to their actual positions in the entire ancient book image.

[0033] Furthermore, determining the starting point includes:

[0034] The candidate image block that covers the least number of unpainted character boxes is selected as the starting point of the sliding window.

[0035] Furthermore, the connected domain analysis and morphological processing of the damaged text position mask map includes:

[0036] In the horizontal and vertical directions, the mask areas with intervals below the preset threshold are connected, and a binary filling algorithm is used to automatically fill in the small holes in the mask;

[0037] The area of ​​the newly generated connected regions is counted, and only small connected regions with an area lower than the minimum character box area threshold are retained and incorporated into the mask area to ensure that the damaged background between the character boxes can be correctly repaired.

[0038] Furthermore, before inputting the image block, the damaged text content map, and the damaged text position mask map into the repair model, the following steps are also included:

[0039] The resized image blocks, damaged text position mask maps, and damaged text content maps are converted into Tensor formats and normalized.

[0040] The second technical solution adopted by the present invention is:

[0041] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a text-level image restoration method for ancient Chinese books as described above.

[0042] The third technical solution adopted by the present invention is:

[0043] A computer-readable storage medium stores at least one instruction, at least one program, a code set, or an instruction set, which is loaded and executed by a processor to implement a text-level image restoration method for ancient Chinese books as described above.

[0044] The fourth technical solution adopted by the present invention is:

[0045] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method for restoring an image of an ancient Chinese book at a text-level.

[0046] The beneficial effects of the present invention are as follows: the present invention can automatically repair damaged ancient book images at the text level based on the damaged positions and contents of the damaged text annotated by human experts, using diffusion model technology combined with an autoregressive repair method, and effectively maintain the overall consistency of the font style and background during the repair process. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0048] Figure 1 This is a schematic diagram of the process of creating an "intact-damaged" data pair in an embodiment of the present invention;

[0049] Figure 2 It is the damaged original image of the ancient book in the embodiment of the present invention;

[0050] Figure 3 This is a manually labeled schematic diagram in an embodiment of the present invention;

[0051] Figure 4 is a network architecture diagram of a repair model based on a diffusion model in an embodiment of the present invention;

[0052] Figure 5 is a schematic diagram of a text position and content decoder according to an embodiment of the present invention;

[0053] Figure 6 1 is a schematic diagram of the autoregressive paragraph-level repair process in an embodiment of the present invention;

[0054] Figure 7 This is the ancient book image finally restored in the embodiment of the present invention;

[0055] Figure 8 This is a flowchart of the steps of a method for repairing Chinese ancient book images at the paragraph level in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application. For the step numbers in the following embodiments, they are provided only for the convenience of explanation and are not intended to limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0057] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the embodiments of the present application. The singular forms of "a", "said", and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise clearly defined, words such as setting, installing, and connecting should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above words in the present invention in combination with the specific content of the technical solution.

[0058] In the description of this application, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on this application.

[0059] In the description of this application, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The terms "first" and "second" are used solely to distinguish technical features and are not to be construed as indicating or implying relative importance, or as implicitly specifying the number or order of the technical features indicated.

[0060] In the description of this application, "and / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0061] Explanation of terms:

[0062] VGG: The abbreviation of Visual Geometry Group, is a classic deep convolutional neural network (CNN) architecture.

[0063] To address existing technical issues, the present invention proposes an autoregressive, page-level automatic restoration scheme for ancient Chinese book images based on a diffusion model. This scheme can efficiently restore page-level ancient Chinese book images, effectively improving the restoration efficiency and accuracy during the ancient book digitization process. The scheme covers four main steps: data preparation, data preprocessing, ancient book image block restoration, and page-level overall restoration. In the data preparation and preprocessing stage, ancient book images with damage characteristics are first collected from the Internet, and the damaged areas and their contents are accurately annotated manually to ensure a high-quality data foundation for subsequent restoration tasks. In the ancient book image block restoration stage, the entire page of the ancient book image is cropped into image blocks of appropriate sizes according to the processing capacity of the diffusion model. Combined with the manually annotated damage location and content, targeted restoration of each image block is achieved. To address the difficulty of page-level restoration, the present invention innovatively designs an autoregressive image block splicing method. Through a sliding window strategy, the entire page of ancient book images is restored continuously and consistently, significantly improving the effect and efficiency of large-scale restoration. Compared with traditional restoration methods that mainly rely on manual labor, the restoration system proposed in this invention not only greatly shortens the restoration cycle and reduces manpower and material resources, but also ensures a high degree of consistency in style and content of the restoration results, providing strong technical support for the protection and digitization of ancient Chinese books.

[0064] Example 1

[0065] like Figure 8 As shown, this embodiment provides a method for restoring Chinese ancient book images at a paragraph level, which can not only effectively restore the overall appearance of the ancient book image, but also maintain the consistency of the writing style of the restored text with the original text. The method specifically includes the following steps:

[0066] S1. Data preparation: Obtain ancient book images and filter out intact and damaged ancient book images.

[0067] For example, images of ancient books are collected online, and intact and damaged images are manually screened. In some embodiments, if the collected electronic files of ancient books are originally in .pdf format, to improve the efficiency of subsequent data processing, viewing, and storage, images are extracted from the .pdf files using a dedicated tool and uniformly converted into .png format image files.

[0068] S2. Data preprocessing: For intact ancient book images, damage is simulated by random masking to construct training data pairs; for damaged ancient book images, the location of the incomplete text is marked and the specific content of the incomplete text is determined.

[0069] In some embodiments, step S2 specifically includes the following steps:

[0070] S21. For an intact image, this embodiment cuts it into image blocks of 512×512 pixels and randomly destroys its text area by random masking, thereby forming an “intact-damaged” data pair, such as Figure 1 Specifically, the character area is randomly masked at a ratio of 0.05-0.2 based on the position of the character box. Masking methods include all-white masking, all-black masking, and fuzzy masking to simulate various types of damage in ancient documents, providing rich data support for subsequent model training.

[0071] S22. For damaged pictures, e.g. Figure 2 In this embodiment, human experts mark the position of the damaged text accurately by marking character boxes, and record the marking results in a unified format. The recording format is (x 左上角 ,y 左上角 ,x 右下角 ,y 右下角 ).

[0072] S23: For the damaged character positions marked in step S22, human experts are further required to consult relevant literature to verify and restore the content of each damaged character. Finally, the damaged character positions and their corresponding contents obtained by manual marking are as follows: Figure 3 shown.

[0073] S3. Generation of incomplete text: Build a restoration model based on the diffusion model, and use the constructed training data to train the restoration model so that the model can learn the writing styles of different ancient books.

[0074] In some embodiments, a generative network architecture based on a diffusion model is designed, which integrates a text position and content decoder, a U-shaped backbone network, and a character-level supervision mechanism. Through this design, damaged images, damaged position mask images, and damaged character content images can be used as input to generate ancient book image restoration results that are highly consistent in character style and background. The specific network structure is as follows: Figure 4 shown.

[0075] 1) Text location and content decoder

[0076] The text position and content decoder generates the corresponding damaged text position mask map and damaged text content map based on the marked damaged text position and content. This process provides accurate spatial and semantic information for subsequent character generation, as shown in the figure. Figure 5 shown.

[0077] 2) U-shaped backbone network

[0078] As an optional implementation, the structure diagram of the U-shaped backbone network is shown in Table 1. The size of the image is expressed as H×W.

[0079] Table 1 U-shaped backbone network structure

[0080]

[0081] The input to the U-shaped backbone network consists of a damaged original image of the ancient book, an image of the damaged text content, a mask of the damaged text location, and a noise map. The damaged original image and the noise map are three-channel images, while the damaged text content image and the mask of the damaged text location are single-channel images. All images are stacked along the channel dimension to form an eight-channel input vector. This eight-channel vector is fed into the U-shaped backbone network for denoising, ultimately outputting a three-channel repaired image, achieving high-quality restoration of the ancient book page.

[0082] 3) Model training

[0083] In order to further improve the accuracy and style consistency of the characters in the repaired area, this embodiment introduces a character supervision mechanism. Specifically, the character supervision mechanism uses a pre-trained VGG network to extract feature representations of the repaired area in the repaired image and the target image, respectively, and constrains the model by calculating the feature difference between the two in the repaired character area. The calculation formula is:

[0084]

[0085] In actual operation, the repaired image and the target image are respectively input into the VGG network to extract features, and then the repair area mask map is used to calculate only the features within the mask area, thereby achieving efficient supervision of the characters in the repair area.

[0086] As a specific implementation, the training parameters are as follows:

[0087] a) Iterative Algebra: 150

[0088] b) Image resolution: 512×512

[0089] c) Optimizer: AdamW

[0090] d) Learning rate: 0.0001

[0091] e)Weight decay:0.00001.

[0092] S4. Autoregressive page-level restoration: Using the trained model, we gradually restore the page-level ancient book images in the form of overlapping sliding windows using an autoregressive approach.

[0093] In some embodiments, see Figure 6 , step S4 specifically includes the following steps:

[0094] S41. Obtain the size information of the ancient book image to be repaired, set the size and sliding step of the image block, initialize the set of character frames to be repaired, and record the area where each character to be repaired is located.

[0095] S42: Enter an autoregressive loop and first determine whether there are any unrepaired characters. If all characters have been repaired, terminate the loop; if there are any unrepaired characters, continue with the subsequent steps.

[0096] S43: Count the minimum horizontal and vertical coordinates, as well as the maximum horizontal and vertical coordinates, of all currently unrepaired character boxes to form a minimum bounding rectangle of the unrepaired character boxes. Use the top-left, top-right, bottom-left, and bottom-right corners of this bounding rectangle as starting points for the candidate image blocks, and calculate the number of unrepaired character boxes that can be completely covered by the image block corresponding to each starting point.

[0097] S44. In order to ensure that the initial image block has enough complete Chinese characters for the model to use as a reference, this embodiment selects the candidate image block that covers the least number of unrepaired character boxes as the starting point of the sliding window.

[0098] S45. From the selected starting point, the window is moved across the image according to the set sliding step size and direction, generating image blocks in sequence. Whenever a character box is completely covered by an image block, it is marked as repaired; if the character box is only partially covered, it is not marked as repaired.

[0099] S46, see Figure 5 ,For each generated image block, a damaged text content map and a damaged text position mask map are produced respectively.

[0100] S47. Further connected domain analysis and morphological processing are performed on the damaged text location mask. Specifically, the masked regions with spacing below a preset threshold are connected horizontally and vertically, and a binary filling algorithm is used to automatically fill in small holes in the mask. The areas of the newly generated connected regions are further calculated, and only small connected regions with an area below the minimum character box area threshold are retained and incorporated into the masked region to ensure that the damaged background between the character boxes is correctly repaired.

[0101] S48: Save each damaged image (i.e., image block), damaged text content image, damaged text position mask image, and its position information to the repaired image block set. Repeat the above steps until all character boxes are completely repaired.

[0102] S49. For each damaged image (i.e., image block) generated, the size of its damaged text content image and damaged text position mask image is uniformly adjusted to a preset standard size (512×512 pixels) to adapt to the input requirements of the subsequent repair model.

[0103] S410: Convert the resized image block damage map, damaged text location mask map, and damaged text content map into Tensor format and normalize them. These tensor data are then loaded onto a designated computing device (e.g., a GPU) to improve computational efficiency in the subsequent repair process.

[0104] S411. Use the repair model trained in step S3 to infer the input damaged image, damaged text content image and damaged text position mask image to obtain a repaired image block.

[0105] S412, paste the repaired image block back to the original image according to its actual position in the entire ancient book image. Then, extract the next image block from the repaired image block set generated in S48, and cut it out from the updated original image, and continue the subsequent repair operation. By looping the above steps, the repair and splicing of all the areas to be repaired are gradually completed, and finally the complete ancient book image after repair is obtained. The final repair result is as follows Figure 7 shown.

[0106] In summary, the method of the present invention has at least the following advantages and beneficial effects compared to the prior art:

[0107] (1) Compared with previous methods that only repair single characters in ancient books, the method proposed in this paper can repair damaged ancient book images at the image block level. This method fully utilizes the intact Chinese characters in the damaged image as a style reference, thereby generating repaired character images with consistent style and natural integration with the background. At the same time, due to the introduction of a character supervision mechanism, this method shows significant performance advantages, especially in terms of the accuracy of Chinese character repair.

[0108] (2) This paper innovatively proposes an autoregressive approach to ancient book image restoration, extending the previously block-level restoration approach to full-text restoration. By generating a damaged text content map and a position mask map, combined with morphological operations, this approach can accurately locate and restore Chinese characters and backgrounds throughout an entire page of an ancient book, achieving a higher level and wider range of ancient book image restoration.

[0109] (3) The present invention can efficiently complete the restoration of ancient book images, achieving a complete restoration of an ancient book image in just a few minutes. Compared with traditional restoration methods that rely on manual labor and often take days or even months, the present invention greatly improves restoration efficiency and significantly reduces reliance on manual labor, providing strong technical support for the preservation and digitization of ancient books.

[0110] (4) The present invention has good flexibility and portability and can be easily deployed on operating systems such as Windows 10 and Ubuntu 18.04. In addition, the solution facilitates algorithm optimization and integration, and the overall implementation and application process is simple and efficient.

[0111] Example 2

[0112] An embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the following Figure 8 A text-level image restoration method for ancient Chinese books is shown in FIG.

[0113] It is understood that the memory may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the data storage area may store data created based on the use of the server, etc.

[0114] The processor may include one or more processing cores. The processor utilizes various interfaces and circuits to connect various components within the server. It executes various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, as well as accessing data stored in memory. Optionally, the processor may be implemented using at least one of the following hardware forms: digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor may integrate one or a combination of a central processing unit (CPU) and a modem. The CPU primarily processes the operating system and application programs, while the modem handles wireless communications. It is understood that the modem may not be integrated into the processor and may be implemented separately via a single chip.

[0115] Since the electronic device is an electronic device corresponding to a text-level Chinese ancient book image restoration method in an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0116] Example 3

[0117] An embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 8 A text-level image restoration method for ancient Chinese books is shown in FIG.

[0118] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0119] Since the storage medium is a storage medium corresponding to a text-level Chinese ancient book image restoration method in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0120] Example 4

[0121] In some possible implementations, various aspects of the methods of the embodiments of the present invention may also be implemented in the form of a program product, which includes program code. When the program product is executed on a computer device, the program code is used to cause the computer device to perform the steps of the method for restoring an image of an ancient Chinese book at a text level according to various exemplary embodiments of the present application as described above in this specification. The executable computer program code or "code" used to perform the various embodiments may be written in a high-level programming language such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0122] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0123] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0124] The above embodiments are intended only to illustrate the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent changes or modifications made based on the essence of the present invention are intended to be covered by the scope of protection of the present invention.

Claims

1. A method for restoring Chinese ancient book images at the paragraph level, characterized in that: The following steps are involved: Data preparation: Obtain images of ancient books and filter out intact and damaged ones; Data preprocessing: For intact ancient book images, damage is simulated by random masking to construct training data pairs; For damaged images of ancient books, mark the location of the incomplete text and determine the specific content of the incomplete text; Incomplete text generation: Build a restoration model based on the diffusion model and use the constructed training data to train the restoration model, so that the model can learn the writing styles of different ancient books; Autoregressive page-level restoration: Using the trained model, we gradually restore the page-level ancient book images in the form of overlapping sliding windows using an autoregressive approach.

2. The method for restoring an image of an ancient Chinese book at a text level according to claim 1, characterized in that: The data preprocessing includes: For intact ancient book images, the images are cut into image blocks of a preset size, and the text areas of the image blocks are randomly destroyed by random masking to form "intact-damaged" data pairs; For damaged ancient book images, the location of the incomplete text is marked by marking character frames, and the marking results are recorded in a unified format; For the marked incomplete character positions, the content of each incomplete character is restored to obtain an image of the incomplete character repair.

3. The method for restoring ancient Chinese book images at the text level according to claim 1, characterized in that: The restoration model takes the damaged image, the damaged position mask image and the damaged character content image as input, and generates an ancient book image restoration result that is consistent in character style and background; The repair model includes a text position and content decoder and a U-shaped backbone network; The text position and content decoder is used to generate a corresponding damaged text position mask map and a damaged text content map according to the marked damaged text position and content; The input of the U-shaped backbone network consists of the damaged original image, the damaged text content image, the damaged text position mask image and a noise image, and the output is a repaired image.

4. The method for restoring an image of an ancient Chinese book at a paragraph level according to claim 1, characterized in that: The character supervision mechanism is used to train the repair model, and the loss function is: In the formula, VGG() represents the VGG network, x 修复图 To repair the image, x 目标图 is the target image, x 修复区域掩码 is the repair area mask map.

5. The method for restoring an image of an ancient Chinese book at a paragraph level according to claim 1, characterized in that: The autoregressive length-level repair includes: Get the image of the ancient book to be restored, set the size and sliding step of the image block, initialize the set of character frames to be restored, and record the area where each character to be restored is located; Enter the autoregressive loop to determine whether there are still unrepaired characters; if all characters have been repaired, terminate the loop; If there are any unrepaired characters, continue with the subsequent steps; Count the minimum horizontal and vertical coordinates, as well as the maximum horizontal and vertical coordinates, of all currently unrepaired character boxes to form the minimum bounding rectangle of the unrepaired character boxes. Use the four corner positions of the bounding rectangle as the starting points of the candidate image blocks and calculate the number of unrepaired character boxes that are completely covered by the image block corresponding to each starting point. Determine the starting point, move the window on the image according to the set sliding step size and direction, and generate image blocks in sequence; whenever a character box is completely covered by an image block, it is marked as repaired; For each generated image block, a damaged text content map and a damaged text position mask map are generated respectively; Conduct connected domain analysis and morphological processing on the damaged text position mask map; Save each image block, damaged text content map, damaged text position mask map and its position information to the repaired image block set until all character boxes are completely repaired and covered; For each generated image block, uniformly adjust the size of its damaged text content map and damaged text position mask map to a preset standard size; Input the image block, damaged text content map, and damaged text position mask map into the restoration model for inference to obtain the restored image block; Paste the repaired image blocks back into the original image according to their actual positions in the entire ancient book image.

6. The method for restoring an image of an ancient Chinese book at a paragraph level according to claim 5, characterized in that: Determining the starting point includes: The candidate image block that covers the least number of unpainted character boxes is selected as the starting point of the sliding window.

7. The method for restoring an image of an ancient Chinese book at a paragraph level according to claim 5, characterized in that: The connected domain analysis and morphological processing of the damaged text position mask map includes: In the horizontal and vertical directions, the mask areas with intervals below the preset threshold are connected, and a binary filling algorithm is used to automatically fill in the small holes in the mask; The area of ​​the newly generated connected regions is counted, and only small connected regions with an area lower than the minimum character box area threshold are retained and incorporated into the mask area to ensure that the damaged background between the character boxes can be correctly repaired.

8. The method for restoring an image of an ancient Chinese book at a paragraph level according to claim 5, characterized in that: Before inputting the image block, the damaged text content map, and the damaged text position mask map into the repair model, the following steps are also included: The resized image blocks, damaged text position mask maps, and damaged text content maps are converted into Tensor formats and normalized.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

Images