A method, device, equipment and medium for restoring digitized ancient Chinese book images

Abstract

The present invention discloses a method, device, equipment, and medium for digital restoration of ancient Chinese book images, wherein the method includes the following steps: obtaining an image of an ancient book with missing text and original text data corresponding to the image of the ancient book; using a preset Chinese character detection model to obtain the positioning frame coordinates of all text on the ancient book image, checking the missing positions on the original image based on the positioning frame coordinates and the original text data, and obtaining the positions of the text that needs to be repaired; obtaining the writing style in the ancient book image, and generating a text image of the corresponding writing style based on the text that needs to be repaired; performing pixel-level segmentation processing on the generated text, and pasting the text image at the corresponding position on the ancient book image based on the obtained position of the repaired text. The present invention is the first to propose a technical solution for digital restoration of ancient books, which can effectively save a lot of manpower and time costs required for digital restoration of ancient books. The present invention can be widely used in the field of digital protection of ancient books.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and digital protection of ancient books, and in particular to a method, device, equipment and medium for repairing digitized images of ancient Chinese books. Background Art

[0002] Ancient books have always held immeasurable historical and cultural value. Ancient books handed down through history are generally handwritten or printed copies, mostly preserved in paper form or on stone tablets. In recent years, with the advancement of internet technology and document digitization, many ancient books have been digitized through scanning, photographing, and other methods for preservation, dissemination, and sharing online. However, due to improper conservation or other historical reasons, pages of ancient books may suffer from damage, such as holes, tears, or faded content. These damages are also reflected in scanned electronic documents. Therefore, restoring images of these damaged ancient books would be of great historical significance and social value.

[0003] There are two problems to be solved in the restoration of ancient book images. (1) The location of the damage in each ancient book is different. If the handwritten electronic handwriting font is manually added in the form of a sticker to complete the corresponding location, the labor cost will be too high. (2) The handwriting style of each ancient book is different. If the handwriting style of each ancient book is manually learned and the corresponding Chinese characters are written according to the original content, the manpower and time cost will be too high and unrealistic. Therefore, the restoration of digital ancient book images requires an automated process to save manpower and time costs, but there is currently a lack of corresponding technical solutions. Summary of the Invention

[0004] In order to at least solve one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide a method, device, equipment and medium for digitizing Chinese ancient book image restoration based on defect adaptive positioning and deep adversarial generative network.

[0005] The first technical solution adopted by the present invention is:

[0006] A method for restoring a digitized ancient Chinese book image comprises the following steps:

[0007] Obtaining images of ancient books with missing text and the original text data corresponding to the images of the ancient books;

[0008] Use the preset Chinese character detection model to obtain the positioning frame coordinates of all characters on the ancient book image, check the missing positions on the original image based on the positioning frame coordinates and the original text data, and obtain the positions of the characters that need to be repaired;

[0009] Obtain the writing style in the ancient book image, and generate a text image of the corresponding writing style according to the text that needs to be repaired;

[0010] The generated text is segmented at the pixel level, and the text image is pasted on the corresponding position of the ancient book image according to the position of the obtained patched text.

[0011] Furthermore, the method adopts a preset Chinese character detection model to obtain the positioning frame coordinates of all characters on the ancient book image, checks the missing positions on the original image based on the positioning frame coordinates and the original text data, and obtains the positions of the characters that need to be repaired, including:

[0012] Use deep learning Chinese character detection to obtain the positioning frame coordinates of all characters on ancient book images;

[0013] Based on the obtained positioning frame coordinates, all positioning frames are clustered using the K-Means clustering algorithm and the DBI (Davies-Bouldin) index optimization algorithm to obtain the positioning frames of all texts belonging to the same column, and the positioning frames are arranged according to the original text order;

[0014] Compare the number of positioning boxes in each column of the ancient book image with the number of characters in the current column of the original text to filter out columns with unequal numbers of characters, which are considered to be missing columns. For each missing column that is filtered out, randomly select a column with the same number of original characters from other complete columns as a reference column, and use the IoU matching algorithm to calculate the missing position.

[0015] Furthermore, the obtained positioning frame coordinates are clustered using the K-Means clustering algorithm and the DBI index optimization algorithm, including:

[0016] Calculate the midpoint coordinates (x 中点 ,y 中点 );

[0017] According to the x coordinates of the points in all positioning boxes 中点 , clustering was performed using the K-Means clustering algorithm, combined with the DBI index optimization algorithm to obtain the best clustering effect;

[0018] The midpoint y coordinates of the positioning boxes belonging to the same column are sorted in ascending order using the quick sort algorithm so that the boxes in each column are arranged in the order of the original text.

[0019] Furthermore, for each missing column that is filtered out, a column with the same number of original characters is randomly selected from other complete columns as a reference column, and the IOU matching algorithm is used to calculate the missing position, including:

[0020] Note that the positioning box for the missing column is The reference column's positioning box is Calculate the IOU between boxes;

[0021] Position the box according to each reference column Position a box with each missing column The IOU between them is used to calculate the score value, which is used to filter the specific missing position.

[0022] Furthermore, the calculation formula of the score value is:

[0023]

[0024]

[0025] Where N is the number of positioning boxes of the missing column, and M is the number of positioning boxes of the reference column; The x-coordinate of the positioning box for the j-th potentially missing column, The x coordinate of the positioning box for the i-th reference column; x offset is the position difference between the reference column and the current potential missing column; The offset x-coordinate of the positioning box for the i-th reference column, that is, the reference column simulation is offset to the position of the potential missing column model for calculating IOU; is the y coordinate of the i-th positioning box in the potential missing column, The y coordinate of the positioning box for the jth reference column; P i The i-th positioning box of the reference column corresponds to the missing column to determine whether there is a missing mark, score i When P ≥ 0.1 i =1, indicating no missing, score i <0.1 i = 0, indicating that there is a missing column. Its specific physical meaning is: use the i-th frame of the reference column and all the frames of the potential missing column to calculate the IOU and sum them. If the frame has a certain degree of overlap with the frame of the potential missing column (i.e. score i ≥0.1), it is considered that there is no missing column in the same position. If the box does not overlap with any box of the potential missing column or the overlap is very low (i.e. score i <0.1), it means that the potential missing column is missing at this position, so the position of the missing can be recorded (j col ,u), indicating that the jth col The i-th word is missing in the column, j col The current potential deletions are listed in the subscripts throughout the figure.

[0026] Furthermore, the step of obtaining the writing style in the ancient book image and generating a text image of the corresponding writing style based on the text to be repaired is achieved by a deep adversarial generative network;

[0027] The deep adversarial generative network includes a generator module, a discriminator module and a style extraction module;

[0028] The deep adversarial generative network is trained in the following way:

[0029] Get single-word image data of multiple categories of ancient books, and use one of them as the content template image I src (i.e. the font is a standard template font), and the rest of the categories are used as style reference pictures I ref Input the network for training; first train the discriminator module and then train the generator module:

[0030] Training of the discriminator module: Use the style extraction module to extract I ref The style feature vector Encode I using the content encoder in the generator module src The content features get the feature vector Will and Input the decoder in the generator module for fusion and output the generated image I fake ; will I fake and I fake Input the discriminator and output the probability P that each picture may be a real picture fake and P ref ; Use P fake and P ref Calculate the adversarial loss and gradient accumulation loss; finally, add all losses together and use back gradient propagation to supervise and optimize the parameters of the discriminator module and style extraction module;

[0031] Training the generator module: Use the style extraction module to extract I ref and I src The style feature vector and Encode I using the content encoder in the generator module src The content features of Will and Input the decoder in the generator module for fusion and output the generated image I fake ;Will and Input the decoder in the generator module for fusion and output the generated image I rec ; will I fake The content encoder of the input generator module outputs the content feature vector of the forged image Will I fake and I rec Input the discriminator and output the probability P that each picture may be a real picture fakeand P rec ; Use P fake and P rec Calculate the adversarial loss function and the gradient accumulation loss function; use I rec and I src Calculate the reconstruction loss; use and Calculate the reconstruction loss; finally, add up all the losses and use back gradient propagation to supervise and optimize the parameters of the generator module and style extraction module;

[0032] The moving average strategy is used to migrate the parameters of the generator module and the style extraction module to two other generator modules and style extraction modules with the same structure to achieve momentum update of the parameters.

[0033] Furthermore, the pixel-level segmentation processing is performed on the generated text, and the text image is pasted at the corresponding position on the ancient book image according to the obtained position of the repaired text, including:

[0034] Text Picture I gen Perform binarization to obtain image I bin , get picture I bin The position of the black pixel is used as the position mask of the text to segment the text image I gen Pixels belonging only to the text part char ;

[0035] Calculate the width w and height h of the pasted picture according to the position coordinates of the patched text, and set the pixel Pix char Scale it according to w and h, and paste it into the picture of the original missing ancient book according to its pixel-level position.

[0036] The second technical solution adopted by the present invention is:

[0037] A digital Chinese ancient book image restoration device, comprising:

[0038] A data acquisition module is used to acquire images of ancient books with missing text and the original text data corresponding to the images of the ancient books;

[0039] The Chinese character detection module is used to obtain the positioning frame coordinates of all characters on the ancient book image using a preset Chinese character detection model, and to check the missing positions on the original image based on the positioning frame coordinates and the original text data to obtain the positions of the characters that need to be repaired;

[0040] The style generation module is used to obtain the writing style in the ancient book image and generate a text image of the corresponding writing style according to the text to be repaired;

[0041] The text repair module is used to perform pixel-level segmentation on the generated text and paste the text image at the corresponding position on the ancient book image according to the position of the repaired text.

[0042] The third technical solution adopted by the present invention is:

[0043] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a method for restoring a digitized image of an ancient Chinese book as described above.

[0044] The fourth technical solution adopted by the present invention is:

[0045] A computer-readable storage medium stores at least one instruction, at least one program, a code set, or an instruction set, which is loaded and executed by a processor to implement a method for restoring a digitized image of an ancient Chinese book as described above.

[0046] The fifth technical solution adopted by the present invention is:

[0047] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above-mentioned method for restoring a digitized ancient Chinese book image.

[0048] The present invention offers the following beneficial effects: It is the first to propose a technical solution for the restoration of digitized ancient books, effectively reducing the significant manpower and time costs associated with such restoration. Furthermore, the present invention adaptively detects missing text in images of ancient books and calculates the specific coordinates of the missing locations, resulting in a robust positioning algorithm that can handle a variety of situations. Furthermore, the present invention can generate text in corresponding styles based on the writing styles of different ancient books, providing a diverse approach and significantly increasing the degree of restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0050] Figure 1 This is a flowchart of the steps of a method for restoring a digitized Chinese ancient book image in an embodiment of the present invention;

[0051] Figure 2 Schematic diagram of the coordinate format of the single-word detection positioning frame in an embodiment of the present invention;

[0052] Figure 3 is a schematic diagram of an ancient book picture with partial damage in an embodiment of the present invention;

[0053] Figure 4 2 is a schematic diagram of the result of performing text detection on an ancient book image in an embodiment of the present invention;

[0054] Figure 5 is a schematic diagram of the result of clustering the centers of the positioning boxes according to an embodiment of the present invention;

[0055] Figure 6 1 is a schematic diagram of IOUs between positioning boxes in an embodiment of the present invention;

[0056] Figure 7 Schematic diagram of IOU matching in an embodiment of the present invention;

[0057] Figure 8 2 is a schematic diagram of an ancient book image after restoration in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0059] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0060] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0061] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0062] Explanation of terms:

[0063] DBI index: Davies-Bouldin Index, is an indicator used to evaluate the quality of clustering algorithms.

[0064] Image processing is part of computer vision technology. In recent years, computer technologies related to machine learning and deep learning have flourished, among which text detection technology, text recognition technology and style font generation technology based on deep learning have achieved considerable results. These technologies can repair incomplete pictures of ancient books, such as using machine learning algorithms to locate and calibrate missing content; or using style font generation technology to generate style fonts for Chinese characters of specified content by learning the different writing styles of each ancient book, eliminating the cost of manually learning font writing styles. To this end, the present invention proposes a complete set of digital Chinese ancient book image restoration technology solutions, which combines traditional image processing, machine learning and deep learning technologies to solve the above two fundamental problems of ancient book image restoration in an end-to-end scenario, so that the technical solution can be deployed on a computer platform and put into practical use. The specific contents are as follows:

[0065] Example 1

[0066] like Figure 1 As shown, this embodiment provides a method for restoring a digitized image of an ancient Chinese book, comprising the following steps:

[0067] S1. Obtaining an image of an ancient book with missing text and the original text data corresponding to the image of the ancient book;

[0068] S2. Using a preset Chinese character detection model to obtain the positioning frame coordinates of all characters on the ancient book image, and checking the missing positions on the original image based on the positioning frame coordinates and the original text data to obtain the positions of the characters that need to be repaired;

[0069] S3, obtaining the writing style in the ancient book image, and generating a text image of the corresponding writing style according to the text to be repaired;

[0070] S4. Perform pixel-level segmentation processing on the generated text, and paste the text image at the corresponding position on the ancient book image according to the obtained position of the repaired text.

[0071] The method of this embodiment uses deep learning-based Chinese character detection and traditional image processing technology to adaptively detect the missing positions of ancient book images and complete the positioning, uses a deep adversarial generative learning model to learn and generate font styles, and places the generated text with the corresponding writing style in the original ancient book image according to the positioning to complete the restoration.

[0072] The above method is explained in detail below with reference to the accompanying drawings and specific embodiments.

[0073] This embodiment provides a method for restoring digitized ancient Chinese book images based on defect adaptive localization and a deep adversarial generative network, including the following steps:

[0074] Step 1: Data preparation.

[0075] As an implementation method, electronic version files of ancient books, such as Dunhuang manuscripts, are searched on the Internet. Specifically, the electronic version file format to be found is .pdf, which is convenient for viewing, storing and extracting ancient book images therein.

[0076] Step 2: Data preprocessing.

[0077] After finding missing images of ancient books, the original text corresponding to the images is searched and recorded on the Internet. In some embodiments, the missing images and the original text are named the same; in addition, the individual words in the entire ancient book image are segmented to form a single word dataset for different ancient books.

[0078] As a specific implementation, step 2 specifically includes the following steps:

[0079] Step 2.1: The manually selected ancient book images are stored in .png format, and the original text of the ancient book is stored in .txt format. The ancient book images are named according to their own unique sutra volume number. The names of the ancient book images and the original text record files are the same, such as "Golden Light Sutra Volume 2.png" and "Golden Light Sutra Volume 2.txt".

[0080] Step 2.2: For each incomplete volume of ancient books, at least one incomplete picture and one complete picture of the ancient books need to be intercepted, and the corresponding original text content is obtained at the same time. In this embodiment, there are a total of 28 volumes of ancient books.

[0081] Step 2.3: Select a complete image of an ancient book from each volume, cut and save the text in it, and form a single-word dataset for each ancient book.

[0082] Step 3: Adaptive search and location of missing text.

[0083] Specifically, the positioning frame coordinates of all the characters on the ancient books are obtained through deep learning Chinese character detection technology, and the K-Means clustering algorithm, DBI (Davies-Bouldin) index optimization algorithm and IOU matching algorithm are used to check the missing positions on the original image and locate the repaired characters based on the positioning frame coordinates.

[0084] As an optional implementation, step 3 specifically includes the following steps:

[0085] Step 3.1: Training a single Chinese character detection model. This embodiment uses the existing yolov5s model and the ancient book single character dataset MTHv2 for training to obtain a single character detector for Chinese characters in ancient book images.

[0086] Step 3.2: For a partially incomplete picture of an ancient book (the rest of the picture is intact), use the single word detection model in step 3.1 to detect all the words on the picture and obtain the positioning box of each word in the format of (x 左上角 ,y 左上角 ,x 右下角 ,y 右下角 ),like Figure 2 shown.

[0087] For ease of description and understanding, this embodiment uses a damaged ancient book picture named test.png as an aid. Figure 3 is the original image; the text detection example effect is as follows Figure 4 As shown, the positioning frame and coordinate information corresponding to each Chinese character are obtained.

[0088] As an optional implementation, all the text positioning boxes are stored in a specified path (the path can be arbitrarily specified in the code) and named "image name_detect". txt file (ie test_detect.txt), the format of which is "x 左上角 y 左上角 x 右下角 y 右下角 confidence level".

[0089] All boxes are arranged in a disordered manner in the file, so the next step is to rearrange the positions of the boxes to obtain the positioning boxes of the text belonging to the same column.

[0090] Step 3.3: Read all the positioning box coordinates stored in the .txt file in step 3.2, cluster all the boxes using the K-Means clustering algorithm and the DBI (Davies-Bouldin) index optimization algorithm to obtain the positioning boxes of all the text belonging to the same column, and use the quick sort algorithm to arrange the positioning boxes in the order of the original text.

[0091] In some embodiments, step 3.3 specifically includes the following:

[0092] Step 3.3.1: Calculate the midpoint coordinates of all positioning boxes and save the midpoint coordinates of each box. The midpoint coordinates are calculated as follows:

[0093]

[0094] Step 3.3.2: Extract the x coordinates of all the points in the positioning frame separately 中点 , use K-Means clustering algorithm for clustering, combined with DBI (Davies-Bouldin) index optimization algorithm to obtain the best clustering effect. The algorithm pseudo code is as follows:

[0095]

[0096]

[0097] Among them, the K-Means clustering algorithm needs to determine the clustering subject and the number of clustering categories. In this algorithm, the clustering subject is x 中点 The number of cluster categories is the number of columns in the current incomplete ancient book image. Since it is impossible to predict how many columns there are in the missing ancient book image, it is necessary to set col best As the number of columns, and calculated through iteration. Labels is the result of clustering, which is a one-dimensional array, in which each element represents which column a certain box belongs to. For example, the label of the first column is 0, the label of the second column is 1, the label of the third column is 2, and so on; at this time, if labels = [1,3,0,4,0,1…], it means that the first positioning box belongs to the second column, the second positioning box belongs to the fourth column, the third positioning box belongs to the first column, and so on. The smaller the DBI index, the better the clustering result, and the closer it is to the actual cluster distribution. Therefore, through the comparison in the algorithm, we can get the optimal number of clustering columns corresponding to the minimum DBI, which is col best , and it is also considered that this number is the number of columns of the incomplete ancient book image itself. Finally, the optimal col best Re-perform KMeans clustering to obtain labels fAs the final clustering result. Therefore, Algorithm 1 can adaptively calculate the number of columns of the current incomplete ancient book image and complete the clustering. The clustering result of test.png is as follows Figure 5 As shown, we can use labels f Groups together the filters for all boxes belonging to the same column.

[0098] Step 3.3.3: Use the quick sort algorithm to sort the y coordinates of the midpoints of the positioning boxes belonging to the same column in ascending order so that the boxes in each column are arranged in the order of the original text.

[0099] Step 3.4: Compare the number of positioning boxes in each column of the image with the number of words in the current column of the original text. Filter out columns with unequal word counts, which are considered missing columns. For each missing column filtered out, randomly select a column with the same number of original words from other complete columns as a reference column and use the IoU matching algorithm to calculate the missing position.

[0100] In some embodiments, step 3.4 specifically includes the following:

[0101] Step 3.4.1: Note that the location box for the missing column is The reference column's positioning box is Calculate the IOU (Intersection over Union) between boxes, IOU is as follows Figure 6 As shown, the calculation formula is:

[0102]

[0103] Step 3.4.2: Calculate each reference column positioning box Position a box with each missing column The IOU between them is calculated and the score is used to filter the specific missing position:

[0104]

[0105] See also Figure 7 Since the reference column and the missing column do not overlap, the IOU cannot be calculated directly. The reference column needs to be offset to a position close to the missing column. Therefore, x is calculated first. offset As the position difference between the reference column and the current missing column. In formula (8), N is the number of frames in the missing column, and M is the number of frames in the reference column. The offset coordinates of each positioning frame in the reference column are obtained by formula (9): Then calculate the IOU. Formula (10) calculates the score of each positioning box in the reference column i , if score iIf ≥0.1, it means that the i-th frame of the reference column matches the frame of the missing column in the same row, indicating that there is no missing font at the corresponding position of the missing column; otherwise, it means that there is a missing font at that position and the position is recorded (i row ,i col ). The recorded location (i row ,i col ) means that the i col Column missing i row words.

[0106] Step 3.5: The row and column positions of the missing characters have been calculated in step 3.4 (i row ,i col ), this step uses (i row ,i col ) Calculate the specific coordinates of the pre-filled box of the missing text, the format is (x 左上角 ,y 左上角 ,x 右下角 ,y 右下角 ). The embodiment of the present invention calculates the i-th row The average value of the y coordinates of the upper left corner and lower right corner of all the positioning boxes in the row is used as the two x coordinates of the missing box (i.e. x 左上角 and x 右下角 );Calculate the i-th col The average value of the y coordinates of the upper left corner and lower right corner of all the positioning boxes is used as the two y coordinates of the missing box (i.e. y 左上角 and y 右下角 ).

[0107] Step 4: Generate style fonts.

[0108] In some embodiments, the single-word data cut out in step 2 is input into a deep adversarial generative network for training to learn the writing styles of different ancient books, so as to generate a text picture with specified content and style given a style picture and a text content picture.

[0109] As some optional implementations, the deep adversarial generative network includes a generator module, a discriminator module and a style extraction module, which generates a picture with the style and content by inputting a picture with specified content and a picture with a specified style.

[0110] Specifically, in step 2, the single-word image data of 28 kinds of ancient books were sorted out, of which 1 type was used as the content template image 1. src (This font is a standard template font), and the other 27 categories are used as style reference pictures I ref Input the model for training. The training process is a two-stage optimization, first training the discriminator and then training the generator:

[0111] 1) Training of the discriminator module: Use the style extraction module to extract I ref The style feature vector Encode I using the content encoder in the generator src The content features get the feature vector Will and Input the decoder in the generator for fusion and output the generated image I fake . Will I fake and I ref Input the discriminator and output the probability P that each picture may be a real picture fake and P ref . Use P fake and P ref Calculate the adversarial loss and gradient accumulation loss. Finally, add all losses together and use back gradient propagation to supervise and optimize the parameters of the discriminator module and style extraction module.

[0112] 2) Training the generator module: Use the style extraction module to extract I ref and I src The style feature vector and Encode I using the content encoder in the generator src The content features get the feature vector Will and Input the decoder in the generator for fusion and output the generated image I fake ;Will and Input the decoder in the generator for fusion and output the generated image I rec ; will I fake The content encoder of the input generator outputs the content feature vector of the forged image Will I fake and I rec Input the discriminator and output the probability P that each picture may be a real picture fake and P rec . Use P fake and P rec Calculate the adversarial loss function and the gradient accumulation loss function; use I rec and I src Calculate the reconstruction loss; use and Calculate the reconstruction loss. Finally, add up all the losses and use the back gradient propagation to supervise and optimize the parameters of the generator module and style extraction module.

[0113] 3) Use the moving average strategy to migrate the parameters of the generator module and style extraction module to two other generator modules and style extraction modules with the same structure to achieve momentum update of the parameters.

[0114] As an optional implementation, the training parameters of the network model are set as follows:

[0115] 1) Batch size: 16

[0116] 2) Iterations: 10,000

[0117] 3) Learning rate: 0.0001 (learning rate update strategy: cosine annealing strategy)

[0118] 4) Optimizer: RMSProp (generator module, discriminator module), Adam (style extraction module)

[0119] 5) Weight Decay: 0.0001(RMSProp), 0.001(Adam).

[0120] Step 5: Fill in the text.

[0121] In some embodiments, a mask cutting algorithm is used to perform pixel-level segmentation on the generated text, and the text is placed at a corresponding position according to the positioning of the patched text calculated in step 3.

[0122] As an optional implementation, step 5 specifically includes the following steps:

[0123] Step 5.1: Traverse the coordinates of the incomplete position recorded in step 3, obtain the original text content at that position and convert it into a standard font image (i.e. I src ). Select a single word image from the current incomplete ancient book image as the style reference image I ref , will I src and I ref Input deep adversarial generative network model to generate ref Style and I src Content same font image I gen .

[0124] Step 5.2: For each I gen Binarization is performed to obtain I bin , get I bin The position of the black pixel is used as the position mask of the text to separate the I gen Pixels belonging only to the text part char . According to the coordinates of the missing text positioning box calculated in step 3 (x 左上角 ,y 左上角 ,x 右下角 ,y右下角 ), calculate the width of the box as w = x 右下角 -x 左上角 , height is h=y 右下角 -y 左上角 . char Scale it according to w and h, and paste it into the picture of the original missing ancient book according to its pixel-level position.

[0125] At this point, the restoration of ancient books is completed. The results of the restoration of ancient books are as follows: Figure 8 shown.

[0126] In summary, the present invention is equivalent to the prior art and has at least the following advantages and beneficial effects:

[0127] (1) Most existing restoration solutions for ancient books focus on physical restoration. This invention innovatively proposes a restoration solution for digitized Chinese ancient books, which can be automated using computers to save a lot of manpower. At the same time, it has far-reaching social value in the restoration of digitized Chinese ancient books.

[0128] (2) The present invention proposes a missing adaptive location algorithm, which uses the K-Means clustering algorithm and the DBI (Davies-Bouldin) index optimization algorithm to calculate the number of columns of the current incomplete ancient book image, and uses the IOU matching algorithm to adaptively locate the missing text, as described in step 3. The algorithm is simple to implement, highly interpretable, and takes a short time. It is highly adaptable and can accurately find the missing location and perform relatively precise filling and positioning.

[0129] (3) The present invention uses a deep adversarial generative network to learn and generate font styles. It can adaptively learn the different writing styles of each ancient book and imitate them with high similarity, and can restore incomplete ancient book images more deeply. At the same time, the model is trained in an unsupervised manner and does not require manual data labeling, saving a lot of manpower and material resources.

[0130] (4) The digital Chinese ancient book restoration solution described in the present invention has few limitations, high flexibility, and is generally easy to implement and deploy. It can be deployed on computer systems such as Windows 10 and Ubuntu 18.04 using a variety of programming tools (such as Python and C++), and the algorithms therein are easy to improve and integrate.

[0131] Example 2

[0132] This embodiment provides a digitized Chinese ancient book image restoration device, comprising:

[0133] A data acquisition module is used to acquire images of ancient books with missing text and the original text data corresponding to the images of the ancient books;

[0134] The Chinese character detection module is used to obtain the positioning frame coordinates of all characters on the ancient book image using a preset Chinese character detection model, and to check the missing positions on the original image based on the positioning frame coordinates and the original text data to obtain the positions of the characters that need to be repaired;

[0135] The style generation module is used to obtain the writing style in the ancient book image and generate a text image of the corresponding writing style according to the text to be repaired;

[0136] The text repair module is used to perform pixel-level segmentation on the generated text and paste the text image at the corresponding position on the ancient book image according to the position of the repaired text.

[0137] Since the device is a digital Chinese ancient book image restoration device of the embodiment of the present invention, and the principle of solving the problem of the device is the same as Figure 1 The method is similar to that of , so the implementation of the device can refer to the implementation process of the above method embodiment, and the repeated parts will not be repeated.

[0138] Example 3

[0139] An embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the following Figure 1 A method for restoring digitized images of ancient Chinese books is shown.

[0140] It is understood that the memory may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the data storage area may store data created based on the use of the server, etc.

[0141] The processor may include one or more processing cores. The processor utilizes various interfaces and circuits to connect various components within the server. It executes various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, as well as accessing data stored in memory. Optionally, the processor may be implemented using at least one of the following hardware forms: digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor may integrate one or a combination of a central processing unit (CPU) and a modem. The CPU primarily processes the operating system and application programs, while the modem handles wireless communications. It is understood that the modem may not be integrated into the processor and may be implemented separately via a single chip.

[0142] Since the electronic device is an electronic device corresponding to a method for restoring digitized ancient Chinese books images in an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0143] Example 4

[0144] An embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 1 A method for restoring digitized images of ancient Chinese books is shown.

[0145] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0146] Since the storage medium is the storage medium corresponding to a method for digitalizing ancient Chinese book image restoration in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0147] Example 5

[0148] In some possible implementations, various aspects of the methods of the embodiments of the present invention may also be implemented in the form of a program product, which includes program code. When the program product is executed on a computer device, the program code is used to cause the computer device to perform the steps of a method for digitizing an image of an ancient Chinese book according to various exemplary embodiments of the present application as described above in this specification. The executable computer program code or "code" used to perform the various embodiments may be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0149] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0150] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0151] The above embodiments are intended only to illustrate the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent changes or modifications made based on the essence of the present invention are intended to be covered by the scope of protection of the present invention.

Claims

1. A method for restoring a digitized image of an ancient Chinese book, characterized in that: The following steps are involved: Obtaining images of ancient books with missing text and the original text data corresponding to the images of the ancient books; Use the preset Chinese character detection model to obtain the positioning frame coordinates of all characters on the ancient book image, check the missing positions on the original image based on the positioning frame coordinates and the original text data, and obtain the positions of the characters that need to be repaired; Obtain the writing style in the ancient book image, and generate a text image of the corresponding writing style according to the text that needs to be repaired; Perform pixel-level segmentation on the generated text, and paste the text image on the corresponding position on the ancient book image according to the position of the obtained patched text; The step of obtaining the writing style in the ancient book image and generating a text image of the corresponding writing style according to the text to be repaired is achieved by a deep adversarial generative network; The deep adversarial generative network includes a generator module, a discriminator module and a style extraction module; The deep adversarial generative network is trained in the following way: Get single-word image data of multiple categories of ancient books, and use one of them as the content template image I src , the remaining other categories are used as style reference pictures I ref Input the network for training; first train the discriminator module and then train the generator module: Training of the discriminator module: Use the style extraction module to extract I ref The style feature vector Encode I using the content encoder in the generator module src The content features get the feature vector Will and Input the decoder in the generator module for fusion and output the generated image I fake ; will I fake and I ref Input the discriminator and output the probability P that each picture may be a real picture fake and P ref ; Use P fake and P ref Calculate the adversarial loss and gradient accumulation loss; finally, add all losses together and use back gradient propagation to supervise and optimize the parameters of the discriminator module and style extraction module; Training the generator module: Use the style extraction module to extract I ref and I src The style feature vector and Encode I using the content encoder in the generator module src The content features of Will and Input the decoder in the generator module for fusion and output the generated image I fake ;Will and Input the decoder in the generator module for fusion and output the generated image I rec ; will I fake The content encoder of the input generator module outputs the content feature vector of the forged image Will I fake and I rec Input the discriminator and output the probability P that each picture may be a real picture fake and P rec ; Use P fake and P rec Calculate the adversarial loss function and the gradient accumulation loss function; use I rec and I src Calculate the reconstruction loss; use and Calculate the reconstruction loss; finally, add up all the losses and use back gradient propagation to supervise and optimize the parameters of the generator module and style extraction module; The moving average strategy is used to migrate the parameters of the generator module and the style extraction module to two other generator modules and style extraction modules with the same structure to achieve momentum update of the parameters.

2. A digital Chinese ancient book image restoration method according to claim 1, characterized in that: The method uses a preset Chinese character detection model to obtain the positioning frame coordinates of all characters on the ancient book image, checks the missing positions on the original image based on the positioning frame coordinates and the original text data, and obtains the positions of the characters that need to be repaired, including: Use deep learning Chinese character detection to obtain the positioning frame coordinates of all characters on ancient book images; According to the obtained positioning frame coordinates, all positioning frames are clustered using the K-Means clustering algorithm and the DBI index optimization algorithm to obtain the positioning frames of all characters belonging to the same column, and the positioning frames are arranged in the order of the original text; the number of positioning frames in each column on the ancient book image is compared with the number of characters in the current column of the original text, and columns with unequal numbers of characters are screened out, which are considered to be missing columns; for each missing column screened out, a column with the same number of original characters is randomly selected from other complete columns as a reference column, and the IOU matching algorithm is used to calculate the missing position.

3. The method for restoring a digitized Chinese ancient book image according to claim 2, characterized in that: According to the obtained positioning frame coordinates, all positioning frames are clustered using the K-Means clustering algorithm and the DBI index optimization algorithm. include: Calculate the midpoint coordinates (x 中点 ,y 中点 ); According to the x coordinates of the points in all positioning boxes 中点 , clustering was performed using the K-Means clustering algorithm, combined with the DBI index optimization algorithm to obtain the best clustering effect; The midpoint y coordinates of the positioning boxes belonging to the same column are sorted in ascending order using the quick sort algorithm so that the boxes in each column are arranged in the order of the original text.

4. The method for restoring a digitized Chinese ancient book image according to claim 2, characterized in that: For each missing column that is filtered out, a column with the same number of original characters is randomly selected from other complete columns as a reference column. Use the IOU matching algorithm to calculate the missing positions, including: Note that the positioning box for the missing column is The reference column's positioning box is Calculate the IOU between boxes; Position the box according to each reference column Position a box with each missing column The IOU between them is used to calculate the score value, which is used to filter the specific missing position.

5. The method for restoring a digitized Chinese ancient book image according to claim 4, characterized in that: The calculation formula of the score value is: Where N is the number of positioning boxes of the missing column, and M is the number of positioning boxes of the reference column; The x-coordinate of the positioning box for the j-th potentially missing column, The x coordinate of the positioning box for the i-th reference column; x offset is the position difference between the reference column and the current missing column; The offset x coordinate of the positioning box for the i-th reference column; is the y coordinate of the i-th positioning box in the potential missing column, The y coordinate of the positioning box for the jth reference column; P i The i-th positioning box of the reference column corresponds to the missing column to determine whether there is a missing mark.

6. The method for restoring a digitized Chinese ancient book image according to claim 1, characterized in that: The pixel-level segmentation processing is performed on the generated text, and the text image is pasted at the corresponding position on the ancient book image according to the position of the obtained patched text, including: Text Picture I gen Perform binarization to obtain image I bin , get picture I bin The position of the black pixel is used as the position mask of the text to segment the text image I gen Pixels belonging only to the text part char ; Calculate the width w and height h of the pasted picture according to the position coordinates of the patched text, and set the pixel Pix char Scale it according to w and h, and paste it into the picture of the original missing ancient book according to its pixel-level position.

7. A digital Chinese ancient book image restoration device, applied to the method according to any one of claims 1 to 6, characterized in that: include: A data acquisition module is used to acquire images of ancient books with missing text and the original text data corresponding to the images of the ancient books; The Chinese character detection module is used to obtain the positioning frame coordinates of all characters on the ancient book image using a preset Chinese character detection model, and to check the missing positions on the original image based on the positioning frame coordinates and the original text data to obtain the positions of the characters that need to be repaired; The style generation module is used to obtain the writing style in the ancient book image and generate a text image of the corresponding writing style according to the text to be repaired; The text repair module is used to perform pixel-level segmentation on the generated text and paste the text image at the corresponding position on the ancient book image according to the position of the repaired text.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method according to any one of claims 1 to 6.

Images