Digital Chinese ancient book image restoration method, device, equipment and medium

Abstract

The invention discloses a digital Chinese ancient book image restoration method and device, equipment and a medium. The method comprises the following steps: acquiring an ancient book picture with missing characters and original text data corresponding to the ancient book picture; obtaining positioning frame coordinates of all characters on the ancient book picture by adopting a preset Chinese character detection model, checking missing positions on the original picture according to the positioning frame coordinates and the original text data, and obtaining positions of characters needing to be repaired; obtaining a writing style in the ancient book picture, and generating a character picture of the corresponding writing style according to the characters needing to be repaired; and carrying out pixel-level segmentation processing on the generated characters, and pasting the character pictures on the corresponding positions on the ancient book pictures according to the obtained positions of the repaired characters. The technical scheme is provided for digital ancient book repairing for the first time, and a large amount of manpower and time cost needed by digital ancient book repairing can be effectively saved. The method can be widely applied to the field of ancient book digital protection.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and digital protection of ancient books, and in particular to a method, device, equipment and medium for repairing digitized images of ancient Chinese books. Background Art

[0002] Ancient books have always had immeasurable historical and cultural value. Ancient books handed down from history are generally copied or printed, and most of them are preserved in the form of paper or inscriptions. In recent years, due to the in-depth development of Internet technology and document digitization technology, many ancient books have been converted into digital forms by scanning, taking photos, etc., and preserved, and disseminated and shared on the Internet. However, due to improper protection or other historical reasons, some damage will appear on the pages of ancient books, such as holes in the pages, tears, or faded content. These damages will also be retained in the scanned electronic documents. Therefore, if these damaged pictures of ancient books can be restored, it will have great historical significance and social value.

[0003] There are two problems to be solved in the restoration of ancient book images. (1) The damaged parts of each ancient book are not in the same place. If the handwritten electronic handwriting fonts are manually supplemented in the corresponding positions in the form of stickers, the labor cost is too high. (2) The writing style of each ancient book is different. If the writing style of each ancient book is manually learned and the corresponding Chinese characters are written according to the original content, the manpower and time cost is too high and it is not realistic. Therefore, the restoration of digital ancient book images requires an automated process to save manpower and time costs, but there is currently a lack of corresponding technical solutions. Summary of the invention

[0004] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide a method, device, equipment and medium for digital Chinese ancient book image restoration based on missing adaptive positioning and deep adversarial generation network.

[0005] The first technical solution adopted by the present invention is:

[0006] A method for restoring a digitized Chinese ancient book image comprises the following steps:

[0007] Obtaining an image of an ancient book with missing text and the original text data corresponding to the image of the ancient book;

[0008] The preset Chinese character detection model is used to obtain the positioning frame coordinates of all the characters on the ancient book image, and the missing positions on the original image are checked according to the positioning frame coordinates and the original text data to obtain the positions of the characters that need to be repaired;

[0009] Obtain the writing style in the ancient book image, and generate a text image of the corresponding writing style according to the text to be repaired;

[0010] The generated text is segmented at the pixel level, and the text image is pasted on the corresponding position of the ancient book image according to the position of the obtained repaired text.

[0011] Furthermore, the method of using a preset Chinese character detection model to obtain the positioning frame coordinates of all characters on the ancient book image, checking the missing position on the original image according to the positioning frame coordinates and the original text data, and obtaining the position of the characters that need to be repaired includes:

[0012] Use deep learning Chinese character detection to obtain the positioning frame coordinates of all characters on the ancient book images;

[0013] According to the obtained positioning frame coordinates, all positioning frames are clustered using the K-Means clustering algorithm and the DBI (Davies-Bouldin) index optimization algorithm to obtain the positioning frames of all the texts belonging to the same column, and the positioning frames are arranged according to the original text order;

[0014] The number of positioning boxes in each column on the ancient book image is compared with the number of words in the current column of the original text, and columns with unequal numbers of words are screened out, which are considered to be missing columns; for each missing column screened out, a column with the same number of original words is randomly selected from other complete columns as a reference column, and the IOU matching algorithm is used to calculate the missing position.

[0015] Furthermore, the method of clustering all positioning frames using the K-Means clustering algorithm and the DBI index optimization algorithm according to the obtained positioning frame coordinates includes:

[0016] Calculate the midpoint coordinates (x 中点 ,y 中点 );

[0017] According to the x coordinates x of the points in all positioning boxes 中点 , use K-Means clustering algorithm for clustering, combined with DBI index optimization algorithm to obtain the best clustering effect;

[0018] The midpoint y coordinates of the positioning boxes belonging to the same column are sorted in ascending order using the quick sort algorithm so that the boxes in each column are arranged in the order of the original text.

[0019] Furthermore, for each missing column that is screened out, a column with the same number of original characters is randomly selected from other complete columns as a reference column, and the missing position is calculated using the IOU matching algorithm, including:

[0020] Note that the positioning box for the missing column is The reference column positioning box is Calculate the IOU between boxes;

[0021] Position the box according to each reference column Position the box with each missing column The IOU between them is used to calculate the score value, which is used to filter the specific missing positions.

[0022] Furthermore, the calculation formula of the score value is:

[0023]

[0024]

[0025] Where N is the number of positioning boxes of the missing column, and M is the number of positioning boxes of the reference column; The x-coordinate of the location box for the j-th potentially missing column, The x coordinate of the positioning box for the i-th reference column; x offset is the position difference between the reference column and the current potential missing column; The offset x-coordinate of the positioning box for the i-th reference column, that is, the reference column simulation is offset to the position of the potential missing column model for calculating IOU; is the y coordinate of the ith positioning box of the potential missing column, The y coordinate of the positioning box for the jth reference column; P i The i-th positioning box of the reference column corresponds to the missing column to determine whether there is a missing mark, score i ≥0.1P i =1, indicating no missing, score i <0.1 i = 0, indicating that there is a missing column. Its specific physical meaning is: use the i-th frame of the reference column and all the frames of the potential missing column to calculate the IOU and sum them. If the frame overlaps with the frame of the potential missing column to a certain extent (i.e. score i ≥0.1), it is considered that there is no missing in the same position of the missing column. If the box does not overlap with any box of the potential missing column or the overlap is very low (i.e. score i <0.1), it means that the potential missing column is missing at this position, so the position of the missing column can be recorded (j col ,u), indicating that the jth col The i-th word is missing from the column, j col The current potential deletions are listed in the subscripts of the entire figure.

[0026] Furthermore, the step of obtaining the writing style in the ancient book image and generating a text image of the corresponding writing style according to the text to be repaired is achieved by a deep adversarial generative network;

[0027] The deep adversarial generative network includes a generator module, a discriminator module and a style extraction module;

[0028] The deep adversarial generative network is trained in the following way:

[0029] Get single-word image data of multiple categories of ancient books, and use one of them as content template image I src (i.e. the font is a standard template font), and the remaining categories are used as style reference images I ref Input the network for training; first train the discriminator module and then train the generator module:

[0030] Training of the discriminator module: Use the style extraction module to extract I ref The style feature vector Encode I using the content encoder in the generator module src The content features get the feature vector Will and Input the decoder in the generator module for fusion and output the generated image I fake ; will I fake and I fake Input the discriminator and output the probability P that each picture may be a real picture fake and P ref ; Use P fake and P ref Calculate the adversarial loss and gradient accumulation loss; finally add all losses together and use reverse gradient propagation to supervise and optimize the parameters of the discriminator module and style extraction module;

[0031] Training the generator module: Use the style extraction module to extract I ref and I src The style feature vector and Encode I using the content encoder in the generator module src The content features of Will and Input the decoder in the generator module for fusion and output the generated image I fake ;Will and Input the decoder in the generator module for fusion and output the generated image I rec ; will I fake The content encoder of the input generator module outputs the content feature vector of the forged image. Will I fake and I rec Input the discriminator and output the probability P that each picture may be a real picture fakeand P rec ; Use P fake and P rec Calculate the adversarial loss function and the gradient accumulation loss function; use I rec and I src Calculate the reconstruction loss; use and Calculate the reconstruction loss; finally add up all the losses and supervise and optimize the parameters of the generator module and style extraction module through reverse gradient propagation;

[0032] The moving average strategy is used to migrate the parameters of the generator module and the style extraction module to two other generator modules and style extraction modules with the same structure to achieve momentum update of the parameters.

[0033] Furthermore, the pixel-level segmentation processing is performed on the generated text, and the text image is pasted at a corresponding position on the ancient book image according to the position of the obtained repaired text, including:

[0034] Text Image I gen Binarization is performed to obtain image I bin , get picture I bin The position of the black pixel is used as the position mask of the text to segment the text image I gen Pixels belonging only to the text part char ;

[0035] Calculate the width w and height h of the pasted image according to the position coordinates of the patched text, and convert the pixel Pix char Scale it according to w and h, and paste it into the original picture of the missing ancient book according to its pixel-level position.

[0036] The second technical solution adopted by the present invention is:

[0037] A digital Chinese ancient book image restoration device, comprising:

[0038] A data acquisition module, used to acquire an ancient book image with missing text and original text data corresponding to the ancient book image;

[0039] The Chinese character detection module is used to obtain the positioning frame coordinates of all the characters on the ancient book image using a preset Chinese character detection model, and to check the missing positions on the original image based on the positioning frame coordinates and the original text data to obtain the positions of the characters that need to be repaired;

[0040] A style generation module is used to obtain the writing style in the ancient book image and generate a text image of the corresponding writing style according to the text to be repaired;

[0041] The text repair module is used to perform pixel-level segmentation processing on the generated text and paste the text image at the corresponding position on the ancient book image according to the position of the repaired text obtained.

[0042] The third technical solution adopted by the present invention is:

[0043] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, the code set or the instruction set is loaded and executed by the processor to implement a method for repairing a digitized Chinese ancient book image as described above.

[0044] The fourth technical solution adopted by the present invention is:

[0045] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, the code set or the instruction set is loaded and executed by a processor to implement a digital Chinese ancient book image restoration method as described above.

[0046] The fifth technical solution adopted by the present invention is:

[0047] A computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned digital Chinese ancient book image restoration method.

[0048] The beneficial effects of the present invention are as follows: the present invention proposes a technical solution for the repair of digitized ancient books for the first time, which can effectively save a lot of manpower and time costs required for the repair of digitized ancient books. In addition, the present invention can adaptively find the missing text on the ancient book picture and calculate the specific coordinates of the missing position. The positioning algorithm has strong robustness and can cope with a variety of situations. At the same time, the present invention can generate text with corresponding styles according to the writing styles of different ancient books, which has diversity and greatly increases the degree of restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 This is a flowchart of the steps of a method for restoring a digital Chinese ancient book image in an embodiment of the present invention;

[0051] Figure 2 Schematic diagram of the coordinate format of the single-word detection positioning frame in an embodiment of the present invention;

[0052] Figure 3 is a schematic diagram of an ancient book picture with partial incompleteness in an embodiment of the present invention;

[0053] Figure 4 is a schematic diagram of the result of performing text detection on an ancient book image in an embodiment of the present invention;

[0054] Figure 5 is a schematic diagram of the result of clustering the centers of the positioning boxes in an embodiment of the present invention;

[0055] Figure 6 is a schematic diagram of IOUs between positioning frames in an embodiment of the present invention;

[0056] Figure 7 Schematic diagram of IOU matching in an embodiment of the present invention;

[0057] Figure 8 It is a schematic diagram of the restoration of the ancient book picture in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0059] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0060] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0061] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0062] Terminology explanation:

[0063] DBI index: Davies-Bouldin Index is an indicator used to evaluate the quality of clustering algorithms.

[0064] Image processing is part of computer vision technology. In recent years, computer technologies related to machine learning and deep learning have flourished, among which text detection technology, text recognition technology and style font generation technology based on deep learning have achieved considerable results. These technologies can repair incomplete pictures of ancient books, such as using machine learning algorithms to locate and calibrate missing content; or using style font generation technology to generate style fonts for Chinese characters of specified content by learning the different writing styles of each ancient book, eliminating the cost of manually learning font writing styles. To this end, the present invention proposes a complete set of digital Chinese ancient book image restoration technology solutions, combining traditional image processing, machine learning and deep learning technologies to solve the above two fundamental problems of ancient book image restoration in an end-to-end scenario, so that the technical solution can be deployed on a computer platform and put into practical use. The specific contents are as follows:

[0065] Example 1

[0066] like Figure 1 As shown, this embodiment provides a method for restoring a digitized Chinese ancient book image, comprising the following steps:

[0067] S1. Obtaining an image of an ancient book with missing text and original text data corresponding to the image of the ancient book;

[0068] S2, using a preset Chinese character detection model to obtain the positioning frame coordinates of all the characters on the ancient book image, and checking the missing positions on the original image according to the positioning frame coordinates and the original text data to obtain the positions of the characters that need to be repaired;

[0069] S3, obtaining the writing style in the ancient book image, and generating a text image of the corresponding writing style according to the text to be repaired;

[0070] S4. Perform pixel-level segmentation processing on the generated text, and paste the text image at the corresponding position on the ancient book image according to the position of the obtained repaired text.

[0071] The method of this embodiment uses deep learning-based Chinese character detection and traditional image processing technology to adaptively detect the missing positions of ancient book images and complete the positioning, uses a deep adversarial generative learning model to learn and generate font styles, and places the generated text with the corresponding writing style in the original ancient book image according to the positioning to complete the restoration.

[0072] The above method is explained in detail below with reference to the accompanying drawings and specific embodiments.

[0073] This embodiment provides a method for restoring a digitized Chinese ancient book image based on defect adaptive localization and a deep adversarial generative network, comprising the following steps:

[0074] Step 1: Data preparation.

[0075] As an implementation method, electronic files of ancient books, such as Dunhuang manuscripts, are searched on the Internet. Specifically, the electronic file format to be found is .pdf, which is convenient for viewing, storing and extracting pictures of ancient books therein.

[0076] Step 2: Data preprocessing.

[0077] After finding the missing ancient book images, the original text corresponding to the ancient book images is found on the Internet and recorded. In some embodiments, the missing ancient book images and the original text recording files are named the same name; in addition, the single words of the entire ancient book image are cut to form single word data sets of different ancient books.

[0078] As a specific implementation, step 2 specifically includes the following steps:

[0079] Step 2.1: The manually selected ancient book images are stored in .png format, and the original text of the ancient book is stored in .txt format. The ancient book images are named according to their own unique sutra volume numbers. The names of the ancient book images and the original text record files are the same, such as "Golden Light Sutra Volume 2.png" and "Golden Light Sutra Volume 2.txt".

[0080] Step 2.2: For each incomplete volume of ancient books, at least one incomplete ancient book image and one complete ancient book image need to be intercepted, and the corresponding original text content is obtained at the same time. In this embodiment, there are a total of 28 volumes of ancient books.

[0081] Step 2.3: Select a complete picture of an ancient book from each volume of ancient books, cut and save the text in it, and form a single-word dataset for each ancient book.

[0082] Step 3: Adaptive search and location of missing text.

[0083] Specifically, the positioning box coordinates of all the texts on the ancient books are obtained through deep learning Chinese character detection technology, and the K-Means clustering algorithm, DBI (Davies-Bouldin) index optimization algorithm and IOU matching algorithm are used to check the missing positions on the original image and locate the repaired text according to the positioning box coordinates.

[0084] As an optional implementation, step 3 specifically includes the following steps:

[0085] Step 3.1: Training a single Chinese character detection model. This embodiment uses the existing yolov5s model and the ancient book single-character dataset MTHv2 for training to obtain a single-character detector for Chinese characters on ancient book images.

[0086] Step 3.2: For a partially incomplete picture of an ancient book (the rest of the picture is intact), use the single word detection model in step 3.1 to detect all the words on the picture and obtain the positioning box of each word in the format of (x 左上角 ,y 左上角 ,x 右下角 ,y 右下角 ),like Figure 2 shown.

[0087] For ease of description and understanding, this embodiment uses a damaged ancient book picture named test.png as an aid. Figure 3 is the original image; the text detection example effect is as follows Figure 4 As shown, the positioning frame and coordinate information corresponding to each Chinese character are obtained.

[0088] As an optional implementation, the positioning boxes of all the texts are stored in a .txt file (ie, test_detect.txt) named "image name_detect" under a specified path (the path can be arbitrarily specified in the code), and the format is "x 左上角 y 左上角 x 右下角 y 右下角 Confidence".

[0089] All boxes are arranged out of order in the file, so the next step is to rearrange the positions of the boxes to obtain the positioning boxes of the text belonging to the same column.

[0090] Step 3.3: Read all the positioning box coordinates stored in the .txt file in step 3.2, cluster all the boxes using the K-Means clustering algorithm and the DBI (Davies-Bouldin) index optimization algorithm to obtain the positioning boxes of all the texts belonging to the same column, and use the quick sort algorithm to arrange the positioning boxes in the order of the original text.

[0091] In some embodiments, step 3.3 specifically includes the following:

[0092] Step 3.3.1: Calculate the midpoint coordinates of all positioning boxes and save the midpoint coordinates of each box. The midpoint coordinates are calculated as follows:

[0093]

[0094] Step 3.3.2: Take out the x coordinates x of all the points in the positioning frame separately 中点 , use K-Means clustering algorithm for clustering, combined with DBI (Davies-Bouldin) index optimization algorithm to obtain the best clustering effect. The algorithm pseudo code is as follows:

[0095]

[0096]

[0097] Among them, the K-Means clustering algorithm needs to determine the clustering subject and the number of clustering categories. In this algorithm, the clustering subject is x 中点 The number of clustering categories is the number of columns in the current incomplete ancient book image. Since it is impossible to predict how many columns there are in the missing ancient book image, it is necessary to set col best As the number of columns, and calculated through iteration. labels is the result of clustering, which is a one-dimensional array, in which each element represents which column a certain box belongs to. For example, the label of the first column is 0, the label of the second column is 1, the label of the third column is 2, and so on; at this time, if labels = [1,3,0,4,0,1…], it means that the first positioning box belongs to the second column, the second positioning box belongs to the fourth column, the third positioning box belongs to the first column, and so on. The smaller the DBI index, the better the clustering result, and the closer it is to the actual clustering distribution. Therefore, through the comparison in the algorithm, we can get the optimal number of clustering columns corresponding to the minimum DBI, which is col best , and it is also considered that this number is the number of columns of the incomplete ancient book image itself. Finally, the optimal col best Re-perform KMeans clustering to get labels fAs the final clustering result. Therefore, Algorithm 1 can adaptively calculate the number of columns of the current incomplete ancient book image and complete the clustering. The clustering result of test.png is as follows Figure 5 As shown, we can use labels f Groups together the filters for all boxes belonging to the same column.

[0098] Step 3.3.3: Use the quick sort algorithm to sort the midpoint y coordinates of the positioning boxes belonging to the same column in ascending order, so that the boxes in each column are arranged in the order of the original text.

[0099] Step 3.4: Compare the number of positioning boxes in each column on the image with the number of words in the current column of the original text, and filter out columns with unequal numbers of words, that is, consider them as missing columns; for each missing column filtered out, randomly select a column with the same number of original words from other complete columns as a reference column, and use the IOU matching algorithm to calculate the missing position.

[0100] In some embodiments, step 3.4 specifically includes the following:

[0101] Step 3.4.1: Remember the location box of the missing column is The reference column positioning box is Calculate the IOU (Intersection over Union) between boxes. IOU is as follows Figure 6 As shown, the calculation formula is:

[0102]

[0103] Step 3.4.2: Calculate each reference column positioning box Position the box with each missing column The IOU between them is calculated and the score is used to filter the specific missing position:

[0104]

[0105] See also Figure 7 Since the reference column and the missing column do not overlap, the IOU cannot be calculated directly. The reference column needs to be offset to a position close to the missing column. Therefore, x is calculated first. offset As the position difference between the reference column and the current missing column. In formula (8), N is the number of frames in the missing column, and M is the number of frames in the reference column. The offset coordinates of each positioning frame in the reference column are obtained by formula (9): Then calculate the IOU. Formula (10) calculates the score of each positioning box in the reference column i , if score i≥0.1 indicates that the i-th frame of the reference column matches the frame of the missing column in the same row, indicating that there is no missing font at the corresponding position of the missing column; otherwise, it indicates that there is a missing font at that position and the position is recorded (i row ,i col ). The recorded location (i row ,i col ) means that the i-th col Column missing i row Words.

[0106] Step 3.5: The row and column positions of the missing characters have been calculated in step 3.4 (i row ,i col ), this step uses (i row ,i col ) Calculate the specific coordinates of the pre-filled box of the missing text, the format is (x 左上角 ,y 左上角 ,x 右下角 ,y 右下角 ). The embodiment of the present invention calculates the i-th row The average of the y coordinates of the upper left corner and lower right corner of all the positioning boxes in the row is used as the two x coordinates of the missing box (i.e. x 左上角 and x 右下角 );Calculate the i-th col The average value of the y coordinates of the upper left corner and lower right corner of all the positioning boxes is used as the two y coordinates of the missing box (i.e. y 左上角 and 右下角 ).

[0107] Step 4: Generate style fonts.

[0108] In some embodiments, the single-word data cut out in step 2 is input into a deep adversarial generative network for training to learn the writing styles of different ancient books, so as to generate a text image with specified content and style given a style image and a text content image.

[0109] As some optional implementations, the deep adversarial generative network includes a generator module, a discriminator module and a style extraction module, which generates a picture with the style and content by inputting a picture of specified content and a picture of specified style.

[0110] Specifically, in step 2, the single-word image data of 28 kinds of ancient books were sorted out, of which 1 type was used as the content template image I src (This font is a standard template font), and the other 27 categories are used as style reference pictures I ref Input the model for training. The training process is a two-stage optimization, first training the discriminator and then training the generator:

[0111] 1) Training of the discriminator module: Use the style extraction module to extract I ref The style feature vector Encode I using the content encoder in the generator src The content features get the feature vector Will and Input the decoder in the generator for fusion and output the generated image I fake . Will I fake and I ref Input the discriminator and output the probability P that each picture may be a real picture fake and P ref . Use P fake and P ref Calculate the adversarial loss and gradient accumulation loss. Finally, add all the losses together and use reverse gradient propagation to supervise and optimize the parameters of the discriminator module and style extraction module.

[0112] 2) Training the generator module: Use the style extraction module to extract I ref and I src The style feature vector and Encode I using the content encoder in the generator src The content features get the feature vector Will and Input the decoder in the generator for fusion and output the generated image I fake ;Will and Input the decoder in the generator for fusion and output the generated image I rec ; will I fake The content encoder of the input generator outputs the content feature vector of the forged image Will I fake and I rec Input the discriminator and output the probability P that each picture may be a real picture fake and P rec . Use P fake and P rec Calculate the adversarial loss function and the gradient accumulation loss function; use I rec and I src Calculate the reconstruction loss; use and Calculate the reconstruction loss. Finally, add up all the losses and use the reverse gradient propagation to supervise and optimize the parameters of the generator module and the style extraction module.

[0113] 3) Use the moving average strategy to migrate the parameters of the generator module and the style extraction module to two other generator modules and style extraction modules with the same structure to achieve momentum update of the parameters.

[0114] As an optional implementation, the training parameters of the network model are set as follows:

[0115] 1) Batch size: 16

[0116] 2) Iterations: 10000

[0117] 3) Learning rate: 0.0001 (learning rate update strategy: cosine annealing strategy)

[0118] 4) Optimizer: RMSProp (generator module, discriminator module), Adam (style extraction module)

[0119] 5) Weight Decay: 0.0001 (RMSProp), 0.001 (Adam).

[0120] Step 5: Fill in the text.

[0121] In some embodiments, a mask cutting algorithm is used to perform pixel-level segmentation on the generated text, and the text is placed at a corresponding position according to the positioning of the patched text calculated in step 3.

[0122] As an optional implementation, step 5 specifically includes the following steps:

[0123] Step 5.1: Traverse the coordinates of the incomplete position recorded in step 3, obtain the original text content at that position and convert it into a standard font image (i.e. I src ). Select a single-word image from the current incomplete ancient book image as the style reference image I ref , will I src and I ref Input deep adversarial generative network model to generate I ref Style and I src Content same font image I gen .

[0124] Step 5.2: For each I gen Binarization is performed to obtain I bin , get I bin The position of the black pixel above is used as the position mask of the text to segment I gen Pixels belonging only to the text part char . According to the coordinates of the missing text positioning box calculated in step 3 (x 左上角 ,y 左上角 ,x 右下角 ,y右下角 ), the width of the frame is calculated as w = x 右下角 -x 左上角 , height is h=y 右下角 -y 左上角 . char Scale it according to w and h, and paste it into the original picture of the missing ancient book according to its pixel-level position.

[0125] At this point, the restoration of ancient books is completed. The results of the restoration of ancient books are as follows: Figure 8 shown.

[0126] In summary, the present invention is equivalent to the prior art and has at least the following advantages and beneficial effects:

[0127] (1) Most of the existing ancient book restoration solutions are to repair ancient books at the physical level. The present invention innovatively proposes a restoration solution for digitized Chinese ancient books, which can complete automated restoration based on computers and save a lot of manpower. At the same time, it has far-reaching social value in the restoration of digitized Chinese ancient books.

[0128] (2) The present invention proposes a missing adaptive positioning algorithm, which uses the K-Means clustering algorithm and the DBI (Davies-Bouldin) index optimization algorithm to calculate the number of columns of the current incomplete ancient book image, and uses the IOU matching algorithm to perform adaptive text missing positioning, as described in step 3. The algorithm is simple to implement, highly interpretable, and takes a short time, is highly adaptable, and can accurately find the missing position and perform relatively accurate filling positioning.

[0129] (3) The present invention uses a deep adversarial generative network to learn and generate font styles, which can adaptively learn the different writing styles of each ancient book and imitate them with high similarity, and can restore incomplete ancient book images more deeply; at the same time, the model is trained in an unsupervised manner, and does not require manual data annotation, saving a lot of manpower and material resources.

[0130] (4) The digital Chinese ancient book restoration solution described in the present invention has few limitations, high flexibility, and is generally easy to implement and deploy. It can be deployed on computer systems such as Windows 10 and Ubuntu 18.04 using a variety of programming tools (such as Python, C++), and is easy to improve and integrate the algorithms therein.

[0131] Example 2

[0132] This embodiment provides a digital Chinese ancient book image restoration device, comprising:

[0133] A data acquisition module, used to acquire an ancient book image with missing text and original text data corresponding to the ancient book image;

[0134] The Chinese character detection module is used to obtain the positioning frame coordinates of all the characters on the ancient book image using a preset Chinese character detection model, and to check the missing positions on the original image based on the positioning frame coordinates and the original text data to obtain the positions of the characters that need to be repaired;

[0135] A style generation module is used to obtain the writing style in the ancient book image and generate a text image of the corresponding writing style according to the text to be repaired;

[0136] The text repair module is used to perform pixel-level segmentation processing on the generated text and paste the text image at the corresponding position on the ancient book image according to the position of the repaired text obtained.

[0137] Since the device is a digital Chinese ancient book image restoration device of the embodiment of the present invention, and the principle of solving the problem by the device is similar to Figure 1 The method is similar to that of , so the implementation of the device can refer to the implementation process of the above method embodiment, and the repeated parts will not be repeated.

[0138] Example 3

[0139] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 1 A method for restoring a digitized image of an ancient Chinese book is shown.

[0140] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0141] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0142] Since the electronic device is an electronic device corresponding to a method for digitalizing ancient Chinese book image restoration in an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0143] Example 4

[0144] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 1 A method for restoring a digitized image of an ancient Chinese book is shown.

[0145] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0146] Since the storage medium is a storage medium corresponding to a method for digitalizing ancient Chinese book image restoration in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0147] Example 5

[0148] In some possible implementations, various aspects of the method of the embodiment of the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a digital Chinese ancient book image restoration method according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" used to execute various embodiments can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0149] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0150] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0151] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for restoring a digitized Chinese ancient book image, characterized in that: The following steps are involved: Obtaining an image of an ancient book with missing text and the original text data corresponding to the image of the ancient book; The preset Chinese character detection model is used to obtain the positioning frame coordinates of all the characters on the ancient book image, and the missing positions on the original image are checked according to the positioning frame coordinates and the original text data to obtain the positions of the characters that need to be repaired; Obtain the writing style in the ancient book image, and generate a text image of the corresponding writing style according to the text to be repaired; The generated text is segmented at the pixel level, and the text image is pasted on the corresponding position of the ancient book image according to the position of the obtained repaired text.

2. A digital Chinese ancient book image restoration method according to claim 1, characterized in that: The method adopts a preset Chinese character detection model to obtain the positioning frame coordinates of all characters on the ancient book image, checks the missing positions on the original image according to the positioning frame coordinates and the original text data, and obtains the positions of the characters that need to be repaired, including: Use deep learning Chinese character detection to obtain the positioning frame coordinates of all characters on the ancient book images; According to the obtained positioning frame coordinates, all positioning frames are clustered using the K-Means clustering algorithm and the DBI index optimization algorithm to obtain the positioning frames of all the texts belonging to the same column, and the positioning frames are arranged according to the original text order; The number of positioning boxes in each column on the ancient book image is compared with the number of words in the current column of the original text, and columns with unequal numbers of words are screened out, which are considered to be missing columns; for each missing column screened out, a column with the same number of original words is randomly selected from other complete columns as a reference column, and the IOU matching algorithm is used to calculate the missing position.

3. A digital Chinese ancient book image restoration method according to claim 2, characterized in that: According to the obtained positioning frame coordinates, all positioning frames are clustered using the K-Means clustering algorithm and the DBI index optimization algorithm. include: Calculate the midpoint coordinates (x 中点 ,y 中点 ); According to the x coordinates x of the points in all positioning boxes 中点 , use K-Means clustering algorithm for clustering, combined with DBI index optimization algorithm to obtain the best clustering effect; The midpoint y coordinates of the positioning boxes belonging to the same column are sorted in ascending order using the quick sort algorithm so that the boxes in each column are arranged in the original order.

4. The method for restoring a digitized Chinese ancient book image according to claim 2, characterized in that: For each missing column that is screened out, a column with the same number of original characters is randomly selected from other complete columns as a reference column. Use the IOU matching algorithm to calculate the missing positions, including: Note that the positioning box for the missing column is The reference column positioning box is Calculate the IOU between boxes; Position the box according to each reference column Position the box with each missing column The IOU between them is used to calculate the score value, which is used to filter the specific missing positions.

5. A digital Chinese ancient book image restoration method according to claim 4, characterized in that: The calculation formula of the score value is: Where N is the number of positioning boxes of the missing column, and M is the number of positioning boxes of the reference column; The x-coordinate of the positioning box for the j-th potentially missing column, The x coordinate of the positioning box for the i-th reference column; x offset is the position difference between the reference column and the current missing column; The offset x coordinate of the positioning box for the i-th reference column; is the y coordinate of the ith positioning box of the potential missing column, The y coordinate of the positioning box for the jth reference column; P i The i-th positioning box of the reference column corresponds to the missing column to determine whether there is a missing mark.

6. The method for restoring a digitized Chinese ancient book image according to claim 1, characterized in that: The step of obtaining the writing style in the ancient book image and generating a text image of the corresponding writing style according to the text to be repaired is achieved by a deep adversarial generative network; The deep adversarial generative network includes a generator module, a discriminator module and a style extraction module; The deep adversarial generative network is trained in the following way: Get single-word image data of multiple categories of ancient books, and use one of them as content template image I src , the remaining classes are used as style reference images I ref Input the network for training; first train the discriminator module and then train the generator module: Training of the discriminator module: Use the style extraction module to extract I ref The style feature vector Encode I using the content encoder in the generator module src The content features get the feature vector Will and Input the decoder in the generator module for fusion and output the generated image I fake ; will I fake and I ref Input the discriminator and output the probability P that each picture may be a real picture fake and P ref ; Use P fake and P ref Calculate the adversarial loss and gradient accumulation loss; finally add all losses together and use reverse gradient propagation to supervise and optimize the parameters of the discriminator module and style extraction module; Training the generator module: Use the style extraction module to extract I ref and I src The style feature vector and Encode I using the content encoder in the generator module src The content features of Will and Input the decoder in the generator module for fusion and output the generated image I fake ;Will and Input the decoder in the generator module for fusion and output the generated image I rec ; will I fake The content encoder of the input generator module outputs the content feature vector of the forged image. Will I fake and I rec Input the discriminator and output the probability P that each picture may be a real picture fake and P rec ; Use P fake and P rec Calculate the adversarial loss function and the gradient accumulation loss function; use I rec and I src Calculate the reconstruction loss; use and Calculate the reconstruction loss; finally add up all the losses and supervise and optimize the parameters of the generator module and style extraction module through reverse gradient propagation; The moving average strategy is used to migrate the parameters of the generator module and the style extraction module to two other generator modules and style extraction modules with the same structure to achieve momentum update of the parameters.

7. The method for restoring a digitized Chinese ancient book image according to claim 1, characterized in that: The pixel-level segmentation process is performed on the generated text, and the text image is pasted at the corresponding position on the ancient book image according to the position of the obtained repaired text, including: Text Image I gen Binarization is performed to obtain image I bin , get picture I bin The position of the black pixel is used as the position mask of the text to segment the text image I gen Pixels belonging only to the text part char ; Calculate the width w and height h of the pasted image according to the position coordinates of the patched text, and convert the pixel Pix char Scale it according to w and h, and paste it into the original picture of the missing ancient book according to its pixel-level position.

8. A digital Chinese ancient book image restoration device, characterized in that: include: A data acquisition module, used to acquire an ancient book image with missing text and original text data corresponding to the ancient book image; The Chinese character detection module is used to obtain the positioning frame coordinates of all the characters on the ancient book image using a preset Chinese character detection model, and to check the missing positions on the original image based on the positioning frame coordinates and the original text data to obtain the positions of the characters that need to be repaired; A style generation module is used to obtain the writing style in the ancient book image and generate a text image of the corresponding writing style according to the text to be repaired; The text repair module is used to perform pixel-level segmentation processing on the generated text and paste the text image at the corresponding position on the ancient book image according to the position of the repaired text obtained.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

Images