Ancient book text image restoration method based on font texture and structure double-flow fusion restoration

By constructing a double-flow fusion repair model of glyph texture and structure, the targeted problems of data sets and algorithms in the image repair of ancient texts are solved, and high-quality repair of ancient texts is achieved, ensuring the consistency and details of the structure and texture of the repair results. accuracy.

CN119090780APending Publication Date: 2024-12-06SOUTHWEST JIAOTONG UNIV
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411236587.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-04
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The prior art is difficult to effectively repair damaged glyph content in ancient text images, especially in terms of data sets and repair algorithms, resulting in unreasonable repair results and blurred textures.

Method used

Using a method based on glyph texture and structure dual-stream fusion repair, a model including generator and discriminator is constructed. The network structure of the network through dual-stream parallel glyph repair network, bidirectional global feature fusion module and multi-scale context feature aggregation module is trained specifically for The repair model of ancient texts uses joint loss function to optimize the repair effect.

Benefits of technology

The structure and texture consistency of the image restoration results of ancient texts has been achieved, the quality of restoration has been improved, and the problems of artifacts and partial details have been solved, which is of great significance to the electronic preservation of ancient books.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090780A_ABST
    Figure CN119090780A_ABST
Patent Text Reader

Abstract

The invention relates to an ancient book text image restoration method based on font texture and structure double-flow fusion restoration, and belongs to the technical field of image restoration. The invention discloses an ancient book text image restoration method based on font texture and structure double-flow fusion restoration. The method comprises the following steps: constructing a text image restoration data set; constructing a double-flow fusion repair model based on the font texture and the structure; the double-flow fusion repair model based on the font texture and the structure is composed of a generator and a discriminator; training the character pattern texture and structure-based double-flow fusion repair model through the text image repair data set to obtain a trained character pattern texture and structure-based double-flow fusion repair model; and inputting the damaged ancient book text image into the trained character pattern texture and structure-based double-flow fusion repair model, and outputting the model to obtain a repaired ancient book text image. According to the method, the problem of artifacts and the problem of detail part difference are solved, and the method is of great significance to electronic and informatization storage of ancient book text materials.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to an ancient book text image restoration method based on dual-stream fusion restoration of glyph texture and structure, and belongs to the technical field of image restoration. Background Art

[0002] Ancient texts are the bridge between historical culture and modern life. With the changes and development of history, many ancient texts have been damaged to varying degrees during their circulation due to various unstable preservation factors, such as natural aging, mold and moisture, wind and rain erosion, and human damage. These factors will make the handwriting unclear or partially incomplete, thus affecting the integrity and readability of the books. Ancient books and documents record history and are the wealth of human civilization. People use historical documents to understand and study the historical development of society. With the development of digitalization, the digitization of ancient books, that is, converting physical paper ancient books into data, has become an emerging means of protecting ancient books. This method not only improves the efficiency of dissemination, but also protects the original and avoids unnecessary damage. However, in the long-term transmission process of ancient books, human and natural reasons have caused the text of many ancient books to be damaged and corroded. Repairing damaged text is a prerequisite for these documents to be studied, learned and disseminated. At present, the traditional text restoration work is completed by researchers through the pixels around the image and the text features, and then manually repaired with the help of image processing tools. This method is inefficient and takes a lot of time. In order to improve the efficiency of ancient text restoration, this paper explores restoration work based on the fusion of glyph texture and structure.

[0003] Image restoration technology aims to restore the pixels of damaged areas in an image and make the filled image as consistent as possible with the original image at the visual and semantic levels. The roots of image restoration technology can be traced back to the Renaissance, when restoration relied on handicrafts. The restorer repaired the damaged parts based on his deep understanding of the artwork to maintain the style and color consistency of the work. Although this method has achieved certain success in maintaining the visual consistency of artworks, it is inefficient and the quality of the results is highly dependent on the restorer's personal skills and experience. In recent years, with the breakthrough progress of artificial intelligence and deep learning technology, image restoration technology has achieved rapid development. It has been widely used in various fields of life and has attracted more and more attention. In the field of security monitoring, due to the limitations of imaging equipment and unfavorable environmental conditions, monitoring images are often of poor quality, which affects the extraction of information and the detection of criminal behavior. The application of image restoration technology can significantly improve image quality and enhance the efficiency of crime prevention and public safety maintenance. In the medical field, the application of image restoration technology also shows its inestimable value. Due to the limitations of medical imaging equipment or uncertainties in the operation process, the obtained medical images may contain noise, blur or missing key areas, which may interfere with the doctor's accurate diagnosis of the disease. Image restoration technology can effectively remove or reduce these artifacts and restore lost image information, thereby providing clearer and more accurate medical images. In addition, in daily life, image restoration technology has also shown its unique value. It can repair old photos damaged by improper preservation or remove unnecessary information from downloaded images, providing people with a clearer and more complete visual experience. With the widespread application of digital images in various fields of society, efficient and accurate image restoration technology has received more and more attention. Especially at a time when automation and intelligence are constantly improving, exploring more advanced image restoration algorithms is not only an important direction to meet the application needs of computer vision tasks, but also a key step in promoting innovation in digital image processing technology. It has important academic value and broad application prospects.

[0004] Image restoration based on deep learning has made great progress, but the restoration of ancient text images is still full of challenges. Compared with traditional image restoration methods, first of all, in terms of data sets, there is no complete and public ancient text data set and mask data set that simulates real damage. In terms of restoration effect, existing image restoration methods are mainly aimed at natural images, such as face images and scene images, without considering the particularity of ancient text images. These methods will cause unreasonable glyph content structure and blurred texture when used for ancient text image restoration. At the same time, the current image restoration network is purely repaired at the image level, and there is a problem of lack of semantic rationality of the text, which will result in seemingly reasonable but ambiguous restoration results. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention provides an ancient book text image restoration method based on the dual-stream fusion restoration of glyph texture and structure.

[0006] The present invention provides a technical solution to solve the above technical problems: an ancient book text image restoration method based on the fusion restoration of font texture and structure dual streams, comprising the following steps:

[0007] Step 1: Build a text image restoration dataset;

[0008] Step 2: Construct a dual-stream fusion restoration model based on glyph texture and structure;

[0009] The glyph texture and structure dual-stream fusion restoration model consists of a generator and a discriminator;

[0010] The generator includes a dual-stream parallel glyph repair network module, a bidirectional global feature fusion module and a multi-scale context feature aggregation module;

[0011] The dual-stream parallel glyph repair network module includes a glyph structure repair network branch and a glyph texture repair network branch, and the two branches use a codec structure to process glyph texture features and structural features respectively to generate glyph structure features and glyph texture features;

[0012] The bidirectional global feature fusion module processes the character structure feature and the character texture feature respectively through bidirectional processing, and finally fuses them to obtain a fusion feature;

[0013] The multi-scale context feature aggregation module processes the fused features as input features and further refines the fused features to obtain a more visually and semantically coherent restoration result;

[0014] Step 3: Train the dual-stream fusion restoration model based on glyph texture and structure through the text image restoration dataset to obtain the trained dual-stream fusion restoration model based on glyph texture and structure;

[0015] Step 4: Input the damaged ancient book text image into the trained glyph texture and structure dual-stream fusion repair model, and the model outputs the repaired ancient book text image.

[0016] A further technical solution is that in the step one, real ancient book text images on the public network are collected, single text images are selected, and then processed to finally obtain a single text dataset; and the NVIDlA lrregular Mask Dataset dataset is used as the mask image dataset.

[0017] A further technical solution is that the processing method in step one is data cleaning or data augmentation.

[0018] A further technical solution is that in the dual-stream parallel glyph repair network module, during the decoding stage, the texture decoder synthesizes a structure-constrained texture by borrowing the structural features of the structure encoder, while the structure decoder restores the texture-guided structure by extracting the texture features of the texture encoder.

[0019] A further technical solution is that the bidirectional global feature fusion module includes two branches, one is a structural feature branch perceived from texture, and the other is a texture feature branch perceived from structure;

[0020] In each branch, a soft gating mechanism and a multi-scale attention mechanism are used to precisely control the degree of fusion of texture and structural information to obtain two perceptual features. Finally, the two perceptual features are concatenated in the channel dimension, and then global attention is used for the final fusion to obtain the final fused features.

[0021] A further technical solution is that, in the multi-scale context feature aggregation module, an input feature map F is first given, and patches of sizes of 1×1, 3×3 and 5×5 pixels are extracted respectively;

[0022] Then calculate the cosine similarity; and apply the SoftMax function to the similarity to obtain the attention score for each patch;

[0023] Based on the attention map, reuse the extracted patches to reconstruct the feature map;

[0024] After obtaining the two reconstructed features, the two features are concatenated in the channel dimension and the channel attention mechanism is used to obtain the final reconstructed features.

[0025] A further technical solution is that in step three, training is performed through a joint loss, and the joint loss includes reconstruction loss, perception loss, style loss, adversarial loss and intermediate supervision loss.

[0026] The present invention has the following beneficial effects: The present invention mainly involves the production of a data set and the design of a model. For the current field of ancient text image restoration, firstly, there is no standard and rich text data set for training and research, and then there is no dedicated and specific restoration model for text restoration. Therefore, firstly, we established a standard and reliable traditional Chinese ancient text data set, and then designed a glyph restoration network based on the unique structure and details of the text image. Compared with the traditional image restoration model that mainly processes natural images such as faces and scene images, the model we designed is more focused on text image restoration. The structure and texture of the damaged text image after restoration are more consistent and reasonable with the original image parts, solving the problem of artifacts and differences in details, which is of great significance for the electronic and information preservation of ancient text materials. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a structural diagram of a dual-stream parallel glyph repair network module in the generator of the method of the present invention;

[0028] Figure 2 It is a structural diagram of a bidirectional global feature fusion module in the generator of the method of the present invention;

[0029] Figure 3 It is a structural schematic diagram of a multi-scale context feature aggregation module in the generator of the method of the present invention. DETAILED DESCRIPTION

[0030] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] The ancient book text image restoration method based on the dual-stream fusion restoration of glyph texture and structure of the present invention comprises the following steps:

[0032] Step 1: construct a text image restoration dataset, wherein the text image restoration dataset includes a single text dataset and a mask image dataset;

[0033] We collected real ancient book text images on the public Internet, collected traditional Chinese text images of the Tripitaka, selected single character images, and processed them through data cleaning or data augmentation, and finally obtained a single character dataset (TCABT) that meets the needs of actual applications.

[0034] The dataset consists of 12953 images with a resolution of 256*256 and 3 channels, of which 12753 are training sets and 200 are test sets.

[0035] The NVIDlA lrregular Mask Dataset is used as the mask image dataset, which contains 12,000 mask images with holes ranging from 0 to 60%, which can simulate the damage of real images.

[0036] Step 2: Construct a dual-stream fusion restoration model based on glyph texture and structure;

[0037] The glyph texture and structure dual-stream fusion restoration model consists of a generator and a discriminator;

[0038] The generator includes a dual-stream parallel glyph repair network module, a bidirectional global feature fusion module and a multi-scale context feature aggregation module;

[0039] like Figure 1 The dual-stream parallel glyph repair network module shown adopts the u-net codec architecture, and the dual-stream parallel structure includes an upper branch (glyph structure repair network) and a lower branch (glyph texture repair network), wherein the upper branch (glyph structure repair network) processes the glyph texture features to generate glyph structure features, and the lower branch (glyph texture repair network) processes the glyph structure features to generate glyph texture features;

[0040] Specifically, the edge repair network below is used to repair the damaged text structure information and generate a complete text edge map. Then the repaired edge map and the defective text image are input into the feature encoder for feature extraction, and then input into the feature fusion module for feature fusion. The obtained feature information is input into the next stage, and the defective text map and the feature information of the previous stage are input into the glyph texture repair network. The multi-scale fusion block is used to fuse the global and local features, so that the decoder obtains more effective information and generates high-quality image repair results.

[0041] In text images with rich structural information, reasonable text structure information will improve the quality of image restoration. We use text edge maps to reconstruct the structural information of text in images. The content restoration network is also an encoding-decoding structure. The edge restoration network obtains the restored edge map, which provides effective structural information for the content restoration network. The feature encoder is used to obtain the deep feature information of the damaged text image and the restored edge map respectively, and the feature fusion module is used to fuse the features and then input them into the bottleneck layer of the content restoration network. Then, the multi-scale fusion module is used to obtain global information and local information and fuse them again, so that the network can obtain rich feature information and generate restoration results with structural information. In the decoding stage, the texture decoder synthesizes the structure-constrained texture by borrowing the structural features of the structure encoder, while the structure decoder recovers the texture-guided structure by extracting the texture features of the texture encoder. With such a dual-generation architecture, the structure and texture complement each other well, thereby improving the restoration results.

[0042] In the specific network backbone, partial convolution operations are used to replace all ordinary convolution layers to better capture information from irregular boundaries, and more complex predictions are produced by combining low-level and high-level features at multiple scales using skip connections.

[0043] The bidirectional global feature fusion module processes the character structure feature and the character texture feature respectively through bidirectional processing, and finally fuses them to obtain a fusion feature;

[0044] The structure of the bidirectional global feature fusion module is as follows Figure 2As shown in the figure, it includes two branches, one is the structural feature branch from texture perception, and the other is the texture feature branch from structure perception. In each branch, the soft gating mechanism and multi-scale attention mechanism are used to accurately control the degree of fusion of texture and structural information to obtain two perceptual features; finally, the two perceptual features are spliced ​​in the channel dimension, and then global attention is used for the final fusion to obtain the final fusion feature.

[0045] Specifically, the texture feature map output by the decoder is represented as F t , the structural feature graph is represented by F s ,Rate control is performed through multi-scale attention and soft gating mechanisms. The multi-scale attention mechanism has the ability to learn across space, establish long-range dependencies, fuse contextual information of different scales, and produce better pixel-level attention to high-level feature maps. Messages are exchanged between the two types of information. Through this integrated operation, the features are refined and have both texture and structure perception capabilities. Finally, a global attention mechanism is adopted to improve the performance of deep neural networks by reducing information constraints and amplifying global interaction representations, amplifying significant cross-dimensional receptive areas, and dynamically adjusting the fusion of two semantic features.

[0046] Through the soft gating G t F t Merge into F s Finally, the channel cascade fusion F s ' and F t 'To obtain the integrated feature map F b ; The specific calculation is as follows:

[0047] G t =σ(g(Concat(F t ,F s )))

[0048] G s =σ(h(Concat(F t ,F s )))

[0049]

[0050] F b =Concat(F s ',F t ')

[0051] The multi-scale context feature aggregation module processes the fused features as input features and further refines the fused features to obtain a more visually and semantically coherent restoration result;

[0052] In a common coding and decoding structure, the encoding process will cause the input image to shrink, resulting in the loss of feature information. Although the subsequent decoding process will restore the image size, the lost features are not restored. As the receptive field of the higher-level network is relatively larger, the semantic information representation capability is stronger, but the resolution of the feature map is low, and the geometric information representation capability is weak (spatial geometric feature details are lacking); while the receptive field of the low-level network is relatively small, the geometric detail information representation capability is strong, and although the resolution is high, the semantic information representation capability is weak. Therefore, the present invention adopts a multi-scale feature fusion method to combine low-level features with deep-level features. The low-level features guide subsequent decoding and repair, and strengthen the repair of glyph detail information. This can not only solve the problem of feature information loss, but also ensure the combination of feature information of deep and shallow networks.

[0053] In the field of image restoration, the correct use of patch matching on deep feature maps is the key to improving the quality of generated images. Many studies have shown that by performing patch matching at different scales, the details and high-level semantic information of the image can be effectively captured. Specifically, larger patches help capture high-level semantic similarities between images, while smaller patches focus more on capturing detail information. Convolution kernels of different sizes are used to capture pixel patches of different scales and complexities. 1x1 convolution kernel: This smallest convolution kernel is used to perform linear combinations of channels for each pixel without considering spatial information. It is often used to reduce the number of channels, thereby reducing computational complexity and the number of parameters.

[0054] The 1x1 convolution kernel can enhance the feature expression capability by recombining feature channels and realize cross-channel information fusion while maintaining the spatial resolution of the feature map.

[0055] 3x3 convolution kernel: This small convolution kernel is suitable for capturing details and high-frequency features in the image, helping to maintain image quality.

[0056] 5x5 convolution kernel: The medium-sized convolution kernel can capture a wider range of contextual information and help understand the relationship between objects and structures in the image. This design enables the network to learn and represent features at different levels, thereby improving the quality of image restoration.

[0057] First, given the input feature map F, extract patches of pixel sizes 1×1, 3×3, and 5×5, and then calculate the cosine similarity. Since this module is a symmetrical multi-scale structure, the 3×3 patch size is used as an example. The calculation formula is as follows:

[0058]

[0059] Where: f i and f j They are the i-th and j-th patches of the feature map respectively;

[0060] Then, a SoftMax function is applied to the similarity to obtain the attention score for each patch:

[0061]

[0062] Next, based on the attention map, the extracted patches are reused to reconstruct the feature map;

[0063]

[0064] Where: To reconstruct the feature map The i-th patch of ;

[0065] After obtaining the two reconstructed features, the two features are concatenated in the channel dimension and the channel attention mechanism is used to determine which detail level on the image is more important to obtain the final reconstructed features.

[0066] Step 3: Train the dual-stream fusion restoration model based on glyph texture and structure through the text image restoration dataset, iterate until the network converges, and obtain the trained dual-stream fusion restoration model based on glyph texture and structure;

[0067] This model is trained through a joint loss function, including reconstruction loss, perceptual loss, style loss, adversarial loss, and intermediate supervision loss. Through the design of this multi-loss function, not only the pixel accuracy and global consistency of the generated image are enhanced, but also the realism of the image style and texture is ensured, thus achieving high-quality restoration results in image restoration tasks.

[0068] Specifically, G represents the generator, D represents the discriminator, and I gt Represents the original image, E gt represents the edge graph, Y gt represents a grayscale image, M represents a binary mask image, where 0 represents a damaged area and 1 represents an undamaged area. Indicates a damaged image. represents the damaged edge map, Represents a damaged grayscale image. The output of the generator is I out ,E out =G(I in ,E in ,Y in ,M in ).

[0069] 1) Reconstruction loss

[0070] Adoption I out and I gt The L1 distance between them is used as the reconstruction loss, and the formula is defined as follows:

[0071] L rec =E[‖I out -I gt ‖1]

[0072] 2) Perceptual loss

[0073] It quantifies the number of pixels in the feature space defined by the VGG-16 network pre-trained on ImageNet. out and I gt The L1 distance between the feature representations. The calculation formula is as follows:

[0074]

[0075] Where: φ i Represents the activation map of the i-th pooling layer of Vgg16. In the actual process, i∈[1,3];

[0076] 3) Style Loss

[0077] This loss calculation I out and I gt The L1 distance between feature maps is calculated as follows:

[0078]

[0079] in, Represents the activation map φ i Gram moment;

[0080] 4) Fighting Losses

[0081] Adversarial loss is the key to ensuring the visual authenticity of generated images. It uses the discriminator to evaluate the difference between generated images and real images, thereby promoting the generated images to be more consistent with real images in texture and structure. The calculation formula is as follows:

[0082]

[0083] 5) Intermediate loss

[0084] In order to ensure that the model can accurately capture structural and texture features, an intermediate loss is introduced to supervise the outputs of the two decoders, which helps to optimize the restoration quality of structure and texture respectively. The calculation formula is as follows:

[0085]

[0086] Among them, P s and P t is a mapping function composed of convolution kernel residual blocks, which transforms the structural features F s and texture feature F tMapped to corresponding edge maps and RGB images respectively;

[0087] The final joint loss is as follows:

[0088] L joint =λ rec L rec +λ perc L perc +λ style L style

[0089] +λ adv L adv +λ inter L inter

[0090] Where: rec , perc , style , adv and λ inter Respectively represent L rec , L perc , L style , L adv and L inter The corresponding weight coefficient, in the experiment, sets the parameter λ rec =10,λ perc =0.1,λ style =250,λ adv =0.1,λ inter =1;

[0091] The batch size is 6 during training, and the Adam optimizer is used for optimization with a learning rate of 2×10 -4 In the initial stage, the model is 2×10 -4 The main purpose of this stage is to allow the model to quickly approach a better solution space. Subsequently, in order to refine the model performance and improve the quality of image restoration, the learning rate is reduced to 5×10 -5 Fine-tune the model. This strategy of gradually reducing the learning rate helps to reduce the amplitude of parameter updates in the later stages of training and fine-tune model parameters to achieve higher accuracy. The training time for each data set is about 4 days, and then fine-tuning is done for about 1 day to achieve the final convergence of the model.

[0092] The reference indicators used in the present invention are PSNR, SSIM and MAE.

[0093] The calculation formula for PSNR peak signal-to-noise ratio is:

[0094]

[0095] in is the maximum possible pixel value of the image. If each pixel value is represented by B bits of binary, then If it is a color image, the PSNR of the three RGB channels can be calculated separately, and then the average value can be taken;

[0096] SSIM structural similarity

[0097] SSIM is a full-reference metric that takes into account the brightness, contrast, and structure of an image to evaluate the degree of structural distortion of an image. The closer SSIM is to 1, the better the image quality. The SSIM formula is based on three comparisons between samples x and y: brightness, contrast, and structure.

[0098] Brightness measurement formula:

[0099]

[0100] Contrast measurement formula:

[0101]

[0102] Structural measurement formula:

[0103]

[0104] Where c3=c2 / 2. x is the mean of x. y is the mean of y. is the variance of x. is the variance of y. xy is the covariance of x and y. c1=(k1L) 2 and c2=(k2L) 2 are two constants. L is the range of pixel values, i.e. L = 2 B -1. k1=0.01, k1=0.03.

[0105] The formula for SSIM is:

[0106] SSIM(x,y)=l(x,y) α c(x,y) β s(x,y) γ

[0107] Let α=1, β=1, γ=1 in the above formula.

[0108]

[0109] Step 4: Input the damaged ancient book text image into the trained glyph texture and structure dual-stream fusion repair model, and the model outputs the repaired ancient book text image.

[0110] In this implementation, under different mask area ratio conditions, the present invention is qualitatively compared with EdgeConnect, RFR-inpainting, and CTSDG algorithms. The results are shown in Table 1. Our method achieves the best results in all three indicators compared with other methods. The closer the SSIM value is to 1, the better, the larger the PSNR is, the better, and the closer the MAE is to 0, the better. (↑ indicates that the larger the value, the better, ↓ indicates that the smaller the value, the better)

[0111] Table 1 Comparison of objective evaluation indicators of TCABT dataset experimental results

[0112]

[0113]

[0114] The above description is not intended to limit the present invention in any form. Although the present invention has been disclosed through the above embodiments, it is not intended to limit the present invention. Any technician familiar with the profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are within the scope of the technical solution of the present invention.

Claims

1. Ancient book text image restoration method based on dual-stream fusion restoration of glyph texture and structure, It is characterized in that The following steps are involved: Step 1: Build a text image restoration dataset; Step 2: Construct a dual-stream fusion restoration model based on glyph texture and structure; The glyph texture and structure dual-stream fusion restoration model consists of a generator and a discriminator; The generator includes a dual-stream parallel glyph repair network module, a bidirectional global feature fusion module and a multi-scale context feature aggregation module; The dual-stream parallel glyph repair network module includes a glyph structure repair network branch and a glyph texture repair network branch, and the two branches use a codec structure to process glyph texture features and structural features respectively to generate glyph structure features and glyph texture features; The bidirectional global feature fusion module processes the character structure feature and the character texture feature respectively through bidirectional processing, and finally fuses them to obtain a fusion feature; The multi-scale context feature aggregation module processes the fused features as input features and further refines the fused features to obtain a more visually and semantically coherent restoration result; Step 3: Train the dual-stream fusion restoration model based on glyph texture and structure through the text image restoration dataset to obtain the trained dual-stream fusion restoration model based on glyph texture and structure; Step 4: Input the damaged ancient book text image into the trained glyph texture and structure dual-stream fusion repair model, and the model outputs the repaired ancient book text image.

2. According to claim 1, the ancient book text image restoration method based on the dual-stream fusion restoration of glyph texture and structure, It is characterized in that In the step 1, real ancient book text images on the public network are collected, and single-character images are selected and then processed to finally obtain a single-character dataset; and the NVIDlA lrregularMask Dataset dataset is used as the mask image dataset.

3. According to claim 2, the ancient book text image restoration method based on the dual-stream fusion restoration of glyph texture and structure, It is characterized in that The processing method in step 1 is data cleaning or data augmentation.

4. According to claim 1, the ancient book text image restoration method based on the dual-stream fusion restoration of glyph texture and structure, It is characterized in that In the dual-stream parallel glyph inpainting network module, during the decoding stage, the texture decoder synthesizes structure-constrained textures by borrowing the structural features of the structure encoder, while the structure decoder recovers the texture-guided structure by extracting the texture features of the texture encoder.

5. According to claim 1, the ancient book text image restoration method based on the dual-stream fusion restoration of glyph texture and structure, It is characterized in that The bidirectional global feature fusion module includes two branches, one is a structural feature branch perceived from texture, and the other is a texture feature branch perceived from structure; In each branch, a soft gating mechanism and a multi-scale attention mechanism are used to precisely control the degree of fusion of texture and structural information to obtain two perceptual features. Finally, the two perceptual features are concatenated in the channel dimension, and then global attention is used for the final fusion to obtain the final fused features.

6. The ancient book text image restoration method based on the dual-stream fusion restoration of glyph texture and structure according to claim 1, It is characterized in that In the multi-scale context feature aggregation module, the input feature map F is first given, and patches of pixel sizes 1×1, 3×3 and 5×5 are extracted respectively; Then calculate the cosine similarity and apply the SoftMax function to the similarity to obtain the attention score for each patch; Based on the attention map, reuse the extracted patches to reconstruct the feature map; After obtaining the two reconstructed features, the two features are concatenated in the channel dimension and the channel attention mechanism is used to obtain the final reconstructed features.

7. The ancient book text image restoration method based on the dual-stream fusion restoration of glyph texture and structure according to claim 1, It is characterized in that In the step three, training is performed through a joint loss, and the joint loss includes reconstruction loss, perception loss, style loss, adversarial loss and intermediate supervision loss.

Citation Information

Cited By

  • Data authenticity detection method, electronic equipment, storage medium and program product

    CN119648257A

  • Ancient book intelligent identification and restoration method and device based on machine learning

    CN119888234A

  • An Ancient Book Intelligent Recognition and Restoration Method and Device Based on Machine Learning

    CN119888234B

  • Space-level Chinese ancient book image restoration method and related equipment

    CN120708237A