Digital repairing method for ancient damaged silk

Abstract

The invention relates to the technical field of image processing, in particular to a digital repairing method for ancient damaged silk, which comprises the following steps of: respectively improving an encoder and a decoder of an existing U-shaped encoder-decoder architecture model by utilizing the characteristics of an ancient silk pattern in the aspects of color type and distribution, texture details, damaged areas and the like; information such as the shape of the damaged area can be utilized in a self-adaptive mode, feature space information of the non-damaged area can be fully utilized in a multi-scale mode, and therefore high-quality digital repairing of the ancient damaged silk is achieved.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a digital repair method for ancient damaged silk. Background Art

[0002] Ancient silk patterns have formed visual symbols with cultural characteristics in their long evolution process, conveying people's pursuit of beauty in different periods. They are symbols of the culture and thoughts and emotions of various ethnic groups, and embody rich and unique cultural connotations. There are many ancient silk collections, but many silks are partially or over a large area damaged due to aging, mechanical damage and other reasons. At present, ancient damaged silk is usually repaired by restorers manually, which is very dependent on the texture information of traditional silk and the experience of restorers. The repair results are often quite different from the original patterns due to unclear textures of ancient silk or restorers' misunderstanding of ancient silk. In addition, due to the unique inheritance method of silk restorers and the long time required for restorers to accumulate experience, there are only a handful of people who can participate in the repair of damaged silk patterns. In order to better inherit and carry forward the silk culture and restore its original historical appearance, the research on digital restoration technology of ancient silk defective patterns is very critical.

[0003] Classic image restoration algorithms have a certain ability to repair small areas of damage. When the damaged area is too large, it is difficult to generate semantically meaningful content due to the lack of advanced understanding of image content and structure. In recent years, with the rise of deep learning technology, image restoration algorithms based on deep learning have emerged. They restore images by introducing different constraints to optimize the model based on the deep model. Compared with classic restoration algorithms, they can learn higher-level image features, making the restored image structure and texture more consistent.

[0004] Deep learning was first introduced into the field of image restoration through Context Encoders [athak, D.; Krahenbuhl, P.; Donahue, J.; Darrell, T.; Efros, AAContext encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016.], which uses an encoder-decoder neural network structure and uses reconstruction loss and adversarial loss to complete the restoration of damaged areas of the image through feature learning of non-damaged areas. However, this method ignores prior knowledge such as self-similarity of the image, resulting in problems such as texture distortion and color distortion in the restoration results.

[0005] In addition, StructureFlow [Ren, Y.; Yu, X.; Zhang, R.; Li, TH; Liu, S.; Li, G. Structureflow: Image inpainting via structure-aware appearance flow. In Proceedings of the IEEE / CVF International Conference on Computer Vision, Seoul, Republic of Korea, 27 October–2 November 2019.] divides the repair task into two parts: structure reconstruction and texture generation, and proposes a two-stage solution. In the first stage, the structure reconstructor is trained using edge-retaining images. In the second stage, based on the reconstructed structure, the details of the image are generated through the texture generator to achieve image repair. However, if the first stage of this method fails to generate the correct structural information, it will be inherited by the second stage, so the repair results often have problems such as blurred boundaries and structural distortion. Summary of the invention

[0006] In view of the defects of the prior art, the present invention provides a method for digitally repairing ancient damaged silk. The method utilizes the characteristics of ancient silk patterns in terms of color types and distribution, texture details and damaged areas, and improves the encoder and decoder of the existing U-shaped encoder-decoder architecture model respectively. The method can not only adaptively utilize information such as the shape of the damaged area, but also fully utilize the feature space information of the non-damaged area at multiple scales, thereby achieving high-quality digital restoration of ancient damaged silk.

[0007] This scheme provides a digital restoration method for ancient damaged silk, including the following steps:

[0008] Step 1: Construct an ancient silk image dataset and a mask dataset simulating damaged areas:

[0009] Collect thousands of ancient silk images that were not damaged;

[0010] Use cropping, scaling and other operations to unify the resolution of silk images;

[0011] We use flipping, rotation, image generation and other operations to expand ancient silk images to tens of thousands, thereby constructing an ancient silk image dataset;

[0012] According to the known damaged silk characteristics, a mask dataset simulating the damaged area was constructed according to the boundary restricted and unrestricted damage and different silk damaged area ratios;

[0013] Step 2: Build and train a deep neural network framework for progressively refined repair with damaged area perception:

[0014] Use the ancient silk image dataset and mask dataset constructed in step 1 to train the deep neural network framework for gradually refined repair of damaged areas.

[0015] The framework of the progressively refined repair deep neural network with damaged region awareness is a U-shaped encoder-decoder architecture, which consists of a damaged region-aware encoder, a coarse repair decoder and three cascaded fine repair decoders.

[0016] Step 3: Sense the damaged area and learn the features of the non-damaged area:

[0017] Encode the damaged silk image with the mask image added into a high-level feature map using the damaged region-aware encoder in step 2;

[0018] The input of the damaged area-aware encoder is a mask image that simulates the damaged area and a damaged silk image with the mask image added, and the output of the damaged area-aware encoder is a high-level feature map; the damaged area-aware encoder introduces a damaged area-aware component, which can adaptively generate a set of specific convolution kernels for each convolution scanning window dynamically according to the learned damaged area features at the corresponding position to process the input mask image and damaged silk image;

[0019] Step 4: Restore the high-level feature map to the original multi-scale feature map:

[0020] The high-level feature map outputted in step 3 is restored to the original multi-scale feature map using the coarse repair decoder in step 2;

[0021] Step 5: Further refine the repair features:

[0022] The three cascaded fine repair decoders in step 2 are used to further refine the repair features of the multi-scale feature map output in step 4;

[0023] Each fine repair decoder uses dynamic normalization and ELU activation function, and is supervised by perceptual loss, reconstruction loss, style loss, total variation loss, etc.; the first fine repair decoder is supervised by L1 loss function and style loss function, with weight coefficients of 1 and 150 respectively; the second fine repair decoder is supervised by L1 loss function and perceptual loss function, with weight coefficients of 1 and 0.06 respectively; the third fine repair decoder is supervised by L1 loss function, perceptual loss function, style loss function and total variation loss function, with weight coefficients of 1, 0.06, 150 and 0.1 respectively.

[0024] Preferred technical solution 1: In step 1, the thousands of undamaged ancient silk images collected are compressed in PNG format, and after the resolution of the silk images is unified, the ancient silk image data set is expanded by vertical flipping, clockwise 90° rotation, clockwise 180° rotation, clockwise 270° rotation and other operations;

[0025] In order to simulate the damaged characteristics of ancient silk to the greatest extent, the mask dataset for simulating damaged areas is divided into two categories, namely, with boundary restrictions and without boundary restrictions. It is also divided into seven categories according to the size of the damaged area of ​​the simulated ancient silk: (0.01, 0.1], (0.1, 0.2], (0.2, 0.3], (0.3, 0.4], (0.4, 0.5], (0.5, 0.6], (0.6, 0.7], a total of seven categories. The purpose of setting seven categories is to more realistically simulate the various categories of the damaged area size of real ancient silk; some ancient silks are particularly severely damaged, and the defects may even reach 70% of the total area.

[0026] Preferred technical solution 2: In steps 2 and 3, during training, a k×k convolution with a stride of i is applied to And there are N H ×N W convolution windows;

[0027] The operation is as follows: First Perform a k×k convolution with a stride of i and apply the ELU activation function to generate

[0028] Then use 1×1 convolution to generate the convolution kernel tensor Where D is: s l-1 represents the image feature map of the l-1th layer of the encoder, s l represents the image feature map of the lth layer of the encoder; P L and P L-1 Represent the mask feature maps of the lth layer and the l-1th layer of the encoder respectively;

[0029] Finally, for s l-1 All N H ×N W windows for convolution, l The corresponding points in the convolution kernel are re-adjusted to the shape of the damaged area, and the above operation is applied to extract a relatively low-dimensional s l , mapping it to a higher dimension c l , c l Represents the final encoded features of the lth layer of the encoder.

[0030] Preferred technical solution three: In steps 2 and 4, during training, the first coarse repair decoder Cl The input is z l and c l-1 , z l represents the feature map of the lth layer of the coarse repair decoder, c l-1 represents the final encoded features of the encoder's layer l-1, and the output is z l-1 , z l-1 represents the feature map of the l-1th layer of the coarse inpainting decoder. Specifically, a transposed convolutional layer with a stride of 2 is used to transform z l Upsample and then compare it with c l-1 Cascade, and then apply convolution operation to c l-1 and z l Further mixing and using the ELU activation function.

[0031] Preferred technical solution 4: In steps 2 and 5, each fine repair decoder has L-1 decoding blocks and generates L-1 feature maps of different scales; using J l,k Indicates J k The l fine-repair decoding blocks of the fine-repair decoder use j l,k Represents the corresponding l-th feature map; Due to the use of a cascaded fine-repair framework, and the gradual use of supervisory information, such as reconstruction loss, perceptual loss, total variation loss, etc., to improve the restoration result; At the same time, in order to better normalize, the previous fine-repair decoder J k-1 Convolution out high-level feature map j l-1,k-1 , dynamically generate the current fine repair decoder J k The damaged area perception scale factor and offset term are determined, and the damaged area perception can be passed to the next fine repair level through the above-mentioned dynamic normalization process.

[0032] The above structure enables this solution to have the following beneficial effects:

[0033] Based on the characteristics of ancient silk patterns in terms of color type and distribution, texture details and damaged areas, this method can not only utilize information such as the shape of the damaged area, but also make full use of the characteristic spatial information of the non-damaged area, thereby achieving high-quality restoration of ancient damaged silk images. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0035] Figure 1 A flowchart of a digital restoration method for ancient damaged silk provided for this implementation;

[0036] Figure 2Block diagram of the damaged region-aware encoder provided for this implementation;

[0037] Figure 3 A block diagram of a coarse repair decoder provided for this implementation;

[0038] Figure 4 A structural diagram of the fine repair decoder provided for this implementation. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0040] like Figure 1 As shown, a digital restoration method for ancient damaged silk specifically includes the following steps:

[0041] Thousands of undamaged ancient silk images were collected and compressed in PNG format. After unifying the resolution of silk images, the ancient silk image dataset was expanded by vertical flipping, clockwise 90° rotation, clockwise 180° rotation, clockwise 270° rotation and other operations.

[0042] In order to simulate the damaged characteristics of ancient silk to the greatest extent, the mask dataset of simulated damaged areas is divided into 7 categories according to the size of the damaged area of ​​the simulated ancient silk, in addition to the two categories of boundary restriction and no boundary restriction: (0.01, 0.1], (0.1, 0.2], (0.2, 0.3], (0.3, 0.4], (0.4, 0.5], (0.5, 0.6], (0.6, 0.7];

[0043] Build and train a deep neural network framework for progressively refined repair with damaged area awareness:

[0044] Use the dataset constructed in step 1 to train the deep neural network framework for progressively refined repair of damaged areas;

[0045] The framework of the progressively refined repair deep neural network with damaged area awareness is a U-shaped encoder-decoder architecture, which consists of a damaged area-aware encoder that can perceive damaged areas and learn features of non-damaged areas, a coarse repair decoder that restores high-level feature maps to original multi-scale feature maps, and three cascaded fine repair decoders using dynamic normalization and ELU activation function.

[0046] like Figure 2As shown, in this embodiment, the damaged region perception encoder introduces a damaged region perception component, which can adaptively generate a set of specific convolution kernels for each convolution scanning window according to the learned damaged region features of the corresponding position to process the input mask image and damaged silk image. The specific process is as follows:

[0047] The input of the damaged area perception encoder is a mask image that simulates the damaged area and an image of damaged ancient silk. The mask image is processed by convolution (Conv) and ELU activation function, and then convolution is performed to generate a convolution kernel tensor (Θ l ) and the ancient silk image with the mask added is convolved together, and then batch normalized (BN) and activated by the ELU function;

[0048] Finally, it is mapped to a higher dimension c through convolution (Conv) and ELU function l ;

[0049] like Figure 3 As shown, in this embodiment, in order to restore the high-level feature map to the original multi-scale feature map, the coarse repair decoder first converts Figure 2 The high-level feature map output in the process is gradually transformed into convolution (DConv), batch normalization (BN), ELU function activation, and combined with c l-1 After deep concatenation, the convolution (Conv) and ELU function activation are performed to output z l-1 , and continuously iterate to roughly restore the original image;

[0050] like Figure 4 As shown, in this embodiment, in order to further refine the repair features, the lth feature map j in the three cascaded fine repair decoders l,k After the transposed convolution, in order to achieve better normalization, it is combined with the high-level feature map j after batch normalization. l-1,k-1 Convolution, ELU function activation and convolution are multiplied element by element and then combined with the high-level feature map j l-1,k-1 Convolution, ELU function activation and convolution operation result element by element addition, finally through the ELU function activation output feature map j l-1,k , and continuously iterate to accurately restore the original image.

[0051] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A digital restoration method for ancient damaged silk, characterized in that: The steps include: Step 1: Construct an ancient silk image dataset and a mask dataset simulating damaged areas: Collect thousands of ancient silk images that were not damaged; Use cropping, scaling and other operations to unify the resolution of silk images; We use flipping, rotation, and image generation operations to expand ancient silk images to tens of thousands, thereby constructing an ancient silk image dataset; According to the known damaged silk characteristics, a mask dataset simulating the damaged area was constructed according to the boundary restricted and unrestricted damage and different silk damaged area ratios; Step 2: Build and train a deep neural network framework for progressively refined repair with damaged area perception: Use the ancient silk image dataset and mask dataset constructed in step 1 to train the deep neural network framework for gradually refined repair of damaged areas. The framework of the progressively refined repair deep neural network with damaged region awareness is a U-shaped encoder-decoder architecture, which consists of a damaged region-aware encoder, a coarse repair decoder and three cascaded fine repair decoders. Step 3: Sense the damaged area and learn the features of the non-damaged area: Encode the damaged silk image with the mask image added into a high-level feature map using the damaged region-aware encoder in step 2; The input of the damaged area-aware encoder is a mask image that simulates the damaged area and a damaged silk image with the mask image added, and the output of the damaged area-aware encoder is a high-level feature map; the damaged area-aware encoder introduces a damaged area-aware component, which can adaptively generate a set of specific convolution kernels for each convolution scanning window dynamically according to the learned damaged area features at the corresponding position to process the input mask image and damaged silk image; Step 4: Restore the high-level feature map to the original multi-scale feature map: The high-level feature map outputted in step 3 is restored to the original multi-scale feature map using the coarse repair decoder in step 2; Step 5: Further refine the repair features: The three cascaded fine repair decoders in step 2 are used to further refine the repair features of the multi-scale feature map output in step 4; Each fine repair decoder uses dynamic normalization and ELU activation function, and is supervised by perceptual loss, reconstruction loss, style loss, and total variation loss; the first fine repair decoder is supervised by L1 loss function and style loss function, with weight coefficients of 1 and 150 respectively; the second fine repair decoder is supervised by L1 loss function and perceptual loss function, with weight coefficients of 1 and 0.06 respectively; the third fine repair decoder is supervised by L1 loss function, perceptual loss function, style loss function and total variation loss function, with weight coefficients of 1, 0.06, 150 and 0.1 respectively.

2. The digital restoration method for ancient damaged silk according to claim 1 is characterized in that: In the step 1, the thousands of undamaged ancient silk images collected are compressed in PNG format, and after the resolution of the silk images is unified, the ancient silk image dataset is expanded by vertical flipping, clockwise 90° rotation, clockwise 180° rotation, and clockwise 270° rotation operations; In order to simulate the damaged characteristics of ancient silk to the greatest extent, the mask dataset simulating the damaged area is divided into two categories: with boundary restrictions and without boundary restrictions. In addition, it is divided into seven categories according to the size of the damaged area of ​​the simulated ancient silk: (0.01, 0.1], (0.1, 0.2], (0.2, 0.3], (0.3, 0.4], (0.4, 0.5], (0.5, 0.6], (0.6, 0.7], a total of 7 categories.

3. The digital restoration method for ancient damaged silk according to claim 1 is characterized in that: In steps 2 and 3, during training, a k×k convolution with a stride of i is applied to And there are N H ×N W convolution windows; The operation is as follows: First Perform a k×k convolution with a stride of i and apply the ELU activation function to generate Then use 1×1 convolution to generate the convolution kernel tensor Where D is: Finally, for s l-1 Convolution is performed on all NH×NW windows in Θ l The convolution kernel of the damaged area shape is re-adjusted according to the corresponding points in the damaged area, and the above operation is applied to extract the relatively low-dimensional s l , mapping it to a higher dimension c l .

4. The digital restoration method for ancient damaged silk according to claim 3 is characterized in that: In steps 2 and 4, during training, the first coarse repair decoder C l The input is z l and c l-1 , the output is z l-1 Specifically, a transposed convolutional layer with a stride of 2 is used to transform z l Upsample and then compare it with c l-1 Cascade, and then apply convolution operation to c l-1 and z l Further mixing and using the ELU activation function.

5. The digital restoration method for ancient damaged silk according to claim 4 is characterized in that: In steps 2 and 5, each fine-repair decoder has L-1 fine-repair decoding blocks and generates L-1 feature maps of different scales; using J l,k Indicates J k The l fine-repair decoding blocks of the fine-repair decoder use j l,k Represents the corresponding l-th feature map; Due to the use of a cascaded fine-repair framework, and the gradual use of supervisory information, such as reconstruction loss, perceptual loss, total variation loss, etc., to improve the restoration result; At the same time, in order to better normalize, the previous fine-repair decoder J k-1 Convolution out high-level feature map j l -1,k-1 , dynamically generate the current fine repair decoder J k The damaged area perception scale factor and offset term are determined, and the damaged area perception can be passed to the next fine repair level through the above-mentioned dynamic normalization process.

Images