Image inpainting method, model and device based on convolution and transformer hybrid network

Abstract

The application discloses a kind of based on convolution and converter hybrid network's image restoration method, model and device, model construction method includes: step 1: collection tomb mural image data, pre-processing, artificial marking obtains tomb mural detection model dataset;Determine the training set and verification set of repair model;Step 2: the dataset marked by hand is input to detection model training;Step 3: damaged tomb mural is detected and converted into binary mask image, and the image of repair test set is obtained;Step 4: design damaged repair model;Step 5: using repair dataset training obtains tomb mural repair model;Step 6: using test set to test repair model, realize the automatic repair of damaged area.The application makes the detection effect of tomb mural detection model more accurate, in addition, the texture information of image is recovered globally by using the Transformer model of hybrid repair network, and the repair image of higher quality is recovered by combining the fast fourier transform method.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to an image inpainting method, model, and apparatus based on a hybrid network of depth convolution and visual converter. Background Technology

[0002] As a precious cultural heritage, the preservation of tomb murals has always been a major concern for archaeologists and cultural relics conservationists both domestically and internationally. Due to natural disasters, human factors, and climate, the deterioration of tomb murals, including fading, distortion, and peeling, has gradually worsened and is difficult to reverse, leading to a rapid decline in these already scarce and non-renewable cultural resources. In recent years, thanks to the joint efforts of experts and scholars, significant progress has been made in the research and preservation of murals. However, constrained by research methods and technical conditions, most research has focused on the partial restoration of murals. Furthermore, the results of restoration are mostly expressed in the form of papers, archaeological reports, restoration design drawings, and books. Exploration into the long-term preservation and architectural reconstruction of tomb murals is rare and often beyond our capabilities. The quantity and quality of the collected image information directly affect the progress of subsequent research. Undeniably, traditional methods such as evidence collection, manual copying, and on-site investigation played an irreplaceable role in early research; however, these methods also suffer from problems such as low image accuracy, low material utilization, severe data gaps, and the difficulty in reconstructing the three-dimensional space of mural sculptures. For damaged parts of murals, current cultural relic conservationists usually use manual restoration. However, manual restoration requires strict skills from the restorers. They not only need to have many years of work experience, but also need to have high professional knowledge and artistic cultivation, because if there is any oversight during the restoration process, the murals may suffer irreversible damage.

[0003] With the advancement of digital acquisition methods in modern society, a large number of paintings have been collected and displayed digitally. This provides support for digital image analysis using deep learning algorithms. This invention uses digital image analysis algorithms to extract high-dimensional features from tomb murals, then learns feature information from a large training set, uses a detection network to detect damaged areas of the tomb murals, and then digitally restores the damaged areas using a Transformer-based restoration network. Digital image processing technology can detect damaged areas in images through certain mathematical prior models or algorithms, and uses digital image restoration techniques for non-contact recognition and restoration. This not only avoids secondary damage to the image entity caused by traditional manual methods, but also ensures that the restoration results are easily preserved and undamaged. Therefore, the main work of this invention is to construct a digital dataset and use convolutional neural networks and Transformers to identify and restore damaged areas of tomb murals. Summary of the Invention

[0004] To address the shortcomings and deficiencies of existing technologies, this invention provides an image restoration method, model, and apparatus based on a hybrid network of convolutional and transformer networks. By establishing detection and restoration models, high-quality automatic restoration of damaged areas in tomb murals is achieved. This provides technical support for the future application and promotion of deep learning in the digital preservation of cultural heritage, overcoming the problem that existing technologies cannot automatically detect and restore damaged areas in tomb murals. This invention designs a corresponding detection network model to accurately detect damaged areas in tomb murals and automatically restores textured, consistent, and uniform void areas using the designed restoration model.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] On the one hand, a method for constructing an image inpainting model based on a hybrid network of convolutional and transformer networks is provided, including the following steps:

[0007] Step 1: Collect tomb mural image data using digital devices; the collected tomb mural images include complete tomb murals and tomb murals with damage. The collected tomb mural image data is preprocessed, and the damaged areas of the tomb murals are manually annotated using annotation software to obtain the tomb mural detection model dataset; the complete tomb mural images are used as the training set and validation set of the tomb mural restoration model.

[0008] Step 2: Input the manually labeled tomb mural dataset from Step 1 into the tomb mural damage area detection model for training, and obtain the trained detection model;

[0009] Step 3: Use the trained detection model obtained in Step 2 to perform damage detection on the damaged tomb murals, and convert the detection results into binary mask images to obtain the tomb mural restoration test set images;

[0010] Step 4: Design a damage repair model based on a visual feature converter. The damage repair model based on a visual feature converter consists of a Transformer-based structural texture repair model and a content filling model. The structural texture repair model uses fast Fourier convolutional layers for frequency domain learning, combined with a structural feature encoder composed of a fully convolutional network.

[0011] Step 5: Use the tomb mural restoration dataset obtained in Step 1 to train the damage restoration model based on visual feature converter to obtain the tomb mural restoration model.

[0012] Step 6: Use the test set of tomb murals obtained in Step 3 to test the trained hybrid network detection and repair model based on convolution and transformer, so as to realize the automatic repair of the damaged areas of the tomb murals.

[0013] Furthermore, in step 2, the tomb mural damage detection model is used to implement the following process:

[0014] Step 21: The input image first passes through a multi-scale feature extraction network to obtain five feature vectors of different scales. The three feature vectors with the smallest scales are then fed into the feature pyramid for sampling and stitching. After sampling and stitching, five feature maps of different sizes are obtained. This step connects the multi-scale feature extraction network and the feature pyramid laterally. By using the bottom-up connection method of the feature pyramid, low-level features and deep features are fused to obtain features of different sizes.

[0015] Step 22: In the prediction phase, the prediction branch employs an improved anchor-based target detector, including a classification layer, a regression layer, and a mask coefficient layer. The classification layer outputs the confidence score of the target category, the regression layer outputs the predicted bounding box result, and the mask coefficient layer predicts the generated K mask coefficients, each coefficient corresponding to a generated prototype. The feature layer result of the feature pyramid network obtained in Step 21 is used as the input of the prediction branch, and three sets of data are obtained through shared convolutional layers: the category of the anchor point, the regression coordinates of the predicted bounding box, and the mask coefficients.

[0016] Furthermore, in step 2, the loss function consists of three parts:

[0017] Loss = Loss cls +αLoss box +βLoss mask

[0018] Among them, Loss cls Loss box Loss mask These represent the classification loss function, the predicted bounding box loss function, and the mask loss function, respectively, with α and β being weight coefficients. The classification loss function is the cross-entropy loss, the predicted bounding box loss function uses the smoothL1 loss function, and the mask loss function is the pixel-wise binary cross-entropy between the predicted mask and the ground truth mask.

[0019] Loss mask =BCE(M, M) gt )

[0020] Where BCE refers to the binary cross-entropy loss function, M is the prediction mask, and M gt It is a truth mask.

[0021] Furthermore, in step 4, the loss function consists of the following four parts:

[0022] loss = αL l1 +βL adv+γL fm +μL hrf Formula III

[0023] Where α, β, γ, and μ are the weights of each loss function, and L fm For feature matching loss, L adv It consists of discriminative loss and generative loss, L hrf For visual loss; where α = 10, β = 10, γ = 100, μ = 30.

[0024] On the other hand, the present invention provides an image inpainting model construction apparatus based on a hybrid convolutional and transformer network, comprising:

[0025] The data acquisition module is used to acquire the dataset of tomb murals;

[0026] The tomb mural damage detection model module is used to train the preprocessed tomb mural dataset to obtain a tomb mural damage area detection model. This model is used to detect damage areas in the input tomb mural images and automatically outputs the detection results as binary images. The output detection results and the corresponding binary images are used for the tomb mural damage area restoration model.

[0027] The tomb mural damage repair model module is used to train a preprocessed dataset of undamaged tomb murals to obtain a tomb mural damage area repair model. This model is used to perform repair operations on the input damaged tomb mural image and its corresponding binary image to obtain a repaired tomb mural image.

[0028] Thirdly, this invention provides an image restoration method based on a hybrid network of deep convolution and visual converters. The method involves inputting the image of the tomb mural to be restored into a trained tomb mural detection model to obtain the detected damaged area image and its corresponding binary image. Then, the damaged area image and the binary image are input into a trained tomb mural restoration model to obtain the restored tomb mural image. The trained tomb mural detection model and the tomb mural restoration model are obtained through the image restoration model construction method based on the hybrid network of convolution and converters of this invention.

[0029] Fourthly, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the image inpainting model construction method based on a hybrid convolutional and transformer network of the present invention.

[0030] Fifthly, the present invention provides a computer-readable storage medium for storing program instructions that can be executed by a processor to implement the image inpainting model construction method based on a hybrid convolutional and transformer network of the present invention.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] This invention, through the design of a multi-scale feature extraction network combined with a feature pyramid, makes the tomb mural detection model more accurate than existing methods. In addition, by using a hybrid restoration network based on depth convolution and visual converter to globally restore the texture information of the image using the Transformer model, and combining it with the Fast Fourier Transform method, a higher quality restored image is obtained. Attached Figure Description

[0033] Figure 1 This is a flowchart of the method for detecting and repairing damaged areas of tomb murals according to the present invention.

[0034] Figure 2 This is a schematic diagram of the structure of the tomb mural damage detection model in this invention.

[0035] Figure 3 This is a schematic diagram of the structure of the tomb mural repair model in this invention.

[0036] Figure 4 In the image, (a) is the image to be repaired, (b) is the binary image of the detected damaged area, and (c) is the image after repair.

[0037] Figure 5 In the image, (a) is the image to be repaired, (b) is the binary image of the detected damaged area, and (c) is the image after repair. Detailed Implementation

[0038] Example 1:

[0039] like Figures 1 to 3 As shown, the image inpainting model construction method based on a hybrid convolutional and transformer network provided in this embodiment includes the following steps:

[0040] Step 1: Collect image data of tomb murals using digital devices. The collected images include complete tomb murals and images of tomb murals with damage. The collected image data is preprocessed, and the damaged areas are manually annotated using annotation software to obtain a tomb mural detection model dataset. The images of intact tomb murals are used as the training and validation sets for the tomb mural restoration model.

[0041] Step 2: Input the manually labeled tomb mural dataset from Step 1 into the tomb mural damage detection model for training, and obtain the trained detection model.

[0042] The aforementioned model for detecting damaged areas of tomb murals is used to implement the following process:

[0043] Step 21: The input image is first processed by a multi-scale feature extraction network to obtain feature vectors of five different scales. The three smallest feature vectors are then fed into a feature pyramid for sampling and concatenation. After sampling and concatenation, five feature maps of different sizes are obtained. This step horizontally connects the multi-scale feature extraction network and the feature pyramid, using the bottom-up connection method of the feature pyramid to fuse low-level features with deep features, resulting in features of different sizes.

[0044] Step 22: In the prediction phase, an anchor-based target detector (Prediction Head) typically includes a classification layer and a regression layer. The prediction branch in this invention employs an improved anchor-based target detector, which includes not only classification and regression layers but also a masking coefficient layer. The classification layer outputs the confidence score of the target category, the regression layer outputs the predicted bounding box result, and the masking coefficient layer (Protonet) predicts the generated K masking coefficients, each coefficient corresponding to a generated prototype.

[0045] The feature layer results of the feature pyramid network obtained in step 21 are used as the input of the prediction branch. Three sets of data are obtained by sharing convolutional layers: the category of the anchor point, the regression coordinates of the prediction box, and the mask coefficients.

[0046] The loss function in step 2 consists of three parts:

[0047] Loss = Loss cls +αLoss box +βLoss mask Formula I

[0048] Among them, Loss cls Loss box Loss mask Let represent the classification loss function, the predicted bounding box loss function, and the mask loss function, respectively, with α and β being weight coefficients. The classification loss function is defined as the cross-entropy loss, the predicted bounding box loss function uses the smoothL1 loss function, and the mask loss function is defined as the sum of the predicted mask and the ground truth mask (M). gt pixel-wise binary cross-entropy between:

[0049] Loss mask =BCE(M, M) gt Formula II

[0050] Where BCE refers to the binary cross-entropy loss function, M is the prediction mask, and M gt It is a truth mask.

[0051] Step 3: Use the trained detection model obtained in Step 2 to perform damage detection on the damaged tomb murals, and convert the detection results into binary mask images to obtain the tomb mural restoration test set images.

[0052] Step 4: Considering the characteristics of the damaged areas in the tomb murals, a damage repair model based on a visual feature converter was designed. This model consists of a Transformer-based structural texture restoration model and a content filling model. This model leverages the global information extraction advantage of the Transformer to recover the texture structure of the damaged areas in the tomb murals. After completing the texture information, a deep convolutional neural network is used to fill in the missing content information of the tomb murals. Compared to convolutional neural networks, the Transformer can learn longer-term interactions between sequential data in the field of image restoration; therefore, we use the Transformer for overall texture structure restoration. Furthermore, to overcome the high time complexity of the standard self-attention mechanism of the Transformer, we alternately used axial attention and standard attention in the module.

[0053] The structural texture restoration model uses fast Fourier convolutional layers for frequency domain learning, combined with a structural feature encoder composed of a fully convolutional network, to improve the structural restoration effect of tomb murals.

[0054] Our loss function mainly consists of the following four parts:

[0055] loss = αL l1 +βL adv +γL fm +μL hrf Formula III

[0056] Where α, β, γ, and μ are the weights of each loss function, and L fm For feature matching loss, L adv It consists of discriminative loss and generative loss, L hrf For visual loss, where α = 10, β = 10, γ = 100, and μ = 30.

[0057] Step 5: Use the tomb mural restoration dataset obtained in Step 1 to train the damage restoration model based on visual feature converter to obtain the tomb mural restoration model.

[0058] Step 6: Use the test set of tomb murals obtained in Step 3 to test the trained hybrid network detection and repair model based on convolution and transformer, so as to realize the automatic repair of the damaged areas of the tomb murals.

[0059] Example 2:

[0060] This embodiment provides an image inpainting model construction device based on a hybrid convolutional and transformer network, including:

[0061] The data acquisition module is used to acquire the tomb mural dataset, which is used for training and testing the tomb mural damage detection model and the tomb mural damage repair model.

[0062] The tomb mural damage detection model module is used to train a preprocessed tomb mural dataset to obtain a tomb mural damage area detection model. This model is used to detect damaged areas in input tomb mural images. The detection results are automatically output as binary images, and the output detection results and corresponding binary images are used for tomb mural damage area restoration models.

[0063] The tomb mural damage restoration model module is used to train a preprocessed dataset of undamaged tomb murals to obtain a restoration model for damaged areas. This model is used to perform restoration operations on the input damaged tomb mural image and its corresponding binary image to obtain a restored tomb mural image.

[0064] Example 3:

[0065] This embodiment provides an image restoration method based on a hybrid convolutional and transformer network. The image of the tomb mural to be restored is input into a trained tomb mural detection model to obtain the detected damaged area image and its corresponding binary image. The damaged area image and the binary image are then input into a trained tomb mural restoration model to obtain the restored tomb mural image. The trained tomb mural detection model and tomb mural restoration model are obtained through the image restoration model construction method based on a hybrid convolutional and transformer network of this invention.

[0066] Example 4:

[0067] This embodiment provides a computer device, including a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement the image inpainting model construction method based on a hybrid convolutional and transformer network of the present invention.

[0068] Example 5:

[0069] This embodiment provides a computer-readable storage medium for storing program instructions that can be executed by a processor to implement the image inpainting model construction method based on a hybrid convolutional and transformer network of the present invention.

[0070] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, and so on.

[0071] Experimental verification:

[0072] In this embodiment, Figure 4 (a) and Figure 5 (a) The image to be repaired is input into the tomb mural damage detection model to obtain... Figure 4 (b) and Figure 5 (b) Binary image of the detected damaged area, where Figure 4 The test result of (a) is Figure 4 (b), Figure 5 The test result of (a) is Figure 5 (b), will Figure 4 (b) and Figure 5 (b) The binary image of the damaged area shown is input into the tomb mural damage area restoration model to obtain... Figure 4 (c) and Figure 5 (c) Images of the restored tomb murals, among which Figure 4 The repair result of (b) is Figure 4 (c), Figure 5 The repair result of (b) is Figure 5 (c)

[0073] In this experiment, a tomb mural dataset was used to compare the detection method provided by this invention with the existing two-stage segmentation network Mask R-CNN. The comparison was made on the mean accuracy (AP), AP50, AP75, Aps, APm, and APl of the bounding boxes. These six items are the main indicators for evaluating the COCO dataset. The comparison of these indicators for measuring detection performance is shown in Table 1, where a larger AP indicates a higher accuracy. The restoration method provided by this invention was compared with the LAMA method in the prior art. The comparison of the two indicators for measuring restoration performance, namely Peak Signal-to-Noise Ratio (PSNR) and the diversity and quality of generated images (FID), is shown in Table 2. A larger PSNR indicates better image quality, and a smaller FID means that the two distributions are closer, which means that the generated images are of higher quality and have better diversity.

[0074] Table 1 Comparison of the detection method in this invention with existing detection methods

[0075] method AP↑ AP50↑ AP75↑ Aps↑ APm↑ APl↑ Mask R-CNN 13.25 28.05 10.89 7.73 10.89 14.03 This method 19.63 35.58 17.58 12.29 29.10 40.20

[0076] Table 2 Comparison of the repair method in this invention with existing repair methods

[0077] method FID↓ PSNR↑ LAMA 33.11 30.34 This method 27.50 37.95

[0078] Experimental results show that the model established in this invention outperforms existing methods in terms of performance, fully verifying the effectiveness and superiority of the method proposed in this invention.

Claims

1. A method for constructing an image inpainting model based on a hybrid network of convolutional and transformer networks, characterized in that, Includes the following steps: Step 1: Collect tomb mural image data using digital devices; the collected tomb mural images include complete tomb murals and tomb murals with damage. The collected tomb mural image data is preprocessed, and the damaged areas of the tomb murals are manually annotated using annotation software to obtain the tomb mural detection model dataset; the complete tomb mural images are used as the training set and validation set of the tomb mural restoration model. Step 2: Input the manually labeled tomb mural dataset from Step 1 into the tomb mural damage area detection model for training, and obtain the trained detection model; Step 3: Use the trained detection model obtained in Step 2 to perform damage detection on the damaged tomb murals, and convert the detection results into binary mask images to obtain the tomb mural restoration test set images; Step 4: Design a damage repair model based on a visual feature converter. The damage repair model based on a visual feature converter consists of a Transformer-based structural texture repair model and a content filling model. The structural texture repair model uses fast Fourier convolutional layers for frequency domain learning, combined with a structural feature encoder composed of a fully convolutional network. Step 5: Use the tomb mural restoration dataset obtained in Step 1 to train the damage restoration model based on visual feature converter to obtain the tomb mural restoration model. Step 6: Use the test set of tomb murals obtained in Step 3 to test the trained hybrid network detection and repair model based on convolution and transformer, so as to realize the automatic repair of the damaged areas of the tomb murals.

2. The image inpainting model construction method based on a hybrid convolutional and transformer network as described in claim 1, characterized in that, In step 2, the tomb mural damage detection model is used to implement the following process: Step 21: The input image first passes through a multi-scale feature extraction network to obtain five feature vectors of different scales. The three feature vectors with the smallest scales are then fed into the feature pyramid for sampling and stitching. After sampling and stitching, five feature maps of different sizes are obtained. This step connects the multi-scale feature extraction network and the feature pyramid laterally. By using the bottom-up connection method of the feature pyramid, low-level features and deep features are fused to obtain features of different sizes. Step 22: In the prediction phase, the prediction branch employs an improved anchor-based target detector, including a classification layer, a regression layer, and a mask coefficient layer. The classification layer outputs the confidence score of the target category, the regression layer outputs the predicted bounding box result, and the mask coefficient layer predicts the generated K mask coefficients, each coefficient corresponding to a generated prototype. The feature layer result of the feature pyramid network obtained in Step 21 is used as the input of the prediction branch, and three sets of data are obtained through shared convolutional layers: the category of the anchor point, the regression coordinates of the predicted bounding box, and the mask coefficients.

3. The image inpainting model construction method based on a hybrid convolutional and transformer network as described in claim 1, characterized in that, In step 2, the loss function consists of 3 parts. composition: in, , , These represent the classification loss function, the predicted bounding box loss function, and the mask loss function, respectively, with α and β being weight coefficients. The classification loss function is the cross-entropy loss, the predicted bounding box loss function uses the smoothL1 loss function, and the mask loss function is the pixel-wise binary cross-entropy between the predicted mask and the ground truth mask. Where BCE refers to the binary cross-entropy loss function, M is the prediction mask, and M gt It is a truth mask.

4. The image inpainting model construction method based on a hybrid convolutional and transformer network as described in claim 1, characterized in that, In step 4, the loss function consists of the following four parts: + Formula III Where α, β, γ, and μ are the weights of each loss function, and L l1 For image brightness loss, For feature matching loss, It consists of discriminative loss and generated loss. For visual loss; where α=10, β=10, γ=100, μ=30.

5. An image inpainting model construction device based on a hybrid network of convolutional and transformer networks, characterized in that, include: The data acquisition module is used to acquire the dataset of tomb murals; The tomb mural damage detection model module is used to train the preprocessed tomb mural dataset to obtain a tomb mural damage area detection model. This model is used to detect damage areas in the input tomb mural images and automatically outputs the detection results as binary images. The output detection results and the corresponding binary images are used for the tomb mural damage area restoration model. The tomb mural damage restoration model module is used to train a preprocessed dataset of undamaged tomb murals to obtain a tomb mural damage area restoration model. This model is used to perform restoration operations on the input damaged tomb mural image and its corresponding binary image to obtain a restored tomb mural image. The tomb mural damage area restoration model consists of a Transformer-based structural texture restoration model and a content filling model. The structural texture restoration model uses fast Fourier convolutional layers for frequency domain learning, combined with a structural feature encoder composed of a fully convolutional network.

6. An image inpainting method based on a hybrid network of depthwise convolutional and visual converters, characterized in that, The image of the mural to be repaired is input into a trained mural detection model to obtain the detected damaged area image and its corresponding binary image. The damaged area image and the binary image are then input into a trained mural restoration model to obtain the restored mural image. The trained mural detection model and mural restoration model are obtained by the image restoration model construction method based on a hybrid network of convolution and converter as described in any one of claims 1 to 4.

7. A computer device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the image inpainting model construction method based on a hybrid convolutional and transformer network as described in any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program instructions that can be executed by a processor to implement the image inpainting model construction method based on a hybrid convolutional and transformer network as described in any one of claims 1 to 4.

Images