Mural image restoration method and system based on multi-stage zero-increment connection

Through the multi-stage zero-increment connection mural image repair method, combined with edge priors and multi-stage incremental connection feature modules, the problem of lack of global perception and structural guidance for mural image repair in the existing technology is solved by using global and local receptive fields. , achieving a more realistic and clear mural image restoration effect.

CN119090779APending Publication Date: 2024-12-06DALIAN NATIONALITIES UNIVERSITY

Patent Information

Application Number
CN202411137274.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing mural image repair method based on convolutional neural network lacks global perception ability, resulting in unreal repair results and poor repair results in weak structures, making it difficult to provide effective structural guidance.

Method used

The mural image repair method with multi-stage zero-increment connection is adopted, and the feature module is connected through the edge prior guide module and the multi-stage incremental connection. Combining the global and local receptive fields, the edge structure is obtained using Transformer, and multi-scale perception is used in the texture restoration stage. Space denoising, integrating global and local features, and adding texture features in incremental form.

Benefits of technology

It improves the authenticity and clarity of mural image restoration, and is suitable for mural images with complex textures and large damage, providing better repair performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090779A_ABST
    Figure CN119090779A_ABST
Patent Text Reader

Abstract

The invention discloses a mural image restoration method and system based on multi-stage zero-increment connection, and mainly relates to the field of deep learning, and the method comprises the steps: obtaining a mural data set, and carrying out the preprocessing of a to-be-restored mural image; constructing a wall painting image restoration model; the wall painting image restoration model comprises an edge prior guide module for processing wall painting damage edge information, a multi-stage incremental connection feature module and a wall painting image global context information concerning module. The first stage is an edge restoration stage, the features generated in the first stage are used in the second stage, namely a texture restoration stage, and the to-be-restored texture module is subjected to complex texture mural image restoration; training a mural image restoration model by using the preprocessed mural image; and utilizing the trained mural image restoration model to carry out complex texture mural image restoration. Through experimental verification, the method achieves a good repairing effect on natural scenes and mural images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and in particular to a mural image restoration method and system based on multi-stage zero-increment connection. Background Art

[0002] Cultural relics represent the precious relics of the development and evolution of human society and contain extremely precious historical value. However, due to improper storage in the course of history, many cultural relics, including murals and calligraphy and paintings, have suffered from varying degrees of mildew, cracking and fading. These cultural relics are different from general natural landscapes or urban street scenes. They have complex textures and structures, and traditional manual restoration methods are often not competent. Therefore, it is necessary to propose effective mural restoration methods to address the problem of complex textures of mural images and strengthen the protection of traditional cultural relics.

[0003] Some deep learning algorithms have effectively solved the problem of image restoration. For example, in the prior art, partial convolution is proposed to address the particularity of image restoration masks, where pixels in the missing part should not participate in the mask update calculation, thereby effectively solving the problem of invalid pixels participating in the calculation. Another prior art introduces a learnable version of gated convolution (GC) based on partial convolution. This network uses a soft gating mechanism to control the update of the mask, effectively solving the artifact problem that may appear after restoration. Some prior arts also propose a two-stage edge prior network structure, in which the first stage is used to repair the edges, and the second stage performs image restoration based on edge guidance, thereby effectively improving the structurality of the overall result.

[0004] These innovative methods have brought important progress to the field of image restoration. However, these methods still have some problems. The methods based on convolutional neural networks lack effective global perception capabilities. These networks find it difficult to learn the global semantic consistency of images, which may cause the restoration results to appear unrealistic. In addition, when the image structure is weak, the restoration results are highly dependent on the consistency of the structure. Therefore, it is necessary to maintain effective restoration results in the first stage of the structural prior process, and use effective methods to provide prior guidance for the second stage of restoration. Summary of the invention

[0005] In response to the above problems, the present invention proposes a mural image restoration method and system based on multi-stage zero-incremental connection. To address the problems of lack of structural guidance and lack of large receptive field in the restoration of cultural relics images, a two-stage network is used to utilize edge structure guidance and texture structure enhancement, and a restoration network with global and local receptive fields is provided to integrate the overall complex texture information of the mural into the restoration model. It is suitable for mural image restoration with complex textures and large damage, and at the same time has better restoration performance.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] The present invention provides a mural image restoration method based on multi-stage zero-increment connection, comprising the following steps:

[0008] S1: Obtain a mural dataset and preprocess the mural images to be restored;

[0009] S2: constructing a mural image restoration model; the mural image restoration model includes: an edge prior guidance module for processing damaged edge information of the mural, a multi-stage incremental connection feature module, and a module focusing on the global context information of the mural image; in the first stage, i.e., the edge restoration stage, the features generated in the first stage are used in the second stage, i.e., the texture restoration stage, to perform mural image restoration on the complex texture of the texture module to be restored;

[0010] S3: Use the preprocessed mural images to train the mural image restoration model;

[0011] S4: Use the trained mural image restoration model to restore mural images with complex textures.

[0012] Furthermore, the mural image to be restored is preprocessed, including:

[0013] The mural image dataset is preprocessed by cropping, rotating and picking, and the mural image is cropped into a rectangular size of 256x256;

[0014] The mural image dataset is divided into a training set and a test set, and the mural images are divided into a training set, a validation set, and a test set in an 8:1:1 ratio; and data samples of image restoration are obtained.

[0015] Furthermore, the multi-stage incremental connection feature module for processing the damaged edge information of the mural includes:

[0016] Encoder, feature diffusion part and decoder:

[0017] The encoder includes a plurality of convolution blocks of different scales which gradually decrease in size, and is used to extract deep features and reduce the size of the feature map of the mural image to obtain intermediate features;

[0018] The feature diffusion part includes at least one self-attention encoder, which is used to diffuse the intermediate features to obtain the encoded features of the global context;

[0019] The decoder includes a plurality of gradually expressed convolution blocks of different scales, which are used to deeply extract the encoded features of the global context and restore the feature scale to the encoder stage size.

[0020] Furthermore, the feature diffusion part includes layer normalization, axial attention layer, forward layer and self-attention layer:

[0021] The layer normalization solves the problem of internal covariate shift, so that the input distribution of each neural layer remains consistent during the training process; and the distribution of input data of each layer in the network is relatively stable, which accelerates the learning speed of the model; the intermediate features are input to the layer normalization for processing to obtain feature vectors of the same size;

[0022] The axial attention layer decomposes the normalized output feature vector of the layer into its axial components, and then applies attention to each axis independently; the receptive field is the pixels in the same row of the target pixel, and then the row and column attentions are added to get the global attention;

[0023] The forward network layer is used to convert the global attention into new feature representations; these features are usually more conducive to learning and processing of subsequent layers than the original input;

[0024] The self-attention layer is used to find relevant information in the entire input sequence of the new feature representation, strengthen the understanding of the context of the mural image features and capture long-distance dependencies.

[0025] Furthermore, the module focusing on the global context information of the mural image includes:

[0026] The edge feature information output by the decoder is input into a downsampling encoder with three gated convolution blocks to obtain concentrated feature information;

[0027] The concentrated feature information is input into the squeezed attention to strengthen the edge information features; through the global and local feature structure encoders, edge features of different scales are extracted and added to the texture recovery features in the next step in an incremental manner to obtain multi-level structure encoding features.

[0028] Furthermore, the downsampling encoder extracts effective feature encoding of the edge structure image through three gated convolutions, including effective feature gating and improved convolution to obtain the convolution result:

[0029] Y = σ(X*V+c)⊙(X*Y+b);

[0030] Among them, * represents the convolution operation, σ is an activation function used to calculate the gating value, ⊙ represents element-by-element multiplication, X*V+c generates the gating matrix whose value is between 0 and 1 and is used to control the flow of information, and X*W+b is the result of the standard convolution operation.

[0031] Furthermore, the global and local feature structure encoder models the dependency between channels through two operations of squeezing and excitation to improve the quality of feature expression; the convolution result of the feature gated convolution structure is input, and the input feature map is compressed into a vector through global average pooling:

[0032]

[0033] Where X represents the input feature map, H represents the height of the feature map, W represents the width of the feature map, and Z c Represents the global average pooling result of the Cth channel; z is transformed through two fully connected layers and a nonlinear activation function to generate a weight vector s:

[0034] s = σ(W2δ(W1z));

[0035] Among them, W1 and W2 are the weight matrices of the fully connected layer, δ represents the ReLU activation function, and σ represents the sigmoid activation function; by re-weighting the weight vector s to the input feature map X i,j,c On each channel:

[0036] Y i,j,c =s c ·X i,j,c ;

[0037] where Y i,j,c is the feature map X i,j,c Reweighted output feature map;

[0038] Through multi-level and multi-scale deep fusion, the low-frequency features of the generated texture are increased, and a global texture perception domain is obtained; after obtaining the fusion features of the spatial domain and frequency domain, the spatial domain denoising is used to enhance the features of the frequency domain, the denoising module is used to extract the spatial domain noise value of the image, and the original frequency domain feature map is used to eliminate the spatial domain noise value to obtain a purer frequency domain feature; the final feature map has a dual-domain enhanced structure and texture feature map.

[0039] Furthermore, the preprocessed mural image is used to train a mural image restoration model, including:

[0040] The joint loss function is used to calculate the joint loss of the restored mural image and the original mural image in the training set, and the network parameters are optimized through back propagation.

[0041] Furthermore, the joint loss function is trained by calculating multiple loss functions, and the expression is:

[0042] L final =λ L1 L1+λ adv L adv +λ fm Lfm +λ hrf L hrf ;

[0043] Where L1 is the reconstruction loss, L adv To combat the loss, L fm is the feature matching loss and L hrf is, high receptive field loss; L1 is L1, λ adv , fm and λ hrf Represent the weight values ​​of each loss function respectively.

[0044] The present invention also discloses a mural image restoration system based on multi-stage zero-increment connection, comprising:

[0045] A data preprocessing module is used to obtain a mural dataset and preprocess the mural image to be restored;

[0046] A model building module is used to construct a mural image restoration model; the mural image restoration model includes: an edge prior guidance module for processing damaged edge information of the mural, a multi-stage incremental connection feature module, and a module focusing on the global context information of the mural image; in the first stage, i.e., the edge restoration stage, the features generated in the first stage are used in the second stage, i.e., the texture restoration stage, to restore the mural image with complex textures using the texture module to be restored;

[0047] A training module uses the preprocessed mural images to train a mural image restoration model;

[0048] The restoration module uses the trained mural image restoration model to restore mural images with complex textures.

[0049] The beneficial effects of the present invention are as follows:

[0050] In order to solve the problems of lack of structural guidance and large receptive field in the restoration of cultural relics images, a restoration network which utilizes edge structure guidance and provides global and local receptive fields is used to transfer to three directions: structure guidance, global and local feature fusion, and multi-scale spatial perception. In the edge structure restoration stage, Transformer is used to obtain and restore the global edge structure; in the global and local structure feature fusion stage, the structural features of different parts are extracted and incrementally added to the texture restoration network to adaptively generate textures; in the texture restoration stage, the multi-scale perception spatial denoising method is used to restore clearer and more realistic cultural relic images. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0052] Figure 1 Flow chart of a mural image restoration method based on multi-stage zero-increment connection in an embodiment of the present invention;

[0053] Figure 2 It is a diagram of the overall framework structure in an embodiment of the present invention;

[0054] Figure 3 This is a structural diagram of an edge priori guidance module in an embodiment of the present invention;

[0055] Figure 4 This is a diagram of a multi-level encoding structure of edge prior features in an embodiment of the present invention;

[0056] Figure 5 It is a specific structural diagram of the multi-stage feature fusion module in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0058] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0059] like Figure 1 As shown in FIG. 1 , a mural image restoration method based on multi-stage zero-increment connection is shown in FIG. 1 . The specific steps are as follows:

[0060] S1: Obtain a mural dataset and preprocess the mural images to be restored;

[0061] The mural image dataset is preprocessed, including:

[0062] Step S11, performing cropping, rotation and picking preprocessing on the mural image dataset, and cropping the mural image into a rectangular size of 256x256;

[0063] Step S12, dividing the mural image data set into a training set and a test set, dividing the mural image into a training set, a verification set and a test set in a ratio of 8:1:1; and obtaining a data sample for image restoration.

[0064] S2: Build a mural image restoration model;

[0065] Among them, the mural image restoration model is a network structure with multi-level attention fusion and global and local integration, which focuses on global context information to build an effective and reliable mural image restoration model.

[0066] Specifically, the mural image restoration model mainly includes: an edge prior guidance module, a multi-stage incremental connection feature module that processes the damaged edge information of the mural, and a module that focuses on the global contextual information of the mural image.

[0067] Among them: the edge prior guidance module includes: a lightweight self-attention encoder and a U-net feature extraction module. Among them, the lightweight self-attention encoder includes row attention and column attention, which are used for multi-scale feature aggregation to obtain edge global context information; row attention and column attention include self-attention mechanism and local complementary mechanism, which are used to complement the weighting of row and column self-attention. The U-net feature extraction module includes downsampling convolution, layer normalization, self-attention encoder, forward network layer, and upsampling convolution; among them, downsampling convolution is used to reduce the spatial dimension of the feature map, thereby reducing the amount of calculation and the number of parameters of the subsequent layers, speeding up the training and inference speed of the model, and obtaining the intermediate layer features. Layer normalization is used to normalize the input features to a distribution with a mean of 0 and a variance of 1 in each training sample, which helps to reduce this internal covariate shift, making the network more stable and converging faster, and obtaining normalized intermediate layer features. The self-attention encoder is used to capture global semantic information. By self-aligning and weighted combination of different positions in the input sequence, the self-attention encoder can generate richer and more accurate semantic representations and obtain global context intermediate features. The forward network layer is used to convert global attention into new feature representations, which are usually more conducive to the learning and processing of subsequent layers than the original input. Upsampling convolution is used to improve the performance and effect of the model in various visual and image processing tasks by expanding the low-resolution feature map, restoring the size of the original input or increasing the spatial details, and obtaining the feature vector.

[0068] Among them, the multi-stage incremental connection feature module includes: an encoder, a feature diffusion part and a decoder; the encoder includes multiple convolution blocks of different scales that gradually become smaller, which are used to extract deep features and reduce the size of the mural image feature map to obtain intermediate features; the feature diffusion part includes at least one self-attention encoder, which is used to diffuse the intermediate features to obtain the encoded features of the global context; among them, the feature diffusion part includes layer normalization, axial attention layer, forward layer and self-attention layer; layer normalization solves the problem of internal covariate shift, so that the distribution of the input of each neural layer remains consistent during the training process; and makes the distribution of input data of each layer in the network relatively stable, Accelerate the learning speed of the model; input the intermediate features to the layer for normalization and processing to obtain feature vectors of the same size; axial attention layer, decompose the output feature vector of the layer normalization into its axial components, and then apply attention to each axis independently; the receptive field is the pixels in the same row of the target pixel, and then add the row and column attention to get the global attention; forward network layer, used to convert the global attention into a new feature representation; these features are usually more conducive to the learning and processing of subsequent layers than the original input; self-attention layer, used to find relevant information in the entire input sequence of the new feature representation, strengthen the understanding of the context of the mural image features and capture long-distance dependencies. The decoder includes multiple convolution blocks of different scales that are gradually expressed, which are used to deeply extract the encoded features of the global context and restore the feature scale to the encoder stage size.

[0069] The multi-stage incremental connection feature module inputs the feature vector into the multi-level feature guidance module to fuse edge and texture features; the edge and texture features are input into the multi-level gated convolution to obtain the hierarchical guidance feature vector; the hierarchical guidance feature vector is input into the module that focuses on the global context information of the mural image to fuse the global context edge and texture information. Among them, the multi-level feature guidance module includes a linear projection module and a squeeze attention module; the linear projection module changes the representation space of the feature through linear transformation, which is used for operations such as dimensionality reduction or increase of feature dimensions, helps the model learn more efficient and effective feature representation, improves the model's expression ability and performance, and obtains a linear projection vector. The squeeze attention module extracts the global information of each channel, dynamically learns the weight of each channel through the excitation stage, re-weights the feature map of each channel through the scale reweighting stage, enhances the network's ability to pay attention to different feature channels, and improves the model's expression ability and performance through the self-attention model. The multi-level gated convolution uses the gating mechanism to dynamically adjust the features, improves the model's adaptability and expression ability to the input data, captures long-term dependencies and important features in the sequence, and obtains a hierarchical guidance feature vector.

[0070] Among them, the module focusing on the global context information of the mural image includes an incremental coding downsampling structure, segmented squeezed attention, a multi-level feature cross module, a fast Fourier convolution module, and a feature aggregation module. The incremental coding downsampling structure uses a multi-level guided feature vector to introduce edge structure information by adding features with a bias of 0. It gradually builds and updates the internal state when processing the input, which helps the model better capture the long-term dependencies and context information in the sequence and obtains incremental fusion features. The segmented squeezed attention uses the incremental fusion features to re-establish the long-distance dependency of the global context and obtain the long-distance structural feature dependency. The multi-level feature cross module captures rich and complex feature interaction information by crossing features at different levels and scales. The module crosses features at different levels and introduces more nonlinear relationships to improve the performance of the model in complex tasks, enhance its rich expression ability and nonlinear modeling ability of the input data features, and obtain hierarchical features. The fast Fourier convolution module uses the properties of the Fourier transform to convert the convolution operation into the frequency domain for calculation, and then converts the result back to the time domain through the inverse Fourier transform. The fast Fourier algorithm can significantly accelerate the calculation speed of convolution, especially for long sequences or large-sized convolution kernels. The feature aggregation module effectively integrates and summarizes feature information at different levels or sources to enhance the representation and generalization capabilities of the model.

[0071] The edge feature information output by the decoder of the global context information module of the mural image is input into a downsampling encoder with three gated convolution blocks to obtain concentrated feature information; the concentrated feature information is input into the squeezed attention to strengthen the obtained edge information features; through the global and local feature structure encoder, edge features of different scales are extracted and incrementally added to the texture recovery features of the next step to obtain multi-level structural encoding features.

[0072] The downsampling encoder extracts effective feature encoding from the edge structure image through three gated convolutions, including effective feature gating and improved convolution to obtain the convolution result:

[0073] Y = σ(X*V+c)⊙(X*Y+b);

[0074] Among them, * represents the convolution operation, σ is an activation function, such as the sigmoid function, which is used to calculate the gating value, ⊙ represents element-by-element multiplication, X*V+c generates the gating matrix, the value of which is between 0 and 1 and is used to control the information flow, and X*W+b is the result of the standard convolution operation;

[0075] The global and local feature structure encoder models the dependencies between channels through the two operations of "squeezing" and "excitation" to improve the quality of feature expression; the convolution result of the feature gated convolution structure is input, and the input feature map is compressed into a vector through global average pooling:

[0076]

[0077] Where X represents the input feature map, H represents the height of the feature map, W represents the width of the feature map, and Zc represents the global average pooling result of the Cth channel; through two fully connected layers and nonlinear activation functions, z is transformed to generate a weight vector s:

[0078] s = σ(W2δ(W1z));

[0079] Among them, W1 and W2 are the weight matrices of the fully connected layer, δ represents the ReLU activation function, and σ represents the sigmoid activation function. By re-weighting the weight vector s to each channel of the input feature map X:

[0080] Y i,j,c =s c ·X i,j,c ;

[0081] Where Y is the re-weighted output feature map.

[0082] Through multi-level and multi-scale deep fusion, the low-frequency features of the generated texture are increased, and a global texture perception domain is obtained; after obtaining the spatial and frequency domain fusion features, the spatial domain denoising is used to enhance the frequency domain features, the denoising module is used to extract the spatial domain noise value of the image, and the original frequency domain feature map is used to eliminate the spatial domain noise value to obtain a purer frequency domain feature. The final feature map has a dual-domain enhanced structure and texture feature map; the joint loss function is used to calculate the joint loss of the restored mural image and the original mural image in the training set, and the network parameters are optimized through back propagation.

[0083] like Figure 2 As shown in Figure 1, TSR is an edge prior guidance module, including: CNN downsampling network, lightweight Transformer structure repair block and CNN upsampling network; the input of TSR is a binary mask M (replacing the damaged area in the image, the pixel values ​​corresponding to the damaged area in M ​​are all 1, and the undamaged pixel values ​​are all 0), including the damaged image I m , and the edge map of the damaged image (referred to as the damaged edge image) I e The image obtained after channel splicing; the output of TSR is the patched edge structure map I pe .

[0084] GLSFE is a multi-stage incremental connection feature module, including gated convolution downsampling, squeezed attention incremental module, CNN incremental module and gated convolution structure coding upsampling; the input of GLSFE is the result of TSR output after CNN upsampling; the output is the structure coding result.

[0085] MSDFTR is a module that focuses on the global context information of the mural image, including a CNN incremental structure encoding downsampling network, a TDFF block, and a CNN upsampling network. The input of MSDFTR is a binary mask M, a damaged image I m The image after channel splicing also includes the output result of GLSFE; the output of MSDFTR is the repaired image I output .

[0086] The input image is sent to the Transformer structure restoration network of TSR, and is input into the axial attention and self-attention structures to extract structural features for edge restoration. The specific formula is as follows:

[0087] I pe =TSR(I m , I e ,M) (1)

[0088] In the structure recovery reasoning stage, after fast CNN upsampling, the feature map is sent to the global and local structure feature encoding block GLSFE for structure encoding. The encoder extracts local effective structure features around the damaged area of ​​the image through gated convolution and extracts fixed-size global effective structure features through lightweight squeeze attention mechanism for incremental encoding, generating a feature encoding map of multi-scale fusion receptive field. The specific formula is expressed as follows:

[0089] [F LK , F G32 ]=GLSFE(I pe ) (2)

[0090] Where L k represents different local downsampling scales k∈{32, 64, 128, 256}, G32 represents the global feature scale

[0091] Incremental coding features are introduced into the texture restoration stage for structural guidance. The texture restoration stage uses the texture restoration MSDFTR module of multi-scale spatial denoising and fast Fourier convolution to output the final restored image. The specific formula is as follows:

[0092] I output =MSDFTR(I m , F LK , F G32 ,M) (3)

[0093] The overall structure combines edge recovery, structure encoding and texture recovery modules to effectively repair the missing area and finally generate a complete restored image I output .

[0094] like Figure 3 As shown in the figure, TSR is based on the U-Net model. The damaged image with mask is used as the input of the downsampling convolution module. The input 256×256 stacked image is downsampled three times by convolution, and its dimension is reduced to 32×32 image features. These image features are encoded into tokens and sent to the lightweight self-attention module. In order to reduce the quadratic complexity of the standard self-attention mechanism, the axial attention module is used to divide the features into row and column features for self-attention calculation. After being processed by the convolution upsampling network, the image features are upsampled to a 256×256 edge feature map.

[0095] like Figure 4 As shown in the figure, TSR feeds the edge map obtained into the downsampled autoencoder in GLSFE. Specifically, all encoding layers use gated convolution and downsample the feature map to a size of 32×32. The downsampled features are then input into the squeezed attention increment module. After linear projection, vectors q, k, and v are obtained. The three vectors are respectively input into the detail enhancement branch and the squeezed axial attention branch to obtain the global squeezed feature FG32. The specific formula is as follows:

[0096]

[0097] where F detail It is the feature of strengthening the branch through details, F attn is the attention feature obtained by squeezing the axial attention. row represents the row vector of the feature matrix, and column represents the column vector of the feature matrix.

[0098] Small-scale local edge features are extracted in the smallest-scale residual gated convolution. Finally, a three-layer gated convolution decoder is used to decode local features of different scales and add them to the incremental structural feature group. Ultimately, this module generates multi-scale structural features.

[0099] like Figure 5 As shown in the figure, the multi-stage feature fusion module uses the improved fast Fourier frequency domain convolution to remove noise in the spatial domain, and uses the squeezed attention module to enhance the global spatial features. The image with missing holes and the mask of the image are concatenated and input into the multi-stage feature fusion network. The specific expression is as follows:

[0100] I input =Cat[Im ,M] (5)

[0101] In the encoder stage, the multi-scale edge structure encoding features output by GLSFE are added to the encoder stage in a zero-increment manner to effectively guide the generation of textures. After three layers of downsampling, the feature map becomes 1 / 8 of the original, and the number of channels remains unchanged. The specific formulas of the residual local features FL256, FL128, FL64, and FResL32 encoded with edge prior structure features are as follows:

[0102]

[0103] The squeezed attention module extracts and fuses the local and global spatial features, and adds the global guidance structure feature FG32 to obtain the contextual fusion feature of structure and texture. The formula expression is as follows:

[0104] F st =SEA(F down32_2 )+FG 32 (7)

[0105] The global edge structure features encoded by GLSFE are added, and the 32×32 Fst feature map is sent to the multi-level frequency domain adaptive fusion MFDAF module for multi-level frequency domain texture feature allocation. In MFDAF, residual fast Fourier convolution with frequency domain convolution kernels of 3, 5, 7, and 9 is used for feature learning. By setting the feature map grouping of Head to 4, it is divided into C / Head groups in total, and a 1×1 convolution kernel is used for deep feature fusion convolution to obtain the feature map F fusion , the specific formula expression is as follows:

[0106]

[0107] Where j is 3, 5, 7, 9, which is the set FFC convolution kernel size. Fast Fourier convolution kernels of different sizes can capture frequency domain features of different scales. K is 1, 2, 3, 4, which indicates the group of feature maps.

[0108] Through multi-level and multi-scale deep fusion, the low-frequency features of the generated texture are effectively increased, with a global texture perception domain. After obtaining the spatial and frequency domain fusion features, the spatial denoising is used to enhance the frequency domain features. The denoising module extracts the spatial noise value of the image, and then uses the original frequency domain feature map to eliminate the spatial noise value to obtain a purer frequency domain feature. The final feature map has a dual-domain enhanced structure and texture feature map. After three layers of upsampling, the restored mural image is obtained.

[0109] The repaired mural image is processed to obtain the damaged repair result, and the reconstruction loss is calculated with the original image. At the same time, it is used as the input of the comprehensive loss unit, and the overall repair loss is output to guide the discriminator to correct the correct direction of the next network training. The discriminant unit can further distinguish the generated repaired image from the original image.

[0110] S3: Use the preprocessed mural images to train the mural image restoration model;

[0111] In the specific implementation, training the mural image restoration model includes:

[0112] The model is trained by jointly minimizing the loss function with the edge prior guidance module, the multi-stage incremental connection feature module and the module focusing on the global context information of the mural image, and the optimal model parameters are obtained and saved;

[0113] The comprehensive loss includes the fusion loss of four loss functions, namely L1 reconstruction loss, adversarial loss L adv , feature matching loss LFM and high receptive field loss L hrf Optimize the repair network.

[0114] L1 reconstruction loss is used to measure the pixel difference between the real artifact image and the restored artifact image. The formula is as follows:

[0115]

[0116] Where M is a mask containing 0 or 1 pixel values, 1 is the masked part, and 0 is the valid pixel part; is the real cultural relic image, and I is the cultural relic image reconstructed by the network in this paper;

[0117] Adversarial loss L adv , a constraint loss used in GANs training, in order to improve the authenticity of the restored artifact image and produce more natural local details. PatchGAN is used as the discriminator, and the masked area is regarded as a fake sample. The adversarial loss consists of the generator loss LG and the discriminator loss LD. The formula is expressed as follows:

[0118]

[0119] in is the real cultural relic image, and I is the restored cultural relic image; is the gradient penalty; λG p= 1e-3;

[0120] Feature matching loss LFM, the loss function used for training in the GAN network, can effectively improve the authenticity of the restoration results of cultural relics images. The formula is as follows:

[0121]

[0122] High receptive field loss L hrf , improve the large holes of the cultural relic repair model. The formula is as follows:

[0123]

[0124] Among them, φ hrf It is the process of extracting features using a pre-processed ResNet50 with dilated convolution.

[0125] The comprehensive loss, combining four loss functions, is expressed as follows:

[0126] L final =λ L1 L1+λ adv L adv +λ FM L FM +λ HRF L hrf (13)

[0127] Among them, λ L1 , adv , FM and λ HRF Represent the weight values ​​of each loss function respectively.

[0128] During the training process of the discriminator and the overall model, the loss continues to decrease, the images repaired by the restoration model become more realistic and effective, and the discriminant unit is more able to distinguish between generated images and real images. When the training model and the discriminant model are close to Nash equilibrium, it means the end of training, and the final restoration result of the damaged mural image is obtained.

[0129] S4: Use the trained mural image restoration model to restore mural images with complex textures.

[0130] In the above embodiment, in order to solve the problems of lack of structural guidance and lack of large receptive field in the restoration of cultural relic images, a restoration network that utilizes edge structure guidance and provides global and local receptive fields is used, and is transferred to three directions of structural guidance, global and local feature fusion, and multi-scale spatial perception. In the edge structure restoration stage, Transformer is used to obtain and restore the global edge structure; in the global and local structural feature fusion stage, the structural features of different parts are extracted, and the texture restoration network is incrementally added to enable it to adaptively generate textures; in the texture restoration stage, a multi-scale perceptual spatial denoising method is used to restore a clearer and more realistic cultural relic image.

[0131] Corresponding to the above-mentioned mural image restoration method, the present invention further provides a mural image restoration system based on multi-stage zero-increment connection, the system comprising:

[0132] A data preprocessing module is used to obtain a mural dataset and preprocess the mural image to be restored;

[0133] A model building module is used to construct a mural image restoration model; the mural image restoration model includes: an edge prior guidance module for processing damaged edge information of the mural, a multi-stage incremental connection feature module, and a module focusing on the global context information of the mural image; in the first stage, i.e., the edge restoration stage, the features generated in the first stage are used in the second stage, i.e., the texture restoration stage, to restore the mural image with complex textures using the texture module to be restored;

[0134] A training module uses the preprocessed mural images to train a mural image restoration model;

[0135] The restoration module uses the trained mural image restoration model to restore mural images with complex textures.

[0136] As for the repair method of the embodiment of the present invention, since it corresponds to the repair system in the above embodiment, the description is relatively simple. For the relevant similarities, please refer to the description of the repair method part in the above embodiment, which will not be described in detail here.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A mural image restoration method based on multi-stage zero-increment connection, It is characterized in that The steps include: S1: Obtain a mural dataset and preprocess the mural images to be restored; S2: constructing a mural image restoration model; the mural image restoration model includes: an edge prior guidance module for processing damaged edge information of the mural, a multi-stage incremental connection feature module, and a module focusing on the global context information of the mural image; in the first stage, i.e., the edge restoration stage, the features generated in the first stage are used in the second stage, i.e., the texture restoration stage, to perform mural image restoration on the complex texture of the texture module to be restored; S3: Use the preprocessed mural images to train the mural image restoration model; S4: Use the trained mural image restoration model to restore mural images with complex textures.

2. According to the mural image restoration method based on multi-stage zero-increment connection according to claim 1, It is characterized in that Preprocess the mural image to be restored, including: The mural image dataset is preprocessed by cropping, rotating and picking, and the mural image is cropped into a rectangular size of 256x256; The mural image dataset is divided into a training set and a test set, and the mural images are divided into a training set, a validation set, and a test set in an 8:1:1 ratio; and data samples of image restoration are obtained.

3. According to the mural image restoration method based on multi-stage zero-increment connection according to claim 1, It is characterized in that The multi-stage incremental connection feature module for processing damaged edge information of murals includes: Encoder, feature diffusion part and decoder: The encoder includes a plurality of convolution blocks of different scales which gradually decrease in size, and is used to extract deep features and reduce the size of the feature map of the mural image to obtain intermediate features; The feature diffusion part includes at least one self-attention encoder, which is used to diffuse the intermediate features to obtain the encoded features of the global context; The decoder includes a plurality of gradually expressed convolution blocks of different scales, which are used to deeply extract the encoded features of the global context and restore the feature scale to the encoder stage size.

4. According to the mural image restoration method based on multi-stage zero-increment connection according to claim 3, It is characterized in that The feature diffusion part includes layer normalization, axial attention layer, forward layer and self-attention layer: The layer normalization solves the problem of internal covariate shift, so that the input distribution of each neural layer remains consistent during the training process; and the distribution of input data of each layer in the network is relatively stable, which accelerates the learning speed of the model; the intermediate features are input to the layer normalization for processing to obtain feature vectors of the same size; The axial attention layer decomposes the normalized output feature vector of the layer into its axial components, and then applies attention to each axis independently; the receptive field is the pixels in the same row of the target pixel, and then the row and column attentions are added to get the global attention; The forward network layer is used to convert the global attention into a new feature representation; These features are usually more conducive to learning and processing by subsequent layers than the original input; The self-attention layer is used to find relevant information in the entire input sequence of the new feature representation, strengthen the understanding of the context of the mural image features and capture long-distance dependencies.

5. According to the mural image restoration method based on multi-stage zero-increment connection in claim 3, It is characterized in that The module focusing on the global context information of the mural image includes: The edge feature information output by the decoder is input into a downsampling encoder with three gated convolution blocks to obtain concentrated feature information; The concentrated feature information is input into the squeezed attention to strengthen the edge information features; through the global and local feature structure encoders, edge features of different scales are extracted and added to the texture recovery features in the next step in an incremental manner to obtain multi-level structure encoding features.

6. According to the mural image restoration method based on multi-stage zero-increment connection according to claim 5, It is characterized in that The downsampling encoder extracts effective feature encoding from the edge structure image through three gated convolutions, including effective feature gating and improved convolution to obtain the convolution result: Y = σ(X*V+c)☉(X*Y+b); Among them, * represents the convolution operation, σ is an activation function used to calculate the gating value, ⊙ represents element-by-element multiplication, X*V+c generates the gating matrix whose value is between 0 and 1, which is used to control the information flow, and X*W+b is the result of the standard convolution operation.

7. According to claim 5, a mural image restoration method based on multi-stage zero-increment connection, It is characterized in that The global and local feature structure encoder models the dependency between channels through two operations, namely, squeezing and excitation, to improve the quality of feature expression. The convolution result of the feature gated convolution structure is input, and the input feature map is compressed into a vector through global average pooling: Where X represents the input feature map, H represents the height of the feature map, W represents the width of the feature map, and Z c Represents the global average pooling result of the Cth channel; z is transformed through two fully connected layers and a nonlinear activation function to generate a weight vector s: s=σ(W 2 δ(W 1 z)); Among them, W 1 and W 2 is the weight matrix of the fully connected layer, δ represents the ReLU activation function, and σ represents the sigmoid activation function; by re-weighting the weight vector s to the input feature map X i,j,c On each channel: Y i,j,c =s c ·X i,j,c ; where Y i,j,c is the feature map X i,j,c Reweighted output feature map; Through multi-level and multi-scale deep fusion, the low-frequency features of the generated texture are increased, and a global texture perception domain is obtained; after obtaining the fusion features of the spatial domain and frequency domain, the spatial domain denoising is used to enhance the features of the frequency domain, the denoising module is used to extract the spatial domain noise value of the image, and the original frequency domain feature map is used to eliminate the spatial domain noise value to obtain a purer frequency domain feature; the final feature map has a dual-domain enhanced structure and texture feature map.

8. According to the mural image restoration method based on multi-stage zero-increment connection in claim 1, It is characterized in that The preprocessed mural images are used to train the mural image restoration model, including: The joint loss function is used to calculate the joint loss of the restored mural image and the original mural image in the training set, and the network parameters are optimized through back propagation.

9. According to the mural image restoration method based on multi-stage zero-increment connection of claim 8, It is characterized in that The joint loss function is trained by calculating multiple loss functions, and the expression is: L final =λ L1 L 1 +λ adv L adv +λ fm L fm +λ hrf L hrf ; Where L 1 is the reconstruction loss, L adv To combat the loss, L fm is the feature matching loss and L hrf is, high receptive field loss; L1 For L 1 , adv , fm and λ hrf Represent the weight values ​​of each loss function respectively.

10. A mural image restoration system based on multi-stage zero-increment connection, It is characterized in that include: A data preprocessing module is used to obtain a mural dataset and preprocess the mural image to be restored; Model building module, building a mural image restoration model; The mural image restoration model includes: an edge prior guidance module for processing damaged edge information of the mural, a multi-stage incremental connection feature module, and a module focusing on the global context information of the mural image; in the first stage, i.e., the edge restoration stage, the features generated in the first stage are used in the second stage, i.e., the texture restoration stage, to perform mural image restoration on the texture module to be restored with complex textures; A training module uses the preprocessed mural images to train a mural image restoration model; The restoration module uses the trained mural image restoration model to restore mural images with complex textures.

Citation Information

Patent Citations

  • Text image restoration model and method based on structural attention and text perception

    CN116258652A

  • Damaged mural image restoration method and system

    CN116993617A

  • Diffusion model-based mural image restoration method under small sample condition

    CN118172289A

  • Customized space guidance system and method

    KR102381834B1

  • Systems, methods, and apparatuses for implementing self-supervised visual representation learning using order and appearance recovery on a vision transformer

    US20230306562A1

Cited By

  • Mural virtual restoration method based on hyperspectral image

    CN119379563A

  • A virtual restoration method for murals based on hyperspectral images

    CN119379563B

  • Image restoration method based on diffusion mode and prior feature generation and related device

    CN120088170A

  • Image Inpainting Method and Related Device Generated Based on Diffusion Mode and Prior Features

    CN120088170B

  • Transform-based GPR image clutter removal method

    CN121120425A