Porcelain image restoration method and system based on structure guidance and correlation reconstruction

Abstract

The invention discloses a porcelain image restoration method and system based on structure guidance and correlation reconstruction, and belongs to the technical field of image processing. According to the invention, the multi-scale cavity view selection module extracts the structural features of the multi-scale view by adopting a sparse sampling mode, adjusts the structure of a missing region by using the structural information of the neighborhood of the missing region, and constructs a multi-scale structural feature map mainly based on local information restoration; space attention is introduced to adaptively select multi-scale structure restoration results for superposition and fusion, so that the rationality of a restored image structure and the accuracy of subsequent detail feature generation are improved; the progressive gating memory module can capture the pixel correlation of the features under different scales, effectively enhance the learning ability of the module for the features of different scales, and improve the correlation between the missing region generation features and the non-missing region features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a porcelain image restoration method and system based on structure guidance and correlation reconstruction. BACKGROUND

[0002] Porcelain has a history of thousands of years, recording information of culture and living customs of each period, and has great historical research significance. However, due to the influence of material properties, some precious porcelain collections have cracks, discoloration, damage and loss during preservation and inheritance.

[0003] The existing porcelain restoration technology, especially in texture restoration, mainly relies on manual completion, and it is difficult to achieve efficient and accurate restoration. As one of the research hotspots in the field of computer image, the image restoration technology based on deep learning not only has certain advantages in repairing aging marks, removing occlusions and feature completion, but also has short time consumption and high restoration quality. However, the current image restoration technology in the field of porcelain image restoration cannot fully extract the feature structure, and cannot effectively utilize the feature correlation in the texture structure of porcelain, so that the restored image has some complex texture generation content structure blur and distortion.

[0004] The above problems need to be solved, therefore, a porcelain image restoration method and system based on structure guidance and correlation reconstruction are proposed. SUMMARY

[0005] The technical problem to be solved by the present application is how to solve the problems of complex surface texture of porcelain image, blurred edge and strong dependency between features in the existing porcelain restoration technology, and inconsistent style of the same features in different pictures, and provide a porcelain image restoration method based on structure guidance and correlation reconstruction, so that the restored image has clear texture, complete structure and coherent semantics.

[0006] The present application solves the above technical problems by the following technical solutions, and comprises the following steps:

[0007] S1: Constructing a data set

[0008] A plurality of porcelain cultural relic images are collected and processed to obtain a porcelain cultural relic image data set, and the data set is divided into a training set and a test set according to a set proportion;

[0009] S2: Building an image restoration network

[0010] The empty convolution is combined with the spatial attention to design a multi-scale empty visual field selection module, the sliding window self-attention mechanism is combined with the gating mechanism to design a progressive gating memory module, and then the multi-scale empty visual field selection module and the progressive gating memory module are fused into a two-stage restoration network structure to obtain the image restoration network.

[0011] S3: training an image inpainting network

[0012] Based on the training set samples and test set samples in the porcelain cultural relic image dataset, an image inpainting network is trained by constructing a comprehensive loss function, and a trained image inpainting model is obtained.

[0013] S4: image inpainting

[0014] The image to be inpainted is input into the trained image inpainting model, and the inpainted image is output, i.e. an inpainted image is obtained.

[0015] Further, in the step S1, the specific processing process is as follows:

[0016] S11: Obtain multiple porcelain cultural relic images by shooting and direct collection;

[0017] S12: Perform data enhancement and manual screening on the porcelain cultural relic images to obtain a porcelain cultural relic image dataset.

[0018] Further, in the two-stage inpainting network of step S2, two stages are included:

[0019] In the first stage, an 8-layer encoder-decoder structure with upsampling and downsampling is introduced, multi-level features are extracted to capture global context information, and a skip connection mechanism is used to fuse the features at each level of the encoder and decoder to obtain a semantic structure feature map with global information;

[0020] In the second stage, the encoder is a double-branch structure, the lower branch uses a multi-scale field of view selection module to reorganize the structure guided by the feature map with local field of view information, and the upper branch uses a parallel residual structure with convolution kernels of 3 and 5 to learn and fuse the semantic structure feature map generated by the global information in the first stage and the structure guided feature map generated by the local field of view information in the lower branch; a progressive gating memory module is introduced in the middle layer of the encoder and the decoder; the decoder uses a parallel residual structure and a skip connection structure to output the inpainted image.

[0021] Further, in the step S2, the multi-scale field of view selection module includes a multi-scale field of view baseline unit and a field of view selection unit.

[0022] In the multi-scale field of view baseline unit, the input feature map is sequentially connected with depth separable convolution layers with a dilation rate of 1, 3 and 5 to obtain feature maps of different scales, and a 1x1 projection matrix is used to process the output of each layer to obtain feature maps f0, f1 and f2.

[0023] In the field of view selection unit, the feature maps f0, f1 and f2 are first spliced in the channel, and then the spatial position weight feature map and the structure peak value feature map are generated by using the global average pooling and the global maximum pooling. The spatial position weight feature map and the structure peak value feature map are spliced, and then are upgraded to 3 channels by 1×1 convolution and are normalized by the sigmoid function to generate a multi-scale weight matrix attn. Then, the three-layer output of the multi-scale hollow field baseline unit is weighted by respectively splitting the channels. Then, the weighted feature maps are fused by using the corresponding element addition method, and then the feature peak value is adjusted by 1×1 convolution to obtain the output feature map of the multi-scale hollow field selection module.

[0024] Further, in the field of view selection unit, the calculation expression of the multi-scale weight matrix attn is as follows:

[0025] attn=sigmoid(Conv 1×1 (C(GAP(A),GMP(A))))

[0026] Wherein, sigmoid() is an activation function, which is used for input data normalization operation; Conv 1×1 is a normal convolution with a convolution kernel of 1×1, GAP() is global average pooling, GMP() is global maximum pooling, C is dimension splicing, A=C(f0′, f1′, f2′), f i ′=W i f i , W i is a 1×1 projection matrix, and i is 0, 1 or 2.

[0027] Further, in the progressive gating memory module of step S2, an information processing unit, a progressive unit and a memory unit are included.

[0028] In the information processing unit, the input feature map is first upgraded by 1×1 convolution, and then the upgraded feature map is cut into two parts with the same number of channels: one part is used as the query vector Q1 of the sliding window self-attention mechanism; the other part is expanded to two parts by using a depth separable convolution with a convolution kernel of 3×3, which is used as the value vector V1 and V2 of the sliding window self-attention mechanism.

[0029] In the progressive unit, Q1 and V1 are first input into the sliding window self-attention mechanism with a window size of 5×5, and then the output of the self-attention mechanism is processed by the gating convolution, and the output is used as the first level output. The first level output is input into the next layer sliding window self-attention mechanism with a window size of 7×7 as Q2 and V2, so that the self-attention weight matrix is iteratively updated, and then the output is weighted after V2 is weighted, and then the output is processed by the gating convolution as the second level output.

[0030] In the memory unit, the input feature map is first processed by the gate convolution, and the weighted feature map after self-attention of each layer of gate convolution is uniformly input into the memory line, and then 3*3 convolution is used for cross-channel feature fusion and dimension compression to obtain the output feature map under the condition that the input and output channel numbers of the module are the same.

[0031] Further, in the progressive unit, the calculation formula of the self-attention mechanism is as follows:

[0032]

[0033] Where O i is the weighted feature map of the i-th layer sliding window self-attention mechanism, K i ', V i ' are the key vector and value vector after local window sliding operation by the unfold method in the i-th layer sliding window self-attention mechanism, K i ' = unfold(Q i ), V i ' = unfold(V i ), softmax() is an activation function, and d is the group channel number of the self-attention mechanism.

[0034] Further, in the step S3, the comprehensive loss function calculation formula is as follows:

[0035] L = λ L1 L L1 + λ adv L adv + λ per L per + λ style L style + λ tv L tv

[0036] Where L L1 is the pixel reconstruction loss, L adv is the adversarial loss, L per is the perceptual loss, L style is the style loss, L TV is the total variation loss, λ L1 , λ adv , λ per , λ style and λ tv are the weight proportions of the corresponding losses.

[0037] Further, the pixel reconstruction loss measures the pixel-level difference between the repaired image and the real image by L1 norm, and the calculation formula is as follows:

[0038]

[0039] wherein, and δ (x,y) are pixel values of the repaired image and the real image at the (x, y) position respectively, and n is the number of pixel points;

[0040] Adversarial loss L adv for measuring the discriminability of the generative adversarial network and the similarity between the repaired image and the real image, and the calculation formula is as follows:

[0041]

[0042] wherein, E() is an expectation operation, D(δ) and D(δ), are the discrimination values of the real image and the repaired image respectively, and both are generated by the discriminator;

[0043] Perceptual loss L per by comparing the feature map difference of the real image and the repaired image in the trained VGG-16 network, to measure the consistency of the repaired image and the real image in semantic content and detail information, and the calculation formula is as follows:

[0044]

[0045] wherein, δ i are the feature maps of the repaired image and the real image in the i-th layer of the VGG-16 network respectively, and N i is the pixel number of the i-th layer feature map in the VGG-16 network, and ||·|| is the L1 norm;

[0046] Style loss function L style by comparing the Gram matrix difference of the real image and the repaired image in the feature map of each layer of the trained VGG-16 network, to measure the consistency of the statistical characteristics of the texture and style distribution of the repaired image and the real image, and the calculation formula is as follows:

[0047]

[0048] wherein, δ i T δ i are the Gram matrices of the repaired image and the real image in the i-th layer feature map of the VGG-16 network.

[0049] The application also provides a porcelain image repairing system based on structure guidance and correlation reconstruction, which is used for completing the porcelain image repairing work by using the above method, and comprises:

[0050] A data set construction module is configured to collect multiple ceramic cultural relic images and process the same to obtain a ceramic cultural relic image data set, and divide the data set into a training set and a test set according to a set proportion;

[0051] A network construction module is configured to combine a dilated convolution with a spatial attention, design a multi-scale dilated visual field selection module, combine a sliding window self-attention mechanism with a gating mechanism, design a progressive gating memory module, and then fuse the multi-scale dilated visual field selection module and the progressive gating memory module into a two-stage repair network structure to obtain an image repair network.

[0052] A network training module is configured to train the image repair network based on training set samples and test set samples in the ceramic cultural relic image data set and construct a comprehensive loss function, and obtain a trained image repair model.

[0053] An image repair module is configured to input a to-be-repaired image into the trained image repair model and output a repaired image, i.e., obtain a repaired image.

[0054] Compared with the prior art, the present application has the following advantages:

[0055] 1. The two-stage repair network designed in the present application combines structure guidance and correlation reconstruction to process the problem that the edges of the surface texture of the porcelain are not uniform in color and level and there is strong dependency between the features. The structure guidance part includes global semantic structure features mined by a long-distance encoder-decoder and structure features extracted and utilized by a local visual field repair in a self-adaptive visual field selection module, and the two complement each other to enhance the accuracy of guiding the generation of detailed texture. The correlation reconstruction part depends on the calculation of feature correlation by self-attention to guide feature reconstruction with correlation information.

[0056] 2. The multi-scale dilated visual field selection module designed in the present application can analyze and combine structure features of different sizes. It uses three depth separable convolutions with different dilated rates to perform sparse sampling and extract image structure features in different visual fields, and uses spatial attention to adaptively adjust the weights of each region in different visual field feature maps to obtain the most explanatory structure features.

[0057] 3. The progressive gating memory module designed in the present application not only effectively enhances the multi-scale feature learning capability of the module, but also integrates global context information while performing local feature mining and reconstruction, realizes the unification of local correlation reconstruction and global correlation, and uses a progressive hierarchical strategy to stack windows of different sizes layer by layer, propagate local correlation to the global, calculate the feature correlation matching reconstruction feature map in the window, store the feature map adjusted by the gating convolution to the unified memory line layer by layer, and finally fuse the feature maps in the memory line to improve the structure and style consistency of the overall image. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 This is a flowchart illustrating the porcelain image restoration method using structure-guided and correlation-based reconstruction in Embodiment 1 of the present invention.

[0059] Figure 2 This is a schematic diagram of the image restoration network in Embodiment 1 of the present invention;

[0060] Figure 3 This is a schematic diagram of the multi-scale hole field of view selection module in Embodiment 1 of the present invention;

[0061] Figure 4 This is a schematic diagram of the progressive gating memory module in Embodiment 1 of the present invention;

[0062] Figure 5 This is a qualitative comparison diagram of the experimental results of the self-made porcelain dataset comparison verification in Embodiment 2 of the present invention. In the diagram, (a) is a real image example, (b) is a real image damaged image (input image) example after processing, (c) is an example of the repair result of the RFR algorithm, (d) is an example of the repair result of the AOT algorithm, (e) is an example of the repair result of the CTSDG algorithm, (f) is an example of the repair result of the MISF algorithm, (g) is an example of the repair result of the SPA algorithm, (h) is an example of the repair result of the PTG algorithm, and (i) is an example of the repair result of the present invention. Each row in the diagram corresponds to a set of examples. Detailed Implementation

[0063] The embodiments of the present invention are described in detail below. These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0064] Example 1

[0065] like Figure 1 As shown, this embodiment provides a technical solution: a method for restoring porcelain images based on structure-guided and correlation reconstruction, comprising the following steps:

[0066] Step 1: Design a multi-scale hole field of view selection module;

[0067] Step 2: Design a progressive gating memory module;

[0068] Step 3: Construct the image restoration network;

[0069] Step 4: Train and test the image restoration network to obtain the image restoration model;

[0070] Step 5: Input the image to be repaired into the image repair model, and output the repaired image to obtain the repaired image.

[0071] In this embodiment, the specific processing procedure in step one is as follows:

[0072] The designed multi-scale dilated view selection module consists of two parts: a multi-scale dilated view baseline and a view selection mechanism. Dilated convolution can extract features through sparse sampling, reducing the impact of missing pixels and irrelevant features. Depthwise separable convolution can reduce the computational cost of convolution, achieve channel separation, and extract regional features. The multi-scale dilated view baseline combines the advantages of both, using depthwise separable convolutions with dilation rates of 1, 3, and 5 as the sparse sampling baseline. In the baseline, the convolutions with these three dilation rates are concatenated to extract image structural features. A 1×1 convolution is used for nonlinear transformation of the channel dimension to achieve feature peak alignment and information enhancement, outputting structural feature maps f0, f1, and f2 at three scales.

[0073] In the field selection part, the three feature maps f0, f1, and f2 are concatenated using the Cancat method. After concatenation, Global Average Pooling (GAP) is used to generate a spatial location weight feature map to capture multi-dimensional feature dependencies. At the same time, Global Max Pooling (GMP) is used to obtain the structural peak feature map to highlight significant structural features. The concatenated dual-pooling feature maps are then fused by 1×1 convolution, and after dimensionality increase (2→3 channels), they are processed by the sigmoid function to generate a learnable multi-scale weight matrix attn, the expression of which is:

[0074] attn = sigmoid(Conv 1×1 (C(GAP(A),GMP(A))))

[0075] Here, sigmoid() is the activation function used to normalize the input data; Conv 1×1 This is a regular convolution with a 1×1 kernel. GAP() is global average pooling, GMP() is global max pooling, C is dimension concatenation, and A = C(f0′, f1′, f2′). i ′=W i f i W i It is a 1×1 projection matrix, where i takes the values ​​0, 1, or 2.

[0076] The generated multi-scale weight matrix is ​​split into dimensions using the chunking method and weighted to the feature map generated by each view. The feature maps of different views are then fused by adding corresponding elements to reorganize the structural features. A 1×1 convolution is used to adjust the feature peaks to obtain the module output feature map.

[0077] In this embodiment, the specific processing procedure in step two is as follows:

[0078] The designed progressive gating memory module consists of three parts: information processing, progression, and memory. The information processing part uses a 1×1 convolution to increase the dimensionality of the initial input feature map of the module, and then cuts the increased feature map into two parts with the same number of channels: one part serves as the query vector Q1 of the self-attention mechanism; the other part, to ensure that the features are not affected by channel interactions while increasing the number of channels of the feature map and adjusting the missing regions to enable multi-scale feature weighting, uses a 3×3 depthwise separable convolution to expand this part of the feature map into two parts, which serve as the value vectors V1 and V2 of the self-attention mechanism.

[0079] In the progressive part, the feature matrices Q1 and V1 are first input into a sliding window self-attention mechanism with a window size of 5×5. By sliding the window locally, the features generated in the missing region can dynamically aggregate relevant pixel information within the window range, adaptively adjusting the weight matrix of the missing region and changing the feature pixel values ​​of the missing region to achieve correlation reconstruction. Then, the self-attention output is processed by gated convolution to reduce the influence of irrelevant pixels. The output is used as the first-level output, which is then used as Q2 and V2 to input into the next-level sliding window self-attention mechanism with a window size of 7×7. This causes the self-attention weight matrix to be iteratively updated, realizing multi-stage window stacking, thereby establishing long-distance correlations between features at different levels. After weighting V2, the output is then processed by gated convolution as the second-level output. The self-attention mechanism expression is:

[0080]

[0081] Among them, O i K is the weighted feature map of the i-th layer sliding window self-attention mechanism. i ′、V i ′ and ′ are the key vector and value vector respectively after the local window sliding operation using the unfold method in the i-th layer sliding window self-attention mechanism, K. i ′=unfold(Q i V i = unfold(V) i `softmax()` is the activation function used to normalize the input data, and `d` is the number of channels in the self-attention mechanism, with a size of 64.

[0082] In the memory part, gated convolution is first applied to process the initial input of the module, and then the self-attention weighted feature map processed by each layer of gated convolution is input into a unified memory line. Then, 3×3 convolution is used to perform cross-channel feature fusion and dimensionality compression to obtain the output feature map while keeping the number of input and output channels of the module the same.

[0083] In this embodiment, the specific processing procedure in step three is as follows:

[0084] The constructed image inpainting network consists of two stages. In the first stage, an 8-layer upsampling encoder-decoder structure is introduced to extract features at multiple levels to capture global contextual information. A skip connection mechanism is used to achieve efficient fusion of features at each level of the encoder and decoder, solving the semantic discontinuity problem caused by the loss of high-level features in long-distance model inpainting. This allows the construction of a basic semantic structure feature framework in the damaged area, which serves as a scalable feature prior to ensure that the subsequent detailed inpainting stage can integrate the local context of the damaged area for adaptive inference, gradually completing the semantically coherent structural optimization and detail completion.

[0085] In the second stage, to extract richer structural information and improve the structural rationality of the generated image, the encoder establishes a dual-branch network structure: the lower branch uses a multi-scale dilated field-view selection module, which extracts image features from the perspective of multi-scale local fields of view using dilated convolution kernels with sparse distribution characteristics, and constructs a structure-guided feature map mainly for local information repair. The three-layer downsampling gradually adjusts the structural features while aligning the number of feature channels and the size of the feature map between the upper and lower layers and passing them to the upper layer; the upper branch is a global-local information structure-guided feature map fusion branch. This branch uses a small kernel parallel residual structure with 3 and 5 convolution kernels to learn the detailed information of the non-missing regions of the input image, so as to repair the detailed features of the missing regions. During the repair process, it receives the structural features passed from the lower encoder, strengthens the overall structural information, and enhances the model's ability to learn the corresponding positions of missing pixels in the overall structure of the image details, thereby perceiving the overall layout of the image and guiding the generation of detailed information based on structural information to achieve the purpose of accurate repair. The intermediate layer between the encoder and decoder uses a sliding window self-attention mechanism in the progressively gated memory module to partition the overall information, creating a region feature pixel correlation attention map. This map performs correlation-weighted reconstruction on missing regions, adjusting the structure and details of the missing parts to improve the stylistic consistency of image features. The decoder employs a skip connection approach to refine the original input feature information and uses the same residual structure as the encoder to fine-tune the repaired image, refining edges and further improving image quality, outputting the image restoration result.

[0086] In this embodiment, the specific processing procedure in step four is as follows:

[0087] Training on a self-made porcelain dataset: The network training loss function includes pixel reconstruction loss L. L1 , countering losses L adv Perceived loss L per Style loss L style With total variational loss L TV The losses are as follows:

[0088] The pixel reconstruction loss is measured by the L1 norm, which represents the pixel-level difference between the restored image and the original image. The calculation formula is as follows:

[0089]

[0090] in, and δ (x,y) These are the pixel values ​​at position (x,y) of the restored image (image restoration result) and the real image, respectively, where n is the number of pixels.

[0091] Combat loss L adv The formula used to measure the discriminative power of generative adversarial networks and the similarity between the restored image and the real image is as follows:

[0092]

[0093] Where E() is the expectation operation, and D(δ) is the discriminant value of the real image, generated by the discriminator, which is ideally close to 1. On the contrary, It is the discriminant value for the repaired image, also generated by the discriminator, and ideally close to 0.

[0094] Perceived loss L per The consistency between the restored and real images in semantic content and detail information is measured by comparing the differences in feature maps of each layer in the trained VGG-16 network between the real and restored images. The calculation formula is as follows:

[0095]

[0096] in, δ i These are the feature maps of the restored image and the real image in the i-th layer of the VGG-16 network, respectively, and N is the sum of the feature maps of the restored image and the real image. i Let ||||| be the number of pixels in the feature map of the i-th layer in the VGG-16 network, and ||·|| be the L1 norm.

[0097] Style loss function L style The consistency of the statistical properties of texture and style distribution between the restored and real images is measured by comparing the Gram matrix differences of feature maps at each layer of the trained VGG-16 network between the real and restored images. The calculation formula is as follows:

[0098]

[0099] in, δ i T δ i These are the Gram matrices of the feature maps of the i-th layer in the VGG-16 network for the restored image and the real image, respectively.

[0100] Total variational loss function L tv The smoothness of the restored image is measured by comparing the differences between adjacent pixels. The calculation formula is as follows:

[0101]

[0102] in, They are respectively The pixel values ​​below and to the right.

[0103] The formula for calculating the total loss function is as follows:

[0104] L=λ L1 L L1 +λ adv L adv +λ per L per +λ style L style +λ tv L tv

[0105] Among them, L L1 For pixel reconstruction loss, L adv To combat the losses, L per To perceive loss, L style For style loss, L TV For the total variational loss, λ L1 , λ adv , λ per , λ style With λ tv This represents the weighting percentage of the corresponding loss.

[0106] Example 2

[0107] This embodiment conducts corresponding experiments on the repair method in Embodiment 1, as detailed below:

[0108] Dataset: The experiment used images of porcelain artifacts collected from cameras and public online resource databases such as Baidu Images. After data augmentation methods such as random cropping and rotation, and manual screening to remove images with single or irrelevant content, the dataset of porcelain artifact images was expanded to about 28,000 images. Among them, 27,000 images were used as training images, 1,000 images were used as test images, and 12,000 irregular masked images were used to train and test the network.

[0109] Comparison methods: RFR, a recurrent feature inference inpainting algorithm that strengthens hole edge information through iterative inference to gradually repair the image; AOT, an image inpainting algorithm that stacks multi-layer, multi-scale dilated convolutional residual blocks; CTSDG, an image inpainting algorithm that combines structure and texture dual generation, proposing a bidirectional gated feature fusion module and a context feature aggregation module; MISF, a multi-level interactive twin filtering image inpainting algorithm, mainly composed of kernel prediction and semantic and image filtering branches; SPA, an efficient and lightweight image inpainting algorithm that combines Transformer and Generative Adversarial Network; PTG, a smooth-change image inpainting algorithm that repairs from coarse to fine.

[0110] Evaluation Metrics: The experiment used Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) as the core evaluation metrics. PSNR calculates the logarithm of the mean square error between the restored image and the original image, objectively reflecting pixel-level reconstruction accuracy; a higher value indicates less distortion. SSIM, based on human visual characteristics, comprehensively compares the brightness, contrast, and structural information of two images to evaluate perceptual quality; its value ranges from [0,1], with values ​​closer to 1 indicating higher visual consistency. LPIPS extracts multi-level features and calculates differences through a pre-trained deep neural network, better simulating the perceptual characteristics of the human visual system; a lower value indicates higher perceptual similarity between the restored image and the original image.

[0111] Experimental Details: The hardware environment used in the experimental phase of this invention employed an Intel(R) Core(TM) i7-11700K processor and an NVIDIA GeForce RTX 3070 graphics card (8GB VRAM). The software environment was based on the PyTorch 2.1.2 deep learning framework, coupled with a Python 3.11 interpreter and CUDA 12.1 acceleration library. The development tool used was the PyCharm integrated environment. In terms of experimental setup, the input images were preprocessed, uniformly cropped, and scaled to a resolution of 256×256. A batch training strategy was adopted, with a batch size of 2 to balance VRAM usage and training stability. The weight hyperparameters of each part of the loss function were set to λ. L1 =λ tv =λ adv =0.1, λ per =0.05, λ style=120. The network training consisted of 100 epochs, employing a dynamic learning rate scheduling strategy. The first 80 epochs used an initial learning rate of 1e-4 to promote rapid model convergence, while the last 20 epochs reduced the rate to 1e-5 for fine-tuning to ensure stable optimization of the model parameters. Experiments validated the model performance through quantitative and qualitative evaluations.

[0112] Quantitative Evaluation: Table 1 shows that, within the mask coverage range of 0% to 40%, the method of this invention achieved the highest PSNR, SSIM, and LPIPS values ​​compared to other methods. When the mask coverage was between 0% and 30%, the PSNR was improved by 0.56 dB, 0.48 dB, and 0.48 dB respectively compared to the second-best CTSDG method; the SSIM was improved by 0.1%, 0.2%, and 0.4% respectively; and the LPIPS were reduced by 0.1%, 0.3%, and 0.2% respectively. When the mask coverage was between 30% and 40%, the PSNR was improved by 0.30 dB compared to the second-best SPA method; the SSIM was improved by 0.7%; and the LPIPS were reduced by 0.2%.

[0113] Table 1. Quantitative comparison results of different methods on the self-made porcelain dataset.

[0114]

[0115]

[0116] It should be noted that in the table above, bold text indicates the optimal value for each column, "↓" indicates that the lower the better, and "↑" indicates that the higher the better. Ours refers to the image restoration model in this invention.

[0117] Qualitative evaluation: Figure 5 The qualitative experimental results used to validate the self-made porcelain dataset are analyzed as follows:

[0118] like Figure 5 As shown:

[0119] like Figure 5 (c) in the image shows the porcelain image restored by the RFR method. The restored area has a large number of artifacts and the texture is blurred and difficult to distinguish.

[0120] like Figure 5 In the image (d), the texture restoration of the porcelain image restored by the AOT method is incomplete, and there are certain restoration artifacts.

[0121] like Figure 5 In (e), the porcelain image restored by the CTSDG method has a problem with a complex and disordered texture structure, and the restored structure has a broken phenomenon.

[0122] like Figure 5(f) in the image shows that the porcelain image restored by the MISF method has blurred edges, distorted and disordered complex pattern structure, and discontinuous generated features under large mask.

[0123] like Figure 5 In the image, (g) represents the porcelain image restored by the SPA method where features at small to medium distances are stuck together, while features at greater distances cannot be properly connected.

[0124] like Figure 5 In the image (h), the porcelain image restored by the PTG method shows obvious blurring in the damaged area.

[0125] like Figure 5 As shown in (i), compared with other algorithms, the repair results of the method of the present invention show certain advantages in terms of boundary clarity, structural integrity and visual effect.

[0126] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A porcelain image inpainting method based on structure guidance and correlation reconstruction, characterized in that, The method comprises the following steps: S1: constructing a data set Collect multiple ceramic cultural relic images and process them to obtain a ceramic cultural relic image data set, and divide the data set into a training set and a test set according to a set proportion; S2: building an image inpainting network Combine the empty convolution and the spatial attention to design a multi-scale empty visual field selection module, combine the sliding window self-attention mechanism and the gating mechanism to design a progressive gating memory module, then fuse the multi-scale empty visual field selection module and the progressive gating memory module into a two-stage inpainting network structure to obtain the image inpainting network; S3: training the image inpainting network Based on the training set samples and the test set samples in the ceramic cultural relic image data set, and constructing a comprehensive loss function to train the image inpainting network to obtain a trained image inpainting model; S4: image inpainting Input the image to be inpainted into the trained image inpainting model to output an inpainted image, that is, obtain the inpainted image.

2. The porcelain image inpainting method based on structure guidance and correlation reconstruction according to claim 1, characterized in that, In the step S1, the specific processing process is as follows: S11: Obtain multiple ceramic cultural relic images by shooting and direct collection; S12: Perform data enhancement and manual screening on the ceramic cultural relic images to obtain a ceramic cultural relic image data set.

3. The porcelain image inpainting method based on structure guidance and correlation reconstruction according to claim 1, characterized in that, In the two-stage inpainting network of the step S2, two stages are included: In the first stage, an 8-layer encoder-decoder structure with upsampling and downsampling is introduced, multi-level features are extracted to capture global context information, and a skip connection mechanism is used to realize the fusion of features at each level of the encoder and the decoder to obtain a semantic structure feature map with global information; In the second stage, the encoder is a double-branch structure, the lower branch uses a multi-scale empty visual field selection module to reorganize the structure guided by the feature map with local visual field information, and the upper branch uses a parallel residual structure with a convolution kernel of 3 and 5 to learn and fuse the semantic structure feature map generated by the global information in the first stage and the structure guided feature map generated by the local visual field information in the lower branch; the intermediate layer of the encoder and the decoder introduces a progressive gating memory module; The decoder uses a parallel residual structure and a skip connection structure to output the inpainted image.

4. The porcelain image inpainting method based on structure guidance and correlation reconstruction according to claim 1, characterized in that, In the step S2, the multi-scale empty visual field selection module includes a multi-scale empty visual field baseline unit and a visual field selection unit; In the multi-scale empty visual field baseline unit, the input feature map is sequentially connected with depth separable convolution layers with empty rates of 1, 3 and 5 to obtain feature maps of different scales, and a 1x1 projection matrix is used to process the output of each layer to obtain feature maps f0, f1 and f2; In the visual field selection unit, the feature maps f0, f1 and f2 are first spliced in the channel, then global average pooling and global maximum pooling are used to generate a spatial position weight feature map and a structure peak feature map, the spatial position weight feature map and the structure peak feature map are spliced, then the 1x1 convolution is used to upgrade the dimension to 3 channels and the sigmoid function is used for normalization processing to generate a multi-scale weight matrix attn, then the channels are split to weight the outputs of the three layers of the multi-scale empty visual field baseline unit, then the corresponding elements are added to fuse the weighted feature maps, and then the 1x1 convolution is used to adjust the feature peak to obtain the output feature map of the multi-scale empty visual field selection module.

5. The porcelain image inpainting method based on structure guidance and correlation reconstruction according to claim 4, characterized in that, In the field of view selection unit, the calculation expression of the multi-scale weight matrix attn is as follows: attn = sigmoid(Conv 1×1 (C(GAP(A),GMP(A)))) wherein sigmoid() is an activation function for realizing input data normalization operation; Conv 1×1 is a normal convolution of a convolution kernel 1x1, GAP() is global average pooling, GMP() is global maximum pooling, C is dimension concatenation, A=C(f0', f1', f2'), f i '=W i f i , W i is a 1x1 projection matrix, i takes 0, 1, 2.

6. The porcelain image inpainting method based on structure guidance and correlation reconstruction according to claim 5, characterized in that, In the progressive gating memory module of the step S2, an information processing unit, a progressive unit and a memory unit are included; In the information processing unit, 1*1 convolution is first used to upgrade the input feature map, and then the upgraded feature map is cut into two parts with the same number of channels: one part is used as the query vector Q1 of the sliding window self-attention mechanism; the other part is expanded to two parts by using a convolution kernel of 3*3, which is used as the value vector V1 and V2 of the sliding window self-attention mechanism; In the progressive unit, Q1 and V1 are first input into the sliding window self-attention mechanism with a window size of 5*5, and then the output of the self-attention mechanism is processed by the gating convolution, and the output is used as the first level output. The first level output is input into the next layer sliding window self-attention mechanism with a window size of 7*7 as Q2 and V2, so that the self-attention weight matrix is iteratively updated, and the output after weighting V2 is processed by the gating convolution as the second level output. In the memory unit, the input feature map is first processed by the gating convolution, and then input into the memory line together with the self-attention weighted feature map processed by each layer of the gating convolution. Then, 3*3 convolution is used for cross-channel feature fusion and dimension compression to obtain the output feature map under the condition that the number of input and output channels of the module is the same.

7. The porcelain image inpainting method based on structure guidance and correlation reconstruction according to claim 6, characterized in that, In the progressive unit, the calculation formula of the self-attention mechanism is as follows: wherein O i is the weighted feature map of the i-th layer sliding window self-attention mechanism, K i ', V i are the key vector and the value vector after the local window sliding operation in the i-th layer sliding window self-attention mechanism using the unfold method, K i '=unfold(Q i ), V i '=unfold(V i ), softmax() is an activation function, and d is the grouping channel number of the self-attention mechanism.

8. The porcelain image inpainting method based on structure guidance and correlation reconstruction according to claim 1, characterized in that, In the step S3, the comprehensive loss function calculation formula is as follows: L = λ L1 L L1 + λ adv L adv + λ per L per + λ style L style + λ tv L tv wherein, L L1 is the pixel reconstruction loss, L adv is the adversarial loss, L per is the perceptual loss, L style is the style loss, L TV is the total variation loss, λ L1 , λ adv , λ per , λ style and λ tv are the weight proportions of the corresponding losses.

9. The porcelain image inpainting method based on structure guidance and correlation reconstruction according to claim 8, characterized in that, The pixel reconstruction loss measures the pixel-level difference between the restored image and the real image in the L1 norm, and the calculation formula is as follows: wherein, and δ (x,y) are the pixel values of the repaired image and the real image at the (x, y) position, respectively, and n is the number of pixels. Adversarial loss L adv For measuring the discriminative ability of the generative adversarial network and the similarity of the repaired image and the real image, the calculation formula is as follows: Where E() is the expectation operation, D(δ), The discrimination values ​​for the real image and the restored image are respectively generated by the discriminator; Perceptual loss L per The consistency of the inpainting image and the real image in semantic content and detail information is measured by comparing the difference of feature maps at each layer of the trained VGG-16 network between the real image and the inpainting image, and the calculation formula is as follows: wherein, δ i respectively are the feature maps of the repaired image and the real image at the i-th layer of the VGG-16 network, N i is the number of pixels of the i-th layer feature map of the VGG-16 network, ||·|| is the L1 norm. Style loss function L style The consistency of the statistical characteristics of the texture and style distribution of the repaired image and the real image is measured by comparing the Gram matrix difference of each layer feature map of the trained VGG-16 network of the real image and the repaired image, and the calculation formula is as follows: wherein, δ i T δ i are the Gram matrices of the i-th layer feature maps of the repaired image and the real image in the VGG-16 network, respectively.

10. A porcelain image inpainting system based on structure guidance and correlation reconstruction, characterized in that, A method for repairing porcelain images is provided, which comprises the following steps: A data set construction module is configured to collect and process multiple porcelain cultural relic images to obtain a porcelain cultural relic image data set, and divide the data set into a training set and a test set according to a set proportion; A network building module is configured to combine the dilated convolution and the spatial attention to design a multi-scale dilated field selection module, combine the sliding window self-attention mechanism and the gating mechanism to design a progressive gating memory module, and then integrate the multi-scale dilated field selection module and the progressive gating memory module into a two-stage repair network structure to obtain an image repair network; A network training module is configured to train the image repair network based on the training set samples and the test set samples in the porcelain cultural relic image data set and construct a comprehensive loss function. An image repair module is configured to input a to-be-repaired image into the trained image repair model to output a repaired image, i.e., a repaired image.

Images