A full-automatic mural digital restoration method of fusing feature extractors

Abstract

This invention belongs to the field of digital preservation of cultural relics and relates to a fully automated digital restoration method for murals that integrates a feature extractor, aiming to achieve high-quality restoration of murals. The method includes: S1: Constructing and training a mural style feature extractor to extract style feature vectors representing the artistic characteristics of the mural from the mural image; S2: Detecting missing areas in the mural image based on a detection model and outputting the bounding box information of the missing areas; S3: Generating a binary segmentation mask for the missing areas based on a segmentation model, using the bounding box information as a cue; S4: Repairing the missing areas based on a diffusion restoration model, combining the style feature vector and the binary segmentation mask, and outputting the restored mural image.

Description

Technical Field

[0001] This invention relates to the field of digital preservation of cultural relics, and in particular to a fully automated digital restoration method for murals that integrates a feature extractor. Background Technology

[0002] Ancient murals, as precious historical and cultural heritage, are generally damaged by long-term natural weathering, humidity changes, and human factors, resulting in various types of defects such as paint peeling, cracking, and staining. These defects severely affect their artistic value and the integrity of their historical information. Traditional manual restoration methods rely heavily on the experience and subjective judgment of restorers, resulting in inherent limitations such as low efficiency, poor consistency, and weak reproducibility, making it difficult to meet the urgent need for large-scale digital conservation and restoration of murals.

[0003] With the development of computer vision and deep learning technologies, generative model-based image restoration methods have been gradually applied to the field of digital cultural relic preservation. Existing technologies typically employ a general framework of mask-guided restoration combined with generative adversarial networks or diffusion models, where missing areas are manually annotated before the generative model completes the restoration. However, these methods have significant shortcomings in mural restoration: firstly, general feature extraction networks struggle to capture the unique mineral pigment materials, traditional pattern structures, and line tension characteristic of murals, resulting in a stylistic disconnect between the restored area and the original mural; secondly, most existing methods rely on manual pre-annotation of missing areas, failing to achieve end-to-end automated processing from the original mural to the restoration result, severely limiting restoration efficiency and scalability.

[0004] Furthermore, even though some studies have attempted to introduce style transfer techniques, there is still a problem of insufficient coordination between style characteristics and the restoration process: style guidance often remains at the overall level and fails to be effectively combined with the time-stepping mechanism of the diffusion model; the restoration of damaged areas lacks spatial selectivity constraints, which can easily interfere with intact areas. These technical deficiencies make it difficult for existing methods to achieve high-quality digital restoration with natural form and consistent style while maintaining the authenticity of the mural. Summary of the Invention

[0005] To address the problems mentioned in the background art, this invention provides a fully automated digital restoration method for murals that integrates feature extractors, aiming to achieve high-quality restoration of murals.

[0006] The present invention adopts the following technical solution: The present invention provides a fully automatic digital restoration method for murals by fusing feature extractors, comprising: S1: constructing and training a mural style feature extractor to extract style feature vectors representing the artistic characteristics of the mural from the mural image; S2: detecting missing areas in the mural image based on a detection model and outputting the bounding box information of the missing areas; S3: generating a binary segmentation mask for the missing areas based on a segmentation model and using the bounding box information as a prompt; S4: repairing the missing areas based on a diffusion restoration model, combining the style feature vectors and the binary segmentation mask, and outputting the restored mural image.

[0007] The process of constructing the mural style feature extractor in step S1 includes: S11: extracting the material features of the mural image through the spectral perception layer; S12: extracting the pattern features of the mural image through the pattern analysis layer; S13: extracting the line tension features of the mural image through the line tension layer; the style feature vector is generated by fusing the material features, pattern features and line tension features.

[0008] In step S11, the standardized mural image is processed by the spectral simulation convolution module to output a material feature map that represents the degree of material matching.

[0009] In step S12, after downsampling the material feature map, the morphological features of the pattern units are extracted by multi-size convolution kernels; the pattern arrangement pattern features are extracted by autocorrelation operation; and the pattern unit morphological features and pattern arrangement pattern features are fused to output the pattern feature map.

[0010] In step S13, after downsampling the pattern feature map, the line direction features are extracted by direction-sensitive convolution kernel; the quantization parameters of the lines are calculated based on the line direction features; the quantization parameters are integrated and pooled to output the line tension feature vector.

[0011] The detection model is an improved model based on the YOLO framework. The training process of the detection model includes: introducing cross-scale feature consistency constraints in the neck feature fusion module; constructing a total loss function that includes detection loss and feature consistency loss; training the model using the mural defect dataset and outputting high-confidence defect region bounding boxes.

[0012] The segmentation model is a fine-tuned model based on the SAM model. The training process of the segmentation model includes: using bounding boxes as cue information, training the model to output pixel-level binary segmentation masks; using region consistency loss as the core loss function; and using a fine-tuning strategy of freezing pre-trained weights and updating local parameters.

[0013] The diffusion repair model includes a mask gating module and a style feature modulation unit. The mask gating module is used to constrain the repair range during noise prediction. The style feature modulation unit is used to dynamically fuse the style feature vector with the time step of the diffusion process. The style feature modulation unit splits the style features into multiple sub-vectors according to the decoder level to achieve hierarchical style guidance.

[0014] The mask gating module works by: adjusting the binary segmentation mask to the same spatial size as the feature map; generating a spatial gating signal to attenuate the noise prediction value of the preserved region; and fusing the spatial gating signal with the noise prediction result to achieve the constraint of the repair range.

[0015] The training loss function of the diffusion restoration model in step S4 includes: diffusion loss, global style loss, and hierarchical style loss; among which, diffusion loss is used to optimize the pixel-level reconstruction accuracy of the missing area; global style loss is used to ensure that the restoration result is consistent with the overall style of the original mural; and hierarchical style loss is used to enhance the gradual formation of style features during the diffusion process.

[0016] Compared to existing technologies, the beneficial effects are as follows:

[0017] 1. This application provides a fully automated digital restoration method for murals using a fusion feature extractor. Through a closed-loop design encompassing dedicated style extraction, automated detection and segmentation, and dynamic style restoration, it precisely addresses the shortcomings of existing mural restoration techniques, such as poor style adaptability and heavy reliance on manual intervention. By using a spectral simulation convolution module in the spectral perception layer to match the spectral reflectance patterns of high-frequency mineral pigments in murals, such as azurite and ochre, it accurately outputs material feature maps, avoiding the omission of the unique mural attribute of the mineral pigment substrate by general image features. Combined with multi-size convolution and autocorrelation operations in the pattern analysis layer, it can extract the unit morphology of typical patterns such as lotus and flying apsara patterns, and capture symmetrical and continuous arrangement patterns. Furthermore, through direction-sensitive convolution and quantization calculations in the line tension layer, it restores the curvature changes and intersection angles of the mural lines, among other tension characteristics. The dedicated style feature vector generated by the fusion of these three layers provides a unified style reference for restoration, encompassing material, pattern, and line, ensuring a high degree of consistency between the restored content and the mural's dynastic style and artistic logic, avoiding style misalignment caused by general features.

[0018] 2. Compared with the shortcomings of existing technologies where the style features are fixed and cannot be adapted to the needs of the restoration stage, this invention designs a dynamic style feature modulation unit in the diffusion restoration model. The exclusive style feature vector is split into sub-vectors of line focus, pattern focus, and material focus according to the decoder level. These sub-vectors are adapted to the progressive needs of structural construction, pattern filling, and material refinement in the restoration process, so that the restoration content, from macro structure to micro material, is seamlessly connected with the original mural, solving the problems of style fragmentation and stiff details.

[0019] 3. An improved detection model based on the YOLO framework introduces cross-scale feature consistency constraints in the neck feature fusion module, enhancing the consistency of feature representation across small, medium, and large-scale feature maps for similar defects, thus avoiding missed or false detections due to scale differences. A segmentation model fine-tuned from a SAM pre-trained model uses the bounding boxes output by the detection model as cue information. A lightweight fine-tuning strategy of freezing the image encoder and updating the mask decoder is employed, combined with region consistency loss to enhance the overall region matching between the mask and the annotation, especially optimizing the segmentation accuracy of the defect edges. The generated binary segmentation mask accurately defines the defect range, ensuring that subsequent restoration only affects the defective area without interfering with the original intact parts, thus preserving the original information of the cultural relic. Attached Figure Description

[0020] Figure 1 This is a flowchart of a fully automated digital restoration method for murals using a fusion feature extractor, provided in an embodiment of this application.

[0021] Figure 2 This is a schematic diagram of a three-layer architecture for a style feature extractor provided in an embodiment of this application;

[0022] Figure 3 This is a schematic diagram illustrating the implementation of a mask gating module provided in an embodiment of this application;

[0023] Figure 4 This is a schematic diagram illustrating the implementation of a style feature modulation unit that integrates time steps, as provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0025] This application provides a fully automated digital restoration method for murals that integrates feature extractors. For example, such as... Figure 1 As shown, it includes:

[0026] S1: Construct and train a mural style feature extractor to extract style feature vectors that characterize the artistic features of murals from mural images.

[0027] In some embodiments, the process of constructing the mural style feature extractor in step S1 includes: S11: extracting the material features of the mural image through the spectral sensing layer; S12: extracting the pattern features of the mural image through the pattern analysis layer; S13: extracting the line tension features of the mural image through the line tension layer; the style feature vector is generated by fusing the material features, pattern features and line tension features.

[0028] Reference Figure 2 A three-tiered mural style feature extractor, constructed through a progressive logic of material perception, pattern analysis, and line tension, extracts the material features, pattern features, and line tension features of the mural image, respectively.

[0029] As one possible implementation, in step S11, the standardized mural image is processed by a spectral simulation convolution module to output a material feature map representing the degree of material matching.

[0030] For example, suppose the input is a mural image that has undergone size normalization and basic color normalization, with a unified spatial resolution of 512×512 and RGB color channels. A spectral simulation convolution module is constructed, containing three sets of 1×1 convolution kernels. Each set contains 16 independent 1×1×3 convolution kernels, respectively matching the spectral response characteristics of three frequently used mineral pigments in murals: azurite, ochre, and cinnabar. Based on the spectral reflectance characteristics of mineral pigments, azurite has the highest reflectance in the blue light band and the lowest in the red light band. Therefore, the weighting coefficients of the azurite convolution kernel for the RGB channels are set according to the priority B>R>G. Ochre and cinnabar have significantly higher reflectance in the red light band than in the blue-green light band; therefore, the weighting coefficients of the ochre and cinnabar convolution kernels for the RGB channels are set according to the priority R>G>B.

[0031] Each 1×1×3 convolutional kernel can be represented as a 3-dimensional weight vector. ,in These are the weight coefficients for the corresponding RGB channels. For any pixel position in the image... Its RGB value The calculation using a single 1×1×3 convolution kernel is as follows:

[0032]

[0033] in The convolution kernel outputs a feature map on the k-th channel. The response value of the location, This is the bias term for the convolution kernel. Through this operation, each convolution kernel weights and fuses the pixel values ​​of the RGB three channels into a single-channel feature, representing the degree of matching between that location and the spectral characteristics of the corresponding mineral pigment.

[0034] Each convolutional kernel outputs a 16-channel feature map, maintaining a spatial resolution of 512×512, used to quantify the spectral matching degree between different areas of the mural and the corresponding mineral pigments. The three output feature maps are concatenated along their channel dimensions to form a 48-channel feature map. A single 3×3 convolutional layer is used to fuse the concatenated feature maps, with a stride of 1 and zero-padding at the edges to maintain the 512×512 feature map resolution. The final output is a 48-channel material feature map, where each pixel value represents the degree of matching between the corresponding location and the mineral pigment.

[0035] As one possible implementation, in step S12, after downsampling the material feature map, the morphological features of the pattern units are extracted by multi-size convolution kernels; the pattern arrangement pattern features are extracted by autocorrelation operation; and the pattern unit morphological features and the pattern arrangement pattern features are fused to output the pattern feature map.

[0036] For example, the 48×512×512 material feature map output from the spectral sensing layer is downsampled using a 3×3 convolution with a stride of 2 to obtain a 48×256×256 feature map, which serves as the input to the pattern analysis layer. A unit extraction module is constructed, employing a multi-size convolution kernel mechanism. 5×5 convolutions are used for small pattern units, and 7×7 convolutions are used for large pattern units, for a total of 64 kernels. All convolutions have a stride of 1, and zero padding is used at the edges to maintain the 256×256 feature map resolution, outputting a 64-channel unit feature map. Taking the 5×5 convolution as an example, each kernel is a 5×5×48 three-dimensional tensor (5×5 being the spatial dimension and 48 being the number of input channels). For any 5×5×48 local feature block in the input feature map, the calculation process is as follows:

[0037]

[0038] in Let be the weight of the k-th 5×5 convolutional kernel at position i,j and channel c, and let be the feature value of the input feature map at the corresponding position. This is the bias term. Through this operation, 64 convolutional kernels output 64 feature maps of 256×256, with each channel corresponding to the morphological features of a specific pattern unit.

[0039] A pattern-building module is constructed to perform autocorrelation operations on the unit feature map, calculating the feature similarity between each pixel and its 8 neighboring pixels: for each pixel in the unit feature map, its 8 neighboring pixels are taken. , , The calculation is as follows:

[0040]

[0041] in For the unit feature map in eigenvectors, Let n be the coordinates of the nth neighboring pixel. This represents the Gaussian kernel bandwidth.

[0042] The result is a feature map reflecting the arrangement pattern of the pattern units. The feature map is then compressed by 1×1 convolution to output a 32-channel feature map of the arrangement pattern, while maintaining a spatial resolution of 256×256. Each channel corresponds to the response intensity of a certain arrangement pattern (such as symmetrical or continuous).

[0043] Finally, the 64-channel unit feature map and the 32-channel patterned feature map are concatenated along the channel dimension to form a 96×256×256 feature map, which is then fused using 64 3×3 convolutional kernels (stride 1, zero padding).

[0044]

[0045] in This is the spliced ​​feature map. and To integrate the weights and biases of the convolution kernels, a 64×256×256 pattern feature map is finally output, which integrates the correlation information between the shape and arrangement pattern of the pattern units.

[0046] As one possible implementation, in step S13, after downsampling the pattern feature map, the line direction features are extracted by a direction-sensitive convolution kernel; the quantization parameters of the lines are calculated based on the line direction features; the quantization parameters are integrated and pooled to output the line tension feature vector.

[0047] For example, the 64×256×256 pattern feature map output by the pattern analysis layer is downsampled using a 3×3 convolution with a stride of 2 to obtain a 64×128×128 feature map, which is then used as the input to the line tension layer. .

[0048] A line detection module is constructed, configured with 8 3×3 directional sensitive convolutional kernels, with an angle interval of 22.5°, covering 0° to 157.5°. Each convolutional kernel is a 3×3×64 three-dimensional tensor. Taking the 0° directional convolutional kernel as an example, its weight distribution exhibits the edge detection characteristic of "positive weights in the middle row and negative weights in the top and bottom rows," producing a strong response to horizontal lines. For the input 64×128×128 feature map, the calculation of each directional convolutional kernel is as follows:

[0049]

[0050] in Indicates direction index. The weights of the convolution kernel in the d-th direction are used. After stride 1 and zero-padding, an 8×128×128 directional feature map is output. The pixel value of each channel quantizes the response intensity of the corresponding directional line.

[0051] A tension calculation module is constructed by first binarizing the directional feature map to obtain line masks in eight directions. For each connected line in the mask, the rate of curvature change is calculated: taking three consecutive pixels on the line... , , The curvature is estimated using second-order difference, and the difference between adjacent curvatures is calculated as the rate of change. Simultaneously, the number of pixels and the intersection angles of lines in different directions are statistically analyzed.

[0052] The above quantized features are integrated into a tension feature map using 32 3×3 convolutional kernels (8 input channels, 32 output channels):

[0053]

[0054] in The weights of the tension convolution kernel are used to output a 32×128×128 tension feature map, where each channel corresponds to the spatial distribution of a certain quantized feature of the line. Global average pooling is then performed on the tension feature map to output a 32-dimensional vector. This vector quantization characterizes the statistical properties of line tension.

[0055] The 48×512×512 material feature map output from the spectral sensing layer is downsampled to 256×256 using a 3×3 convolution with a stride of 2. This downsampled feature map is then concatenated with the 64×256×256 pattern feature map output from the pattern analysis layer along the channel dimension, forming a 112-channel feature map. This concatenated feature map is compressed using a 3×3 convolution with a stride of 1 and zero edge padding, outputting a 64×256×256 feature map. Global average pooling is performed on this feature map to obtain a 64-dimensional vector, which is then concatenated with the 32-dimensional concatenated vector output from the line tension layer to form a 96-dimensional vector. This 96-dimensional vector is then fused using a 1×1 convolution, outputting a 96-dimensional mural-specific style feature vector. This vector integrates the correlation information of material, pattern, and line, completing the construction of a three-order progressive architecture.

[0056] To further enhance the ability of the style feature extractor to extract line structure, pattern logic, and material-specific features from mural images, the style feature extractor is trained as follows:

[0057] A mural style feature dataset was constructed, containing images of murals from different dynasties and in different preservation states, including both restored and unrestored murals. The dataset covers various substrate types such as clay, stone, and brick, as well as typical pattern types such as lotus, flying apsara, and scrolling grass patterns. Each mural image was annotated, including mineral pigment type, pattern unit type and arrangement pattern, line tension feature labels, and missing bounding boxes and masks of the missing areas, preparing for subsequent training of multiple models.

[0058] The complete image data is preprocessed, including standardizing the size to 512×512, normalizing the basic colors, and dividing it into training, validation and test sets according to the proportions.

[0059] The network initialization parameters are set as follows: the weights of the three 1×1 convolutional kernels in the spectral sensing layer are initialized based on the spectral response characteristics of the mineral pigments; the weights of the remaining convolutional layers are initialized using a He normal distribution; and the bias term is initialized to 0. The optimizer is set to Adam, and the initial learning rate is dynamically adjusted using a cosine annealing strategy. The weight decay coefficient is initialized, and the batch size is set according to the hardware configuration. The number of training epochs is set to N. After each epoch, the validation set loss is calculated. If the validation loss does not decrease for M consecutive epochs, training is stopped, and the optimal model parameters are saved.

[0060] Material matching loss is calculated using mean square error loss. The material feature map output by the spectral sensing layer is matched with the pixel-level mask labeled by the mineral pigments. The expression is:

[0061]

[0062] in, For the feature map height and width, For the number of channels, Output the material feature map. The label is a mineral pigment mask (a pixel value of 1 indicates that the corresponding pigment is at that position, and 0 indicates otherwise).

[0063] The pattern analysis loss is calculated by combining cross-entropy loss and cosine similarity loss. :

[0064]

[0065] in, To calculate the cross-entropy of the pattern unit classification loss, the vector of the unit feature map after global pooling is calculated with the pattern unit label. For the pattern loss, the cosine distance between the pattern feature map and the feature vector of the labeled pattern (symmetric, radial, etc.) is calculated, and the expression is:

[0066]

[0067] in, The characteristic vector of the arrangement pattern The feature vector is the arrangement pattern of the annotation.

[0068] The line feature loss is calculated using mean squared error loss, and the 32-dimensional line feature vector output from the line tension layer is matched with the labeled line feature reference vector. The line feature reference vector is constructed based on quantified parameters such as the rate of curvature change, length distribution, and intersection angle of the mural lines, and its expression is:

[0069]

[0070] Where is the d-th dimension feature of the line tension layer output vector. The d-th dimension of the labeled reference vector is used to enhance the learning accuracy of the line tension characteristics through the loss model.

[0071] Define a multi-task loss function, which is a weighted sum of the material matching loss, pattern parsing loss, and line feature loss mentioned above. The expression is as follows:

[0072]

[0073] in, Let be the weight coefficient, and satisfy... .

[0074] The model training process is executed, followed by testing and optimization. Training set images are input into the network in batches, and forward propagation yields material, pattern, and dynasty features. The multi-task loss function is calculated. Network parameters are updated via backpropagation, and the optimizer adjusts the weights of each layer based on the loss gradient. After each training round, model performance is evaluated on the validation set, and the accuracy of feature extraction is calculated. Various metrics are calculated on the test set, and the loss function weight coefficients are adjusted. The training process is repeated until the requirements are met, and finally, the trained style feature extractor model is saved.

[0075] S2: Based on the detection model, detect the missing areas of the mural image and output the bounding box information of the missing areas.

[0076] In some embodiments, the detection model is an improved model based on the YOLO framework; the training process of the detection model includes: introducing cross-scale feature consistency constraints in the neck feature fusion module; constructing a total loss function that includes detection loss and feature consistency loss; training the model using a mural defect dataset and outputting high-confidence defect region bounding boxes.

[0077] The YOLO framework is used as the basic architecture for detecting mural defects. The core of this study is to address the problem of poor consistency in detecting similar defects (such as peeling at different locations or cracks with similar shapes) and the missed detection of some samples in mural scenes. Scene adaptation optimization is achieved by introducing cross-scale feature consistency constraints.

[0078] The overall architecture of the detection model follows the three-level structure of YOLO: backbone network, neck feature fusion, and head detection output. The backbone network adopts CSPDarknet, which is responsible for extracting multi-scale semantic features from the input image. The neck feature fusion module adopts PANet structure to realize cross-scale fusion of features at different levels. The head output layer is set as a single-class detection branch, which is specifically for bounding box prediction and confidence calculation for the "mural defect area".

[0079] A new cross-scale feature consistency module is added to the neck feature fusion module. The core principle of this module is to reduce the risk of missed detections due to scale differences by strengthening the consistency of feature representation of the same type of defect across feature maps of different scales. Specifically, three key scale feature maps output by PANet (denoted as P3, P4, and P5, corresponding to receptive fields from smallest to largest) are selected. Each scale feature map is compressed to 256 dimensions using a 1×1 convolution, resulting in feature maps F3, F4, and F5 of uniform dimension. For any sample of the same type of defect in the training batch, feature vectors Fs (small-scale features), Fm (medium-scale features), and Fl (large-scale features) corresponding to the defect region are extracted from F3, F4, and F5, respectively. The cosine similarity between any two scale feature vectors is calculated to construct the cross-scale feature consistency loss function.

[0080]

[0081] This loss function penalizes the feature differences of the same type of defect at different scales, forcing the model to learn scale-independent and consistent defect feature representations, thereby improving the detection stability of the same type of defect.

[0082] The overall loss function and training strategy of the detection model are determined. The mural dataset built during the training of the style feature extractor is reused. This dataset already includes bounding box and category annotations for the missing regions. The training and validation sets are divided according to a predetermined ratio. The overall loss function is a weighted sum of YOLO's native CIoU loss and cross-scale feature consistency loss, expressed as:

[0083]

[0084] in For CIoU loss weights, The feature consistency loss weights satisfy the following conditions: ,and The value should be no less than 0.2 to ensure the effectiveness of consistency constraints during training. The optimizer used is SGD, with an initial learning rate set to a preset value. A cosine annealing strategy is employed to dynamically adjust the learning rate, and the weight decay coefficient is set according to experimental results to suppress overfitting. During training, after each iteration, the false negative rate (the percentage of undetected samples among similar defects) and mean accuracy (mAP@0.5) of the validation set are calculated. When the false negative rate and mean accuracy remain excellent and stable for several consecutive iterations, training is stopped and the model parameters are saved.

[0085] The inference process of the detection model is as follows: The mural image to be detected is first standardized to adjust its spatial resolution to match the training data, and then input into the trained detection model. The detection model extracts multi-scale features through a backbone network. After processing by the neck feature fusion module and the cross-scale feature consistency module, the head output layer generates the bounding box coordinates of the defect area. Confidence scoring is used. A confidence threshold of 0.85 is set, and bounding boxes with confidence scores higher than this threshold are selected as valid detection results. False detection regions with low confidence scores are removed, and finally, the bounding box information of all valid missing regions is output, providing accurate region range input for subsequent segmentation module inference.

[0086] S3: Based on the segmentation model, using bounding box information as a cue, generate a binary segmentation mask for the missing region.

[0087] In some embodiments, the segmentation model is a fine-tuned model based on the SAM model; the training process of the segmentation model includes: using bounding boxes as cue information, training the model to output pixel-level binary segmentation masks; using region consistency loss as the core loss function; and using a fine-tuning strategy of freezing pre-trained weights and updating local parameters.

[0088] We employ SAM (SegmentAnythingModel) as the foundational architecture for fine-grained segmentation of mural defect areas. Its core relies on a flexible mechanism for prompting and mask generation, adapted to the mask generation task under bounding box constraints in mural defect scenarios. The model as a whole follows the three-tiered structure of SAM: image encoder, prompt encoder, and mask decoder. The image encoder uses a VisionTransformer architecture to encode the input image into a high-dimensional feature map; the prompt encoder processes bounding box prompts to generate corresponding prompt features; and the mask decoder, based on the fusion of image and prompt features, outputs a pixel-level segmentation mask.

[0089] For example, the training data and task settings for the segmentation model are first determined. The training process reuses the mural dataset used during the training of the style feature extractor and the detection model. This dataset already contains pixel-level mask annotations and corresponding bounding box annotations for the missing areas. The training task is set as "mask prediction guided by bounding box cues," that is, the input is the mural image and the bounding box of the missing area, and the output is a binary segmentation mask consistent with the annotations. The focus is on optimizing the prediction accuracy of the mask at the edges of the missing areas.

[0090] The training loss function and fine-tuning strategy for the segmentation model are designed. Considering the potential ambiguity at the edges of the damaged areas of the mural (such as the transition zone between moldy and normal areas), Dice loss is adopted as the core loss function to enhance the overall consistency between mask prediction and annotation. Its expression is as follows:

[0091]

[0092] in This represents the value of the mask predicted by the model at each pixel. The corresponding values ​​for the labeled mask are used. The fine-tuning strategy adopts a frozen pre-trained weights and local parameter update mode: all parameters of the image encoder are frozen, and only the parameters of the cross-attention layer and output convolutional layer of the mask decoder are updated to reduce the number of training parameters and accelerate convergence.

[0093] The training and validation process of the segmentation model is executed. During training, mural images, corresponding bounding box cues, and pixel-level mask annotations are input in batches. The segmentation model generates predicted masks through feature fusion and calculates the Dice loss based on the difference between the predicted results and the annotations to drive parameter updates. An adaptive optimizer is used to regulate the learning process, and a weight decay mechanism is introduced to suppress overfitting. In the validation phase, after each training round, the average Dice coefficient and edge pixel matching rate are evaluated on an independent validation set. When these metrics remain stable for several consecutive rounds without improvement, the model is considered to have converged, training is stopped, and the parameters are saved.

[0094] S4: Based on the diffusion restoration model, combined with style feature vectors and binary segmentation masks, the damaged areas are restored, and the restored mural image is output.

[0095] In some embodiments, the diffusion repair model includes a mask gating module and a style feature modulation unit; the mask gating module is used to constrain the repair range during noise prediction; the style feature modulation unit is used to dynamically fuse the style feature vector with the time step of the diffusion process; wherein, the style feature modulation unit splits the style features into multiple sub-vectors according to the decoder level to achieve hierarchical style guidance.

[0096] As one possible implementation, the mask gating module works by: adjusting the binary segmentation mask to the same spatial size as the feature map; generating a spatial gating signal to attenuate the noise prediction value of the preserved region; and fusing the spatial gating signal with the noise prediction result to achieve the repair range constraint.

[0097] For example, refer to Figure 3 We construct the basic architecture of a mask-guided diffusion repair model and adopt an improved U-Net diffusion model. The core relies on the "conditional diffusion process of the defect area" to achieve the repair range constraint. Its technical principle is to control the spatial scope of noise prediction by segmenting the mask.

[0098] The input layer receives the image of the mural to be restored. Binary segmentation mask and random noise that follows a standard normal distribution The diffusion process proceeds according to a preset time step. Noise is gradually added to the image to generate a noisy image. Its calculation formula is ,in These are predefined noise scheduling parameters used to control the noise intensity at different time steps.

[0099] To achieve the constraint of "only repairing the masked missing areas", and to ensure that the model always focuses on the missing areas and ignores the preserved areas in the process from "feature extraction" to "noise prediction", a mask gating module is embedded in each residual block of U-Net (a total of 8, 4 for the encoder and 4 for the decoder).

[0100] Let the input feature map of the current residual block be... , dimension , where is the number of channels. , To determine the spatial dimensions, first, the segmentation mask is... Perform bilinear interpolation to adjust its spatial dimensions to Obtain the adaptation mask Configure a 1×1 convolutional kernel with the same number of channels as the current residual block (weight matrix is...). ),right Perform channel expansion:

[0101]

[0102] in This represents the convolution operation. For the bias term, output Dimensions .

[0103] right The sigmoid activation function is used to generate a spatially gated signal, so that the value range of each element of G is limited to [0,1]:

[0104]

[0105] The corresponding location of the reserved area The location corresponding to the defective area .

[0106] The noise prediction branch within the residual block outputs the original noise. After (with the same dimension as X), it is fused with the gated signal G through element-wise multiplication:

[0107]

[0108] in This represents the Hadamard product operation. This operation forces the noise prediction values ​​of the retained regions to be attenuated, while only the noise prediction values ​​of the missing regions are retained.

[0109] Finally, The residual is added to the input feature X of the residual block to complete the residual connection, ensuring that the mask constraint runs through the entire feature extraction and noise prediction process.

[0110] Through the above mechanism, the diffusion repair model applies spatially selective constraints to noise prediction in each residual block, and strengthens the characteristic of "only repairing the defective area" layer by layer from the bottom feature to the top feature, so as to avoid the interference of the repair process on the original preserved area.

[0111] Reference Figure 4 The design incorporates a cross-time-step fusion mechanism for style features and diffusion processes. Style feature modulation units are embedded in each time step of the decoder. The core principle is to adapt style features to time steps and perform hierarchical modulation processing to achieve precise guidance of style features for different generation stages.

[0112] Perform time-step adaptation of style features, and convert the 96-dimensional style feature vector output by the style feature extractor constructed in step S1. (Including material characteristics) Pattern characteristics Line features The input is a linear mapping layer, which also introduces time step encoding information—the time step t is converted into a high-dimensional vector through sinusoidal position encoding, as shown in the following formula:

[0113]

[0114] in For encoding dimensions, Generate time-step encoding vectors consistent with the F dimension. .

[0115] Will and spliced ​​together in the channel dimension Input linear mapping layer:

[0116]

[0117] in For the mapping weight matrix, As a bias term, it outputs a style feature vector that adapts to the current time step, ensuring that the style features are dynamically adjusted over time as the diffusion process progresses.

[0118] Will The vectors are split into three sub-vectors according to the decoder level, corresponding to the upsampling of the first three layers:

[0119] Corresponding to 64×64 layers, focus on line features:

[0120] Corresponding to 128×128 layers, focusing on pattern features

[0121] Corresponding to 256×256 layers, focusing on material characteristics

[0122] Emphasizing the key points of each stage by using time-step-dependent weighting coefficients. , , ,in , , These represent the weights of the features at different time steps.

[0123] A style feature modulation unit is inserted between the transposed convolution and feature fusion stages of the first three upsampling layers in the decoder. Taking the first upsampling layer as an example:

[0124] Transposed convolution output feature map After being concatenated with the corresponding layer skip connection features (64×64×256) of the encoder, it is input into the style feature modulation unit:

[0125] In step 32 Expanded to a 256-dimensional channel vector through 1×1 convolution. For each channel c, according to Perform channel scaling to enhance the guidance of line features on the global structure.

[0126] Style-modulated feature map , , The residual blocks of the corresponding layers of the decoder are input to participate in the upsampling process, and finally a 512×512 repair result is generated by transposed convolution of the 4th layer.

[0127] As one possible implementation, the training loss function of the diffusion restoration model in step S4 includes: diffusion loss, global style loss, and hierarchical style loss; wherein, diffusion loss is used to optimize the pixel-level reconstruction accuracy of the missing area; global style loss is used to ensure that the restoration result is consistent with the overall style of the original mural; and hierarchical style loss is used to enhance the gradual formation of style features during the diffusion process.

[0128] For example, the total loss function is expressed as follows:

[0129]

[0130] in Let be the weight coefficient, and satisfy... This ensures that style constraints play a dominant role in the loss function.

[0131] Diffusion loss This method optimizes pixel-level reconstruction accuracy for defective regions, and its computational scope is limited to the defective regions labeled by the segmentation mask: iterating through all time steps. With image pixels Calculation model predicts noise With real noise The L2 norm, multiplied by the segmentation mask. To perform region filtering, the result is normalized by dividing by the product of the total number of pixels in the defective region and the total number of time steps. The expression is:

[0132]

[0133] Global style loss To ensure consistency between the restored image and the overall style of the preserved area, the final restored image generated by the diffusion model is used. Input the feature extractor constructed in step 1 and obtain its output style feature vector. ;calculate The cosine distance between the feature vector F and the style feature vector of the preserved region, minus 1, is used as the loss value, expressed as:

[0134]

[0135] Where "・" represents the vector dot product operation. This represents the L2 norm of a vector.

[0136] Hierarchical style loss To enhance the gradual formation of style features during the diffusion process, the generation result corresponding to the intermediate time step of the diffusion process is selected. Compare it with the repair reference image Input the data into the feature extractor respectively; extract the intermediate feature maps of both layers in the spectral sensing layer, pattern analysis layer, and line tension layer; calculate the mean square error of each layer's feature map and sum them; the expression is:

[0137]

[0138] in The first The height, width, and number of channels of the layer feature map The feature map of the intermediate generated result. To repair the feature map of the reference image.

[0139] This application also provides an end-to-end inference chain for mural damage repair. The core relies on a pre-trained detection model, segmentation model, style feature extractor, and diffusion repair model. The entire process of automatic repair is achieved by following the progressive logic of "damage localization → fine segmentation → style extraction → constraint repair". The input and output of each module are seamlessly connected through standardized data formats to ensure the coherence of the inference process and the reliability of the results.

[0140] Performing defect region localization and fine segmentation: The image of the mural to be restored is input into the trained detection model. After feature extraction from the backbone network of the detection model, cross-scale feature fusion, and consistency constraint processing, the model outputs a set of bounding boxes for the defect regions with a confidence level ≥ 0.85 (each bounding box is represented by coordinates). (Characteristics); The image of the mural to be restored and the bounding box set are input into the fine-tuned segmentation model. Using the bounding boxes as prompts, a binary segmentation mask with the same size as the image is generated through the collaborative operation of image encoding, prompt encoding, and mask decoding. ,in Corresponding to the damaged area.

[0141] Perform style feature extraction on non-masked regions based on the binary mask output by the segmentation model. The original image of the mural to be restored was filtered by region, and pixels of the reserved areas were extracted to form the reserved area image. This operation is achieved through pixel-by-pixel multiplication. ( (For element-wise multiplication), ensuring that only intact mural areas are retained as input for feature extraction; The style feature extractor, after training, is input into the mineral pigment spectrum matching layer of the spectral perception layer, the unit and arrangement pattern analysis layer of the pattern analysis layer, and the direction detection and tension quantification layer of the line tension layer. Finally, a 96-dimensional style feature vector is output, which accurately represents the style attributes of the preserved area.

[0142] Stylized restoration under mask constraints is performed by inputting the mural image to be restored, a binary segmentation mask, and style feature vectors into a trained diffusion restoration model. The model initiates time-step denoising iterations according to a preset total time step. During the iteration, the mask gating module constrains the noise prediction range throughout the process using spatial gating signals, retaining only the noise prediction results for the missing areas. The style feature modulation unit fuses F with the time-step encoding and dynamically modulates the decoder features according to a predetermined staged strategy. When the iteration reaches time step t=0, the noise is completely removed, generating the final restored image.

[0143] The above description, in conjunction with preferred technical solutions, provides a further detailed explanation of the present invention and should not be construed as limiting the specific implementation of the invention to these descriptions. For those skilled in the art, simple deductions and substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A full-automatic mural digitalization repair method of a fusion feature extractor, characterized in that, include: S1: Construct and train a mural style feature extractor to extract style feature vectors representing the artistic characteristics of murals from mural images. The process of constructing the mural style feature extractor includes: S11: Extracting material features of the mural image through a spectral perception layer; in step S11, the standardized mural image is processed by a spectral simulation convolution module containing multiple sets of convolution kernels that match the spectral response characteristics of three mineral pigments: azurite, ochre, and cinnabar, and outputting a material feature map representing the degree of material matching; S12: Extracting pattern features of the mural image through a pattern analysis layer; S13: Extracting line tension features of the mural image through a line tension layer; the style feature vector is generated by fusing the material features, pattern features, and line tension features; S2: Detecting missing areas in the mural image based on a detection model and outputting the bounding box information of the missing areas; S3: Generating a binary segmentation mask for the missing areas based on a segmentation model, using the bounding box information as a prompt; S4: Repairing the missing areas based on a diffusion repair model, combining the style feature vector and the binary segmentation mask, and outputting the repaired mural image.

2. A method for full-automatic repair of mural digitalization according to the fusion feature extractor, according to claim 1, characterized in that, In step S12, after downsampling the material feature map, the morphological features of the pattern unit are extracted by multi-size convolution kernels; the pattern arrangement pattern features are extracted by performing autocorrelation operation on the morphological features of the pattern unit. By integrating the morphological features of the pattern units and the pattern arrangement rules, a pattern feature map is output.

3. The method of claim 2, wherein the method is a full-automatic digital restoration method for murals, characterized in that, In step S13, after downsampling the pattern feature map, the line direction features are extracted by direction-sensitive convolution kernel; the quantization parameters of the lines are calculated based on the line direction features; the quantization parameters are integrated and pooled to output the line tension feature vector.

4. The method of claim 1, wherein the method is a full-automatic mural digitalization repair method of fusion feature extractor, characterized in that, The detection model is an improved model based on the YOLO framework; the training process of the detection model includes: introducing cross-scale feature consistency constraints in the neck feature fusion module; constructing a total loss function that includes detection loss and cross-scale feature consistency loss; training the model using a mural defect dataset and outputting high-confidence defect region bounding boxes; wherein, the cross-scale feature consistency loss is used to penalize feature differences of the same type of defect at different scales.

5. The fully automated digital restoration method for murals according to claim 1, characterized in that, The segmentation model is a fine-tuned model based on the SAM model; the training process of the segmentation model includes: using bounding boxes as cue information, training the model to output pixel-level binary segmentation masks; using region consistency loss as the core loss function; and using a fine-tuning strategy of freezing pre-trained weights and updating local parameters.

6. The fully automated digital restoration method for murals according to claim 1, characterized in that, The diffusion repair model includes a mask gating module and a style feature modulation unit; the mask gating module is used to constrain the repair range during noise prediction; the style feature modulation unit is used to dynamically fuse the style feature vector with the time step of the diffusion process; wherein, the style feature modulation unit splits the style features into multiple sub-vectors according to the decoder level to achieve hierarchical style guidance.

7. The method of claim 6, wherein the method is a full-automatic mural digitalization repair method of fusing feature extractors, characterized in that, The working process of the mask gating module includes: adjusting the binary segmentation mask to the same spatial size as the feature map to obtain an adaptation mask; applying the Sigmoid function to the adapted mask after channel expansion to generate a spatial gating signal, which is used to attenuate the noise prediction value of the preserved region; and fusing the spatial gating signal with the noise prediction result to achieve the repair range constraint.

8. The method of claim 1, wherein the method is a full-automatic mural digitalization restoration method of fusing feature extractors, characterized in that, The training loss function of the diffusion restoration model in step S4 includes: diffusion loss, global style loss, and hierarchical style loss. The diffusion loss is used to optimize the pixel-level reconstruction accuracy of the missing area; the global style loss is used to ensure that the restoration result is consistent with the overall style of the original mural; and the hierarchical style loss, which is calculated and summed at the intermediate time step of the diffusion process, is used to enhance the gradual formation of style features during the diffusion process.

Images