Lightweight virtual restoration method for fresco based on inactive generative model

Abstract

The application discloses a kind of light-weight mural virtual repair methods based on non-activation generative model, belong to digital image processing and computer vision technical field.The existing deep learning repair model is aimed at solving the technical problems of unbalanced semantics and texture recovery when facing mural and other cultural heritage.By stacking the damaged image and the mask into a multi-channel tensor input into the pre-trained AFMNet model, the model uses an encoder-bottleneck-decoder architecture, the encoder uses a multi-domain partial convolution module without activation function to extract and fuse local texture and frequency domain global structure information and update the mask simultaneously, the bottleneck part models the lowest resolution feature and its mask through a lightweight multi-head attention module to capture long-range dependencies, and the decoder upsamples the features transmitted through the skip connection to reconstruct the image details.This method is mainly used for efficient and high-fidelity digital virtual repair of damaged ancient murals and other visual cultural heritage.

Description

Technical Field

[0001] This invention belongs to the field of digital image processing and computer vision, and relates to the digital virtual restoration of damaged visual cultural heritage (especially ancient murals) using deep learning technology. Specifically, it relates to a lightweight virtual restoration method for murals based on an activation-free generative model. Background Technology

[0002] Ancient murals are invaluable carriers of human cultural heritage, containing rich historical, cultural, and artistic information. They provide crucial insights into the socio-cultural life of various peoples during specific historical periods and possess immeasurable value for the study of human history and artistic development. However, due to the combined effects of natural weathering and human factors, these precious artworks are increasingly suffering from varying degrees of damage, including paint loss, surface cracking, scratches, and fissures. This damage not only undermines the artistic integrity of the murals but also threatens the preservation of the historical information they convey. Therefore, conservatism research on murals has become an urgent priority.

[0003] With the rapid development of computer vision and digital information technology, virtual restoration of damaged mural images has become a promising solution. Compared with traditional physical restoration methods, digital restoration has the advantages of being non-invasive, reversible, and repeatable. It can generate high-fidelity restoration plans without touching the artifact itself, providing a reference for physical restoration or for digital archiving, academic research, and public display.

[0004] In the field of heritage restoration, image restoration techniques are mainly divided into two categories. The first category is classical algorithms, such as sample-based and diffusion-based methods. These traditional techniques are effective in dealing with small, narrow damages (such as cracks), but their core limitation lies in their inability to understand the deep semantic content of the image. Therefore, when faced with large areas of missing material, they often produce results with blurred textures, distorted structures, or illogical content, failing to meet the high fidelity requirements of cultural relic restoration.

[0005] With the rise of deep learning, image inpainting technology has achieved revolutionary breakthroughs. While early convolutional neural networks (CNNs) and generative adversarial networks (GANs) improved performance, they were limited by the local receptive field of standard convolutions, making it difficult to capture long-distance image features. This resulted in insufficient semantic coherence when processing irregular or large-area missing areas. To address this issue, various advanced techniques have been developed, such as introducing attention mechanisms, Transformer architectures, or Fast Fourier Convolutions (FFC) to establish global contextual connections, and employing global-local dual discriminators to ensure overall harmony and realistic details. These methods have significantly improved the semantic consistency of the inpainting results.

[0006] In recent years, state-of-the-art generative models, such as diffusion models (e.g., Stable Diffusion), have achieved unprecedented levels of photorealism and semantic coherence. However, these models also share a significant drawback: extremely slow inference speed. Their iterative generation process requires numerous computational steps, resulting in high computational costs, which limits their practical application on devices requiring real-time feedback or with limited computing resources.

[0007] A comprehensive analysis of the limitations of existing technologies allows for a precise definition of the technological gap that this invention aims to fill. These challenges are not isolated but interconnected, collectively constituting the core difficulties in the field of digital restoration of cultural heritage.

[0008] (1) Problem 1: Semantic-texture dichotomy in murals Murals typically contain complex, non-repetitive semantic content (such as figures, animals, and mythological scenes) and fine, high-frequency textural details (such as brushstrokes and pigment particles). Existing technologies often struggle to perfectly restore both simultaneously. Models that excel at structural restoration may blur textural details, while models focused on texture generation may disrupt the overall semantic structure.

[0009] (2) Problem 2: Data Scarcity and Overfitting Dilemma An inherent characteristic of the cultural heritage field is the small size and highly specialized nature of datasets. The trend in general computer vision to improve performance by simply increasing model depth and the number of parameters is counterproductive in mural restoration tasks. Large models trained on limited mural data are prone to overfitting, resulting in poor generalization ability and an inability to handle unseen damage patterns.

[0010] (3) Problem 3: Low computational efficiency and deployment barriers. The most advanced models, such as large Transformer and diffusion models, have huge computational requirements, demanding powerful hardware resources and long inference times. This sets a high barrier for cultural heritage conservationists and researchers to apply the technology in practical work (especially in field environments), hindering its popularization and deployment.

[0011] (4) Problem 4: Error propagation in large missing areas. During the inpainting process, image inpainting models may use newly synthesized (and potentially defective) pixels as references for subsequent inpainting, which can amplify and propagate errors. This problem is particularly severe in continuous large-area damaged areas because the available reference information is very limited.

[0012] These technical challenges are deeply interconnected. Data scarcity (Problem 2) means that models cannot solve complex semantic-texture dichotomy problems (Problem 1) by learning from massive amounts of data as they do in general domains, unless their architecture is highly efficient and specialized. The pursuit of high performance has given rise to computationally intensive models (Problem 3), which is not only impractical but also exacerbates the risk of overfitting on small datasets (Problem 2). Therefore, a successful solution cannot address any one of these problems in isolation but must be a holistic design that balances efficiency, data sensitivity, and representational power simultaneously. Summary of the Invention

[0013] One object of the present invention is to solve at least the above-mentioned problems and / or defects, and to provide at least the advantages described below.

[0014] The "lightweight" nature of AFMNet in this invention is not only for improving efficiency, but also a prerequisite for robust learning in data-constrained environments, and a direct response to this series of interconnected challenges.

[0015] Therefore, the present invention provides the following technical solution:

[0016] A lightweight virtual restoration method for murals based on an activation-free generative model includes the following steps:

[0017] Receive input data, which includes a damaged image and a corresponding binary mask image, wherein the binary mask image is used to identify the complete area and the damaged area in the image;

[0018] The damaged image is stacked with the binary mask image to form a multi-channel input tensor;

[0019] The multi-channel input tensor is input into the trained AFMNet inpainting model to generate the inpainted complete image;

[0020] The AFMNet repair model employs an encoder-bottleneck-decoder architecture, and:

[0021] The encoder includes at least one multi-domain partial convolutional module (AFM Block) without activation function, which is used to simultaneously extract and fuse local texture information and global structural information based on frequency domain transformation of the input feature map, and to simultaneously update the input mask;

[0022] The bottleneck includes a lightweight multi-head attention module for modeling the lowest-resolution feature map output by the encoder and its corresponding mask to capture long-range dependencies.

[0023] The decoder includes at least one upsampling module for upsampling the feature map after the bottleneck processing and combining it with the feature map passed from the encoder via skip connections to reconstruct image details.

[0024] Preferably, in the lightweight virtual restoration method for murals based on an activation-free generative model, the execution steps of the activation-free multi-domain partial convolution module include:

[0025] Input feature map With input mask Multiply to obtain the masked feature map tensor. ;

[0026] The masked feature map tensor is input into a parallel spatial domain convolution path and a frequency domain convolution path, respectively.

[0027] The spatial domain convolution path extracts local texture features through standard convolution operations;

[0028] The frequency domain convolution path extracts global structural features by performing forward discrete cosine transform (DCT), frequency domain convolution operations, and inverse discrete cosine transform (IDCT).

[0029] Based on the number of valid pixels in the input mask, the fused feature map is renormalized. When there are valid pixels in the receptive field, the output feature map is scaled according to the ratio of the number of valid pixels to the total number of pixels.

[0030] The masked feature map tensor is propagated in the following manner:

[0031]

[0032] This represents the transpose of the convolution kernel for that path. This represents the transpose of the convolution kernel for that path. and These represent the forward discrete cosine transform and the inverse discrete cosine transform, respectively. , Calculate the sum of all elements in the matrix, where 1 is a matrix of all 1s of the same size as the mask M, sum(1) is the sum of all elements in the matrix of all 1s, and b is the bias term; preferably, the bias term b is a learnable parameter, which is initialized using the Xavier initialization method;

[0033] The outputs of the spatial domain convolution path and the frequency domain convolution path are concatenated and fused.

[0034] The renormalized feature map The features are sequentially input into a simplified channel attention (SCA) module and a simple gate module for feature optimization.

[0035]

[0036] Where SCA represents an adaptive average pooling and a 3 A combination of depthwise separable convolutions of size 3 is used, and finally a point convolution is applied to process the transformed feature map to obtain the output feature map. :

[0037]

[0038] Point convolution is used to integrate the optimized feature maps to obtain the output feature map;

[0039] The input mask is convolutionally downsampled to obtain the updated output mask.

[0040] Preferably, in the lightweight mural virtual restoration method based on an activation-free generative model, the execution steps of the lightweight multi-head attention module include:

[0041] The input feature map is processed using depthwise separable convolutions to significantly reduce computational complexity and the number of parameters, thereby generating query tensor Q, key tensor K, and value tensor V.

[0042] The query tensor, key tensor, and value tensor are split into multiple attention heads along the channel dimension;

[0043] For each attention head, calculate the product of its query tensor and the transpose of its key tensor to obtain the original attention score;

[0044] The original attention score is scaled using a learnable scaling parameter δ, enabling the model to dynamically adjust the sharpness of the attention distribution.

[0045] The input mask is integrated into the scaled attention score, and the score corresponding to the invalid position is set to negative infinity to suppress the model's attention to the damaged area.

[0046] Apply the Softmax function to the attention scores after masking to generate an attention map;

[0047] Multiply the attention map by the value tensor of the corresponding attention head to obtain a weighted feature output;

[0048] The weighted feature outputs of all attention heads are concatenated and fused using a single-point convolution to obtain the final output of the lightweight multi-head attention module, as expressed by the following formula:

[0049]

[0050] In the formula , , They represent the first The query tensor matrix, key tensor matrix, and value tensor matrix in the subspace where each attention head resides. The output projection matrix is ​​used to concatenate the outputs of multiple attention heads and map them back to the original dimensions. This represents the number of attention heads. In a preferred embodiment, the number of attention heads h can be set to 8; the initial value of the learnable scaling parameter δ can be set to 1.0 and updated by the optimizer during training.

[0051] Preferably, in the lightweight virtual restoration method for murals based on an activation-free generative model, the simple gating module implements nonlinear transformation in the following way:

[0052] The input feature map, after being processed by the simplified channel attention module, is divided into two parts along the channel dimension. The tensors of the two parts are respectively... ,

[0053]

[0054] Calculating the element-wise product of these two parts yields the tensor after the nonlinear transformation. : This serves as the output of the simple gating module.

[0055] Preferably, in the lightweight virtual mural restoration method based on an activation-free generative model, the trained AFMNet restoration model is obtained through the following method:

[0056] Obtain a training dataset, which includes multiple sets of training samples. Each set of training samples includes a complete image, a simulated damaged image, and a corresponding binary mask.

[0057] Construct an AFMNet repair model;

[0058] The AFMNet inpainting model is trained using a composite loss function, which is a weighted sum of pixel-level reconstruction loss, perceptual loss, style loss, and total variation loss.

[0059] The training employs a transfer learning strategy, including:

[0060] In the first stage, the AFMNet inpainting model is pre-trained using the training dataset to learn general image features and inpainting priors. The loss function used in the first stage of training is a composite loss function.

[0061]

[0062] For pixel-level reconstruction loss applied to known regions, For pixel-level reconstruction loss applied to missing regions, To compute the perceptual loss in the feature space of the pre-trained network, For style loss based on feature map Gram matrix, To promote spatial smoothness, the total variation loss is λ1 to λ5, which are the weights of each loss component.

[0063] In the second stage, after loading the weights of the pre-trained model obtained in the first stage, the AFMNet restoration model is tuned using a mural professional dataset to adapt the model to the semantic structure, artistic style and typical damage patterns of the murals.

[0064] Preferably, in the lightweight mural virtual restoration method based on an activation-free generative model, the multi-channel input tensor is a four-channel tensor, which is formed by stitching together the three-channel damaged image and the single-channel binary mask image along the channel dimension.

[0065] Preferably, in the lightweight mural virtual restoration method based on the non-activation generative model, the skip connection is used to transmit the multi-scale feature map from the corresponding level of the encoder and its corresponding updated mask.

[0066] The decoder's upsampling module uses the nearest neighbor interpolation algorithm for upsampling;

[0067] During the training of the AFMNet repair model, the optimal weight configuration for the composite loss function is: λ1=1.00, λ2=6.00, λ3=0.05, λ4=120.00, λ5=0.10.

[0068] Secondly, a lightweight image inpainting device based on an activation-free generative model includes:

[0069] A data receiving module is configured to receive input data, which includes a damaged image and a corresponding binary mask image.

[0070] The data preprocessing module is configured to stack the damaged image with the binary mask image to form a multi-channel input tensor;

[0071] The image inpainting module includes a trained AFMNet inpainting model configured to process the multi-channel input tensor to generate the inpainted complete image.

[0072] The AFMNet repair model adopts an encoder-bottleneck-decoder architecture, wherein the encoder includes at least one multi-domain partial convolution module without activation function, and the bottleneck includes at least one lightweight multi-head attention module.

[0073] An electronic device includes: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform any of the methods described above.

[0074] A storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods described herein.

[0075] The present invention has at least the following beneficial effects:

[0076] This invention systematically addresses the four core challenges in virtual restoration of murals (semantic-texture dichotomy, data scarcity, low computational efficiency, and error propagation) through its innovative AFM Block, Simple Gate, and lightweight multi-head attention mechanism. Ultimately, it achieves significant benefits in three aspects: "high-fidelity restoration quality," "industrial-grade computational efficiency," and "strong practical applicability," providing a practical and advanced tool for the digital preservation of cultural heritage.

[0077] Other advantages, objectives and features of the present invention will become apparent in part from the following description, and in part from those skilled in the art through study and practice of the invention. Attached Figure Description

[0078] Figure 1 This is a schematic diagram of the overall architecture of the AFMNet repair model of this invention.

[0079] Figure 2 This is a schematic diagram of the structure of the multi-domain partial convolutional module (AFM Block) without activation function in this invention.

[0080] Figure 3 This is a schematic diagram illustrating the working principle of a single attention head in the lightweight multi-head attention module of this invention.

[0081] Figure 4 This is a visual comparison of the results of this invention with other advanced image restoration methods on the open-source dataset Places2.

[0082] Figure 5This is a visual comparison of the present invention with state-of-the-art generative models (including GAN-based LaMa (Large Mask Inpainting) and diffusion-based Stable Diffusion) on mural images.

[0083] Figure 6 This is a demonstration image showing the repair effect of the present invention on common mural defects (such as smoke damage, cracks, and paint peeling). Detailed Implementation

[0084] The present invention will now be described in further detail so that those skilled in the art can implement it based on the description.

[0085] It should be understood that terms such as “having,” “comprising,” and “including” as used herein do not imply the presence or addition of one or more other elements or combinations thereof.

[0086] It should be noted that, unless otherwise specified, the experimental methods described in the following implementation plan are all conventional methods, and the reagents and materials described are all commercially available unless otherwise specified.

[0087] I. Technical Overview and Applications

[0088] 1. Technical Objectives

[0089] This technology (AFMNet for Mural Restoration) systematically addresses the aforementioned technical challenges through its unique design and achieves significant performance improvements, all supported by quantitative data.

[0090] (1) Solving the semantic-texture dichotomy: One of the core innovations of AFMNet is its AFM module, which cleverly integrates information from the spatial, frequency, and channel domains. By using parallel spatial paths (for capturing fine local textures) and frequency paths (using forward discrete cosine transform FDCT to capture global structure), AFMNet can simultaneously model these two very different image features, thus resolving the conflict in this regard among existing models.

[0091] (2) Overcoming data scarcity and overfitting: "Lightweight" is the core design concept and key advantage of this invention. By adopting a parameterless "Simple Gate" (SG) mechanism to replace the complex nonlinear activation function and using an efficient convolutional structure, AFMNet reduces the number of parameters by 34.4% compared to the baseline model C3N. The smaller model size fundamentally reduces the risk of overfitting on the limited mural dataset, making the training process more stable and the model's generalization ability stronger.

[0092] (3) Achieving computational efficiency to support practical deployment: The lightweight architecture directly translates into efficiency advantages in practical applications. The model exhibits extremely fast inference speeds on consumer-grade and even mobile hardware (for example, it only takes 0.3113 seconds to process a 256x256 image on an RTX 4060 Ti), making it a viable tool for cultural heritage conservation workers to use portable laptops for on-site restoration work.

[0093] (4) Preventing error propagation: The lightweight multi-head attention mechanism of this invention is specially designed to jointly process feature maps and their corresponding validity masks. This key design explicitly tells the attention module which pixels are reliable original information and which are newly synthesized, thereby effectively preventing the model from borrowing contextual information from unreliable, filled regions and ensuring high fidelity in large-area repair.

[0094] (5) Superior Quantitative Performance: The performance advantages of this invention are not merely theoretical. In tests on mural datasets, AFMNet outperforms established benchmark methods such as PCUnet (Locally Convolutional U-Shaped Network), C3N, Deepfillv2, and RFR (Recurrent Feature Reasoning for Image Inpainting) in all damage rate (HIAR) ranges. For example, on murals with low damage rates (1-10% missing area), AFMNet achieved a peak signal-to-noise ratio (PSNR) of 34.42 dB.

[0095] 2. Application Areas

[0096] With its core capabilities of high-efficiency and high-fidelity image restoration, this technology not only has core value in the field of cultural heritage protection, but its application potential also extends to multiple related and interdisciplinary fields.

[0097] Cultural Heritage and Archaeological Science (Core Areas)

[0098] (1) Virtual restoration of ancient artworks: The most direct application of this technology is to carry out digital virtual restoration of damaged visual cultural heritage, including but not limited to ancient murals, frescoes, woodblock prints, and old, damaged photographic negatives or historical manuscripts.

[0099] (2) Academic research and analysis: The restored high-fidelity images provide valuable analytical materials for art historians and archaeologists, supporting their in-depth research on historical and artistic elements.

[0100] (3) Digital archiving and protection: As a non-invasive restoration method, this technology is a key tool for achieving permanent digital archiving and protection of precious cultural heritage.

[0101] Media, Entertainment and Design

[0102] (1) Digital exhibitions and museums: can be used to create immersive digital exhibitions and virtual museums, allowing the public to experience and engage with cultural heritage in entirely new ways.

[0103] (2) Film and game production: In the post-production of movies, TV series and video games, it can be used to seamlessly remove unwanted objects (such as wires and stagehands) from the screen, or to generate and repair scene textures to enhance visual effects.

[0104] (3) Creative industries and graphic design: It can be used as a professional tool to repair and restore old and damaged photos for commercial or personal use, and improve image quality.

[0105] Potential applications in other professional and technical fields

[0106] (1) Medical imaging: It has the potential to fill in data gaps or damage in medical scans (such as MRI, CT) images caused by patient movement or sensor artifacts, thereby potentially improving diagnostic accuracy.

[0107] (2) Forensic analysis: It can be used to restore key details in obscured or damaged photographic and video evidence, providing technical support for case investigation.

[0108] (3) Geospatial Science: It can be used to fill the data gaps in satellite and aerial images caused by cloud cover or sensor failure, and improve the integrity and availability of geographic information data.

[0109] II. Detailed Technical Introduction

[0110] 1. Technical Definition

[0111] Our goal is to repair an image masked by an unknown code. Masked color image The mask is a binary image with pixel values ​​of 0 or 1, where 1 represents a complete area pixel and 0 represents a damaged area pixel. The damaged image is denoted as... A four-channel tensor was obtained by superimposing the mask onto the damaged image. The proposed model AFMNet for restoring damaged mural images in this paper enables tensors to... It can be transmitted over the network and the feature map and mask in the tensor are updated synchronously to reconstruct the damaged image, and finally obtain the repaired three-channel color image.

[0112] Our AFMNet mainly consists of three core components:

[0113] 1) Multi-domain partial convolutional module (AFM Block) without activation function

[0114] In mural image completion tasks, challenging scenarios involving large-area defects are frequently encountered. Maintaining image clarity and semantic integrity requires incorporating global information. Therefore, a good image restoration architecture should expand the receptive field as early as possible to mitigate the local correlation problem of convolution. Conventional fully convolutional vision models, such as ResNet, may face slow receptive field growth in the early stages of the network due to their small kernel size. Consequently, many layers in the network may suffer from a lack of global information, potentially wasting significant computational resources on global information completion. Furthermore, in scenarios with wide mask coverage, the convolutional kernel may be entirely within the mask region, limiting its ability to observe only the missing areas and hindering the utilization of effective information. While fast Fourier convolution can introduce global frequency domain information, it also introduces higher spatial complexity, increasing equipment requirements and limiting its application scenarios. To efficiently handle challenging image restoration tasks while reducing computational resources, this paper proposes an activation-free multi-domain partial convolution block (AFM Block) as the basic convolutional block of AFMNet. This block allows feature maps and masks to be updated synchronously, and effectively captures global contextual information and texture details in mural images by introducing frequency domain and channel domain information. It also features a lightweight design, using Simple Gate instead of the nonlinear activation function in traditional image restoration networks, which significantly reduces the intra-block complexity of the model.

[0115] 2) Simple Gate

[0116] Image inpainting is actually a low-level visual task, and therefore does not require complex nonlinear activation mechanisms. Therefore, we introduce a novel mechanism, Simple Gate, to replace the activation function in traditional neural networks, achieving further lightweighting.

[0117] 3) Lightweight Multi-head Attention Block

[0118] For mural images, especially those depicting portraits, birds, and animals, the content is rich and repetitive, requiring a high level of global semantic information. However, existing neural network mechanisms struggle to meet these application needs. The Transformer has been widely used in image restoration tasks due to its ability to capture global information. However, the Transformer's self-attention computation at the global scale leads to quadratic complexity related to the number of tokens, making it insufficient for representing high-resolution images.

[0119] To alleviate this problem, this paper proposes a lightweight multi-head attention module that reduces its intra-block complexity while retaining the ability to capture long-distance dependencies. As a neck between the encoder and decoder, it allows the mask and feature map of the last layer of the encoder to enter the decoder after attention calculation, thereby effectively utilizing the contextual information in different feature subspaces.

[0120] 2. Technical Architecture / Process

[0121] The system adopts a U-Net-style architecture and consists of the following three main parts. For details, please refer to [link / reference]. Figure 1 .

[0122] Encoder: A downsampling layer consisting of a series of novel AFM modules, responsible for extracting hierarchical features from the input.

[0123] Neck / Bottleneck: A lightweight multi-head attention block located at the lowest resolution layer of the network, used to model long-range dependencies and global context.

[0124] Decoder: Consists of a series of upsampling layers using a modified AFM module and nearest neighbor interpolation, used to finely reconstruct image details.

[0125] Skip connections: Direct connections are established between corresponding levels of the encoder and decoder to pass feature maps and masks, thereby preserving low-level spatial information and assisting in high-fidelity reconstruction.

[0126] 3. Key Technologies and Innovations

[0127] The main technological innovations of this technology are as follows:

[0128] Key Innovation 1: Multi-domain Partial Convolutional Block without Activation Function (AFM Block)

[0129] The AFM module is the core building block of the encoder. Its design goal is to obtain the global receptive field early in the network's development, while avoiding the high computational cost of methods like FFC. The algorithm's main mechanisms include dual-domain convolution and feature fusion.

[0130] Dual-domain convolution: This module has two parallel paths: a spatial domain path using standard convolution to capture local texture; and a frequency domain path using forward discrete cosine transform (FDCT) to capture global structural information.

[0131] Feature fusion: The outputs of the spatial and frequency domain paths are concatenated and fused. The result is then renormalized according to a validity mask to prevent bias, a technique inherited from partial convolution.

[0132] Key Innovation 2: Simple Gate

[0133] Traditional nonlinear activation functions (such as ReLU and GELU) increase the computational complexity and number of parameters in the network. Furthermore, image restoration focuses on low-level image features and does not require complex nonlinear transformations. Therefore, this technique replaces them with a simple, parameter-free gating function. Input feature map It is split in two along the channel dimension, and the output is the element-wise product of the two halves.

[0134] The core idea is that multiplication itself introduces sufficient nonlinearity into the task, thus eliminating the need for specialized, complex activation functions. This is a fundamental simplification of the concept of gated linear units (GLUs). This choice significantly reduces the model's complexity and number of parameters, directly contributing to its lightweight nature, and as demonstrated in the ablation experiments below, improving performance. The specific mathematical principles behind this will be explained in detail below.

[0135] Key Innovation 3: Lightweight Multi-Head Attention Mechanism

[0136] To restore the complex, non-repetitive content in a mural, the model must understand long-range dependencies (e.g., connecting parts of a face on either side of a large crack). Standard convolutions cannot do this. While the Transformer excels at this, its quadratic computational complexity is prohibitive.

[0137] A lightweight multi-head attention mechanism, as a key innovative component, enables the effective mask to be processed together with the feature map. This explicitly tells the attention mechanism which pixels are original and which are synthesized, thus preventing it from obtaining contextual information from unreliable, already padded regions.

[0138] In the standard Transformer, the attention score is scaled by a fixed factor (the square root of the key dimension). This is a heuristic and may not be optimal for all tasks or datasets. This invention uses a learnable parameter... This replaces the fixed factor. This parameter is optimized during training, allowing the model to dynamically adjust the "temperature" of the Softmax distribution of the attention map. This makes the attention mechanism more flexible and adaptive. The network can learn the optimal scaling ratio for different feature subspaces and data characteristics, thus achieving better performance and adaptability compared to traditional fixed-scaling attention mechanisms.

[0139] Training and optimization framework

[0140] The network is trained using a composite loss function, which is a weighted sum of five different components, each addressing a different aspect of the repair quality.

[0141] Pixel-level reconstruction loss ( and The l1 loss is applied to the missing region and the known region, respectively. The hole loss has a higher weight to prioritize accurate reconstruction, while the effective region loss ensures that the original parts of the image are not altered.

[0142] Perceived loss ( ): The l1 loss is computed in the feature space of the pre-trained VGG-19 network. This encourages the output to be semantically similar to the real image, thus preserving the global structure and preventing the result from becoming blurred.

[0143] Style loss ( Gram matrices measure the difference between the output and the feature maps of the real image. This enforces stylistic and textural consistency, which is crucial for preserving the artistic quality of the mural.

[0144] Total variation loss ( : A regularization term that penalizes drastic differences between adjacent pixels, thereby promoting spatial smoothness and suppressing visual artifacts.

[0145] The final total loss function is defined as:

[0146]

[0147] Among them, weight It is set based on experience to balance these competing optimization goals.

[0148] 4. Mathematical Model / Formula / Algorithm

[0149] 1) Multi-domain Local Convolutional Block (AFM Block) without activation function

[0150] The specific implementation of a multi-domain local convolution module without activation functions can be represented by the following three formulas:

[0151] First, input feature maps With input mask Perform element-wise multiplication to obtain the masked feature map tensor. This tensor is propagated within the module in the following manner:

[0152]

[0153] This module has two parallel paths: a spatial domain path using standard convolutions to capture local textures. One is the transpose of the convolution kernel for that path; the other is a frequency domain path using forward discrete cosine transform (FDCT) to capture global structural information. (This represents the transpose of the convolution kernel for that path). and These represent the forward discrete cosine transform and the inverse discrete cosine transform, respectively. This method can effectively capture the global information of the input feature map, ensuring the global consistency of the mural restoration results. Its output is scaled by a scaling factor. Perform renormalization. Calculate the sum of all elements in the matrix, where M is a matrix of all ones of the same size as the mask. This operation is performed only if there is at least one valid pixel in the receptive field, as conditionally... As shown.

[0154] Then the feature map after dual-domain convolution is processed. Simplified Channel Attention (SCA) and Simple Gate are used to introduce nonlinear transformations.

[0155]

[0156] Where SCA represents an adaptive average pooling and a The depthwise separable convolutions are combined. Finally, a pointwise convolution is used to process the transformed feature map to obtain the output feature map. :

[0157]

[0158] 2) Simple Gate

[0159] Simple gating consists of two parts: the first part is the tensor along the channel dimension. Split as follows:

[0160]

[0161] Then, for tensors Element-wise multiplication yields the tensor after nonlinear transformation. :

[0162]

[0163] This process can actually be viewed as a simplification of the following formula:

[0164]

[0165] in Indicates input features, Representing different convolution kernels, this is a second-order polynomial nonlinear transformation, so even if the convolution operation is linear, this formula still introduces implicit nonlinearity.

[0166] 3) Lightweight multi-head attention

[0167] Lightweight long-only attention can be expressed by the following formula:

[0168]

[0169] In the formula , , They represent the first The Query, Key, and Value matrix in the subspace where each attention head resides. The output projection matrix is ​​used to concatenate the outputs of multiple attention heads and map them back to the original dimension. We introduce a learnable parameter. To replace the fixed scaling factor in traditional multi-head attention mechanisms This allows the model to be normalized before softmax normalization. By dynamically adjusting the attention distribution across different feature subspaces, the model's flexibility in various repair tasks can be improved.

[0170] The working principle of each attention point is as follows: Figure 3 As shown, we use point convolution (Conv 1) 1) Initial feature extraction is performed using depthwise separable convolution (DConv) and depthwise convolution (DConv). The convolutional feature map is then divided into three groups of uniformly shaped tensors along the channel dimension using chunks. , , These represent Query, Key, and Value, respectively. Through arrangement... , , from Transform to the feature subspace of different attention heads, i.e. , , ,in This represents the number of attention heads. Within the feature subspace of each attention head, [the following is defined]: and Element-wise multiplication yields the attention map, and the attention map is then multiplied by... Element-wise multiplication is used to adjust the model's focus on different spatial locations, and finally, rearrangement is used to transform it back. It also outputs the attention-weighted feature map.

[0171] 5. Sample code / pseudocode

[0172] 1. Pseudocode example of core functionality: Multi-domain local convolution module without activation function (AFM Block)

[0173] Pseudocode example:

[0174] Python

[0175] classActivationFreeMultiDomainBlock(input_features, input_mask):

[0176] / / Step 1: Dual-Domain Feature Extraction & Fusion

[0177] / / Calculation is performed only if there are valid pixels within the convolution kernel.

[0178] If sum(input_mask)>0 then

[0179] / / 1.1 Apply a mask to the input features to block out invalid regions

[0180] masked_input = apply_mask(input_features, input_mask)

[0181] / / 1.2 Parallel Processing: Spatial Domain Path and Frequency Domain Path

[0182] / / Spatial domain path: Capture local texture information

[0183] spatial_path_out = Spatial_Convolution(masked_input)

[0184] / / Frequency domain path: Capture global structural information

[0185] frequency_domain_features = Forward_Discrete_Cosine_Transform(masked_input)

[0186] processed_frequency_features = Frequency_Domain_Convolution(frequency_domain_features)

[0187] frequency_path_out = Inverse_Discrete_Cosine_Transform(processed_frequency_features)

[0188] / / 1.3 Feature Fusion

[0189] / / Concatenate the feature maps from the spatial and frequency domains along the channel dimension.

[0190] fused_features = concatenate(spatial_path_out, frequency_path_out)

[0191] / / Use fusion convolution to further integrate information

[0192] merged_features = Fusion_Convolution(fused_features)

[0193] / / 1.4 Feature Re-normalization

[0194] / / Scale the result based on the number of valid pixels in the mask to eliminate bias.

[0195] / / This is the core idea of ​​partial convolution.

[0196] renormalization_factor = calculate_renormalization_factor(input_mask)

[0197] normalized_features = renormalize(merged_features, renormalization_factor)

[0198] else

[0199] / / If the input is completely invalid, output a zero tensor.

[0200] normalized_features = create_zero_tensor_like(input_features)

[0201] end if

[0202] / / Step 2: Feature Refinement

[0203] / / 2.1 Simple Channel Attention

[0204] Adaptively enhance important feature channels and suppress minor channels.

[0205] attention_features = Simple_Channel_Attention_Module(normalized_features)

[0206] / / 2.2 Simple Gate Mechanism

[0207] / / A parameter-free nonlinear transformation to replace traditional activation functions

[0208] gated_features = Simple_Gate_Module(attention_features)

[0209] / / Step 3: Final Output

[0210] / / 3.1 Use 1x1 convolution (point convolution) for final feature integration

[0211] output_features = Pointwise_Convolution(gated_features)

[0212] / / Step 4: Mask Update

[0213] / / 4.1 Perform a convolution operation on the input mask to match its size with the output feature map, preparing it for the next layer.

[0214] output_mask = update_mask_with_convolution(input_mask)

[0215] / / Returns the processed feature map and the updated mask

[0216] RETURNoutput_features, output_mask

[0217] Auxiliary module description:

[0218] 1. Simple_Channel_Attention_Module(features):

[0219] (1) Calculate the descriptor for each channel using global pooling;

[0220] (2) Generate a set of channel weights based on the descriptors;

[0221] (3) Apply weights to the input feature map and rescale it.

[0222] 2. Simple_Gate_Module(features):

[0223] (1) Divide the input feature map into two parts (part_A, part_B) along the channel dimension on an average basis.

[0224] (2) Returns the element-wise product between the two parts (part_A * part_B).

[0225] 2. Pseudocode example of core functionality: Lightweight multi-head attention

[0226] classLightweightMultiHeadAttention(input_features, input_mask):

[0227] / / Step 1: Efficient Q, K, V Generation

[0228] / / Replace standard convolution with depthwise separable convolution.

[0229] / / This significantly reduces computational complexity and the number of parameters.

[0230] query_tensor = Depthwise_Separable_Conv(input_features)

[0231] key_tensor = Depthwise_Separable_Conv(input_features)

[0232] value_tensor = Depthwise_Separable_Conv(input_features)

[0233] / / Step 2: Multi-Head Splitting & Rearrangement

[0234] / / Split the Q, K, V tensors into multiple "heads" along the channel dimension.

[0235] / / Enables the model to learn multiple dependencies in parallel from different feature subspaces.

[0236] num_heads = GET_CONFIG('number_of_attention_heads')

[0237] multi_head_query = Rearrange_For_MultiHead(query_tensor, num_heads)

[0238] multi_head_key = Rearrange_For_MultiHead(key_tensor, num_heads)

[0239] multi_head_value = Rearrange_For_MultiHead(value_tensor, num_heads)

[0240] / / Step 3: Calculate the attention score

[0241] / / 3.1 Calculate the raw attention score

[0242] transposed_key = Transpose(multi_head_key)

[0243] raw_scores = Matrix_Multiply(multi_head_query, transposed_key)

[0244] / / 3.2 (Key Innovation 1) Applying Learnable Scaling Parameters

[0245] / / Use a learnable parameter delta that is optimized during training instead of a fixed scaling factor.

[0246] / / This allows for dynamic adjustment of the sharpness of the attention distribution, enhancing the model's adaptability.

[0247] learnable_scale_param = GET_TRAINABLE_PARAMETER('delta')

[0248] scaled_scores = raw_scores / learnable_scale_param

[0249] / / Step 4: (Key Innovation Point 2) Joint Mask Processing

[0250] / / Integrate the validity mask (input_mask) into the attention score.

[0251] / / Explicitly prevent the attention mechanism from "borrowing" information from invalid or filled pixel regions.

[0252] / / This prevents the spread of errors.

[0253] masked_scores = Apply_Attention_Mask(scaled_scores, input_mask)

[0254] / / Step 5: Generate Attention Map & WeightFeatures

[0255] / / 5.1 Apply the Softmax function to normalize the scores and generate the final attention map.

[0256] attention_map = Softmax(masked_scores)

[0257] / / 5.2 Apply attention maps to the Value tensor to weight the features.

[0258] weighted_value = Matrix_Multiply(attention_map, multi_head_value)

[0259] / / Step 6: Multi-Head Merging & Final Output

[0260] / / 6.1 Rearrange and merge the weighted results of all heads.

[0261] concatenated_heads = Rearrange_From_MultiHead(weighted_value)

[0262] / / 6.2 The information is fused through a final linear projection (such as a 1x1 convolution) to generate the final output.

[0263] output_features = Pointwise_Convolution(concatenated_heads)

[0264] / / Return the feature map after attention mechanism processing

[0265] RETURN output_feature

[0266] Summary of key innovations in the pseudocode:

[0267] 1. Efficient Q, K, V generation: Using depthwise separable convolutions significantly reduces computational costs;

[0268] 2. Learnable scaling parameter (delta): Replaces the fixed scaling factor in the traditional Transformer, making the attention mechanism more flexible and adaptive;

[0269] 3. Mask-feature joint processing: Explicitly integrating mask information into the attention calculation process is a key design to prevent the model from extracting context from unreliable regions and to ensure repair quality.

[0270] 6. Key Technical Parameters and Performance Indicators

[0271] This paper uses quantitative metrics—PSNR, SSIM, LPIPS, and MSE—as well as visual qualitative assessments to measure the effectiveness of model repair.

[0272] The quantitative indicators are compared with other state-of-the-art methods in this field under different degrees of damage (HIAR), as shown in the table below. The best results are highlighted in bold:

[0273]

[0274] Its visual indicators, such as Figure 3 Our method also achieved better restoration results in terms of visual effects.

[0275] It also achieved good repair performance on the open-source dataset Places2, such as Figure 4 As shown.

[0276] For state-of-the-art methods (Stable Diffusion, the best method in diffusion models, and LaMa, the state-of-the-art method in GANs), quantitative metrics are difficult to measure their merits; their visual metrics, such as... Figure 5 As shown.

[0277] The repair effect is as follows on various common defects (smoke damage, cracks, paint peeling) Figure 6 As shown.

[0278] III. Implementation and Operation Guidelines

[0279] 1. Prerequisites

[0280] 1. Hardware Requirements

[0281] (1) GPU: NVIDIA GPUs are strongly recommended for hardware acceleration. The graphics cards used in the experiment included:

[0282] a) Desktop: NVIDIA GeForce RTX 4060 Ti (16GB VRAM)

[0283] b) Mobile / Laptop Class: NVIDIA GeForce RTX 3060 (6GB VRAM)

[0284] (1) CPU: Intel Core i7-12700KF or equivalent or higher performance processor.

[0285] (2) Memory (RAM): at least 32GB.

[0286] 2. Software Dependencies

[0287] (1) Operating system: Linux or Windows (CUDA supported).

[0288] (2) Programming environment: Python 3.9.

[0289] (3) Core Framework:

[0290] a) torch: 2.2.2 (or higher, must be compatible with CUDA 11.8)

[0291] b) torchvision: 0.17.2 (or later, must be compatible with CUDA 11.8)

[0292] (4) Professional Database:

[0293] a) torch-dct: 0.1.6 (used to implement discrete cosine transform)

[0294] (5) Auxiliary Library:

[0295] a) scikit-image: 0.24.0

[0296] b) Pillow: 11.0.0

[0297] c) scipy: 1.13.1

[0298] 2. Installation and Configuration

[0299] Installation steps

[0300] 1. Environment Setup: It is recommended to use a virtual environment (such as conda or venv) to isolate project dependencies.

[0301] 2. Install PyTorch: Depending on your CUDA version, install the matching torch and torchvision versions from the official PyTorch website.

[0302] 3. Install dependencies: Use pip to install all libraries listed in the "Software Dependencies" section. For example:

[0303] pip install torch-dct==0.1.6 scikit-image==0.24.0 pillow==11.0.0scipy==1.13.1

[0304] 4. Obtain the model code: Deploy the AFMNet model code and related scripts to your working directory.

[0305] Before starting training or inference, the following key parameters need to be configured:

[0306] 1. Optimizer: The Adam optimizer is used.

[0307] 2. Learning rate:

[0308] (1) When pre-training on a large general dataset (such as Places2), the initial learning rate is 0.0001, which is then fine-tuned to 0.00005.

[0309] (2) When training on a professional dataset (such as mural data), the initial learning rate is 0.0001, and the fine-tuning stage is the same as above.

[0310] 3. Weight initialization: The Xavier and Kaiming initialization schemes are adopted.

[0311] 4. Loss Function Weights (Hyperparameters): This is a core configuration for model training and directly affects the repair effect. You can set it based on our experience: =1.00, =6.00, =0.05, =120.00, =0.10.

[0312] 3. Operating Procedures

[0313] Step 1: Data Preparation

[0314] 1. Image Data: Prepare your raw image dataset. All images should be of uniform size, such as 256x256 or 512x512 pixels.

[0315] 2. Mask Generation: Prepare binary masks for training and testing. Research shows that using masks that simulate real-world damage patterns (such as those generated by random walk algorithms) performs better than simple rectangular or brush masks.

[0316] 3. Input format: Stack the damaged image and its corresponding mask into a four-channel tensor as the input to the model.

[0317] Step 2: Model Training (Recommended Process) Since specialized datasets in the field of cultural heritage are typically small, directly training a large model can easily lead to overfitting. Therefore, it is strongly recommended to use a transfer learning strategy to achieve optimal performance.

[0318] 1. Pre-training: Perform thorough initial training on a large-scale, diverse public dataset (such as Places2).

[0319] 2. Transfer and Fine-tuning: After loading the pre-trained model weights, perform a second stage of training (fine-tuning) on ​​your own specialized dataset (such as the mural dataset). During this stage, some parameters of the encoder can be frozen to preserve its generalized feature extraction capabilities.

[0320] 3. Monitoring: During training, closely monitor the changes in the values ​​of each loss function to ensure model convergence.

[0321] Step 3: Reasoning and Image Restoration

[0322] 1. Load Model: Load the weights of the pre-trained AFMNet model.

[0323] 2. Perform repair: Process the image to be repaired and the mask of its damaged area into a specified four-channel input format and input it into the model.

[0324] 3. Obtain the result: The model will output a fully repaired three-channel RGB image.

[0325] 4. Performance Expectations: On the RTX 4060 Ti, the inference time for processing a 256x256 image is approximately 0.31 seconds, and for a 512x512 image it is approximately 0.41 seconds. It can maintain high efficiency even on laptops (RTX 3060) and meet the needs of field work.

[0326] Step 4: Troubleshooting

[0327] Unsatisfactory results: If the repair results are blurry or have artifacts, please check if the loss function weights are configured correctly and confirm whether the recommended training process of "pre-training + fine-tuning" has been followed.

[0328] Specific scenario failure: The model's performance may degrade when processing complex texture loss caused by smoke, resulting in slightly blurry repair results. This is a known limitation of the current model.

[0329] 4. Precautions

[0330] (1) Data Management and Privacy: Cultural heritage data (such as mural images) may be subject to protection policies. When processing, using, and sharing this data, please ensure you comply with relevant laws, regulations, and data sharing agreements. It is recommended to manage and back up sensitive data locally.

[0331] (2) Importance of transfer learning: For small-scale professional datasets, skipping the pre-training step on large public datasets may seriously affect the final performance and generalization ability of the model.

[0332] (3) Evaluation methods: In addition to using quantitative indicators such as PSNR and SSIM for evaluation, qualitative evaluation of visual effects is crucial for cultural heritage restoration tasks. The semantic coherence of the restoration results, the fidelity of the artistic style, and whether visual artifacts are produced all need to be judged by the human eye.

[0333] (4) Model applicability: Although the model is designed for mural restoration, its core technology can also be applied to image restoration tasks in other fields, such as old photo restoration, medical image filling or removal of redundant objects in images.

[0334] IV. Practical Results

[0335] 1) The application of AFMNet technology is expected to bring significant efficiency improvements, cost reductions and quality improvements in the digital preservation of cultural heritage and related fields.

[0336] Improved efficiency and reduced costs: AFMNet is a lightweight network with 34.4% fewer parameters than the baseline model, significantly reducing computational complexity. The model is optimized to achieve fast image processing even on consumer-grade and mobile hardware (such as laptops) (e.g., processing a 512x512 image in just 0.41 seconds on an RTX 4060 Ti), making it ideal for on-site restoration work and real-time applications. Due to its lightweight design, the technology can run efficiently without relying on expensive high-performance computing clusters, thereby reducing the hardware costs for cultural heritage institutions, research units, and small and medium-sized creative studios to deploy the technology.

[0337] 2) Improved Restoration Quality and User Experience: The restoration results are visually coherent and natural, producing fewer artifacts and better preserving the fine textures, brushstrokes, and original artistic style of murals and other artworks. Through an advanced multi-head attention mechanism, the model can understand and reconstruct large-area, non-repetitive, complex content (such as human faces or specific scenes), ensuring the logical and semantic rationality of the restored content. It provides high-definition, high-fidelity digital restoration references for cultural relic conservation experts and art historians, assisting them in academic analysis and physical restoration decisions. Its non-invasive nature also provides high-quality materials for digital archiving and virtual exhibitions.

[0338] 3) Expanding application value: High-quality restoration results can be directly applied to creative fields such as digital exhibitions, film and television production, and game development, providing technical support for the dissemination and innovative development of traditional culture and creating new economic value.

[0339] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. It can be applied to various fields suitable for the embodiments of the present invention. Other modifications can be readily implemented by those skilled in the art. Therefore, without departing from the general concept defined by the claims and their equivalents, the embodiments of the present invention are not limited to the specific details.

Claims

1. A lightweight virtual restoration method for murals based on an activation-free generative model, characterized in that, The process includes the following steps: receiving input data, which includes a damaged image and a corresponding binary mask image, wherein the binary mask image is used to identify the complete region and the damaged region in the image; The damaged image is stacked with the binary mask image to form a multi-channel input tensor; The multi-channel input tensor is input into the trained AFMNet inpainting model to generate the inpainted complete image; The AFMNet repair model employs an encoder-bottleneck-decoder architecture, and: The encoder includes at least one multi-domain partial convolution module without activation function, used to simultaneously extract and fuse local texture information and global structural information based on frequency domain transformation of the input feature map, and to simultaneously update the input mask; The bottleneck includes a lightweight multi-head attention module for modeling the lowest-resolution feature map output by the encoder and its corresponding mask to capture long-range dependencies. The decoder includes at least one upsampling module for upsampling the feature map after the bottleneck processing and combining it with the feature map passed from the encoder via skip connections to reconstruct image details.

2. The lightweight virtual restoration method for murals based on an activation-free generative model as described in claim 1, characterized in that, The execution steps of the multi-domain partial convolution module without activation function include: Input feature map With input mask Multiply to obtain the masked feature map tensor. ; The masked feature map tensor is input into a parallel spatial domain convolution path and a frequency domain convolution path, respectively. The spatial domain convolution path extracts local texture features through standard convolution operations; The frequency domain convolution path extracts global structural features by performing forward discrete cosine transform, frequency domain convolution, and inverse discrete cosine transform. Based on the number of valid pixels in the input mask, the fused feature map is renormalized. When there are valid pixels in the receptive field, the output feature map is scaled according to the ratio of the number of valid pixels to the total number of pixels. The masked feature map tensor is propagated in the following manner: The transpose of the convolution kernel representing the convolution path in the spatial domain. The transpose matrix representing the convolution path kernel in the frequency domain. and These represent the forward discrete cosine transform and the inverse discrete cosine transform, respectively. , Calculate the sum of all elements in the matrix, sum(1) is the sum of all elements in the matrix of all 1s, and b is the bias term; The outputs of the spatial domain convolution path and the frequency domain convolution path are concatenated and fused. The renormalized feature map The features are sequentially input into a simplified channel attention (SCA) module and a simple gate module for feature optimization. Where SCA represents an adaptive average pooling and a 3 A combination of depthwise separable convolutions of size 3 is used, and finally a pointwise convolution is applied to process the transformed feature map to obtain the output feature map: Point convolution is used to integrate the optimized feature maps to obtain the output feature map; The input mask is convolutionally downsampled to obtain the updated output mask.

3. The lightweight virtual restoration method for murals based on an inactivation-free generative model as described in claim 1, characterized in that, The execution steps of the lightweight multi-head attention module include: The input feature map is processed using depthwise separable convolutions to significantly reduce computational complexity and the number of parameters, thereby generating query tensor Q, key tensor K, and value tensor V. The query tensor, key tensor, and value tensor are split into multiple attention heads along the channel dimension; For each attention head, calculate the product of its query tensor and the transpose of its key tensor to obtain the original attention score; The original attention score is scaled using a learnable scaling parameter δ, enabling the model to dynamically adjust the sharpness of the attention distribution. The input mask is integrated into the scaled attention score, and the score corresponding to the invalid position is set to negative infinity to suppress the model's attention to the damaged area. Apply the Softmax function to the attention scores after masking to generate an attention map; Multiply the attention map by the value tensor of the corresponding attention head to obtain the weighted feature output; The weighted feature outputs of all attention heads are concatenated and fused using a single-point convolution to obtain the final output of the lightweight multi-head attention module, as expressed by the following formula: In the formula , , They represent the first The query tensor matrix, key tensor matrix, and value tensor matrix in the subspace where each attention head resides. The output projection matrix is ​​used to concatenate the outputs of multiple attention heads and map them back to the original dimensions. This indicates the number of attention heads.

4. The lightweight virtual restoration method for murals based on an inactivation-free generative model as described in claim 2, characterized in that, The simple gating module achieves nonlinear transformation in the following way: The input feature map, after being processed by the simplified channel attention module, is divided into two parts along the channel dimension. The tensors of the two parts are respectively... , Calculating the element-wise product of these two parts yields the tensor after the nonlinear transformation. : This serves as the output of the simple gating module.

5. The lightweight virtual restoration method for murals based on an inactivation-free generative model as described in claim 1, characterized in that, The trained AFMNet repair model was obtained through the following method: Obtain a training dataset, which includes multiple sets of training samples. Each set of training samples includes a complete image, a simulated damaged image, and a corresponding binary mask. Construct an AFMNet repair model; The AFMNet inpainting model is trained using a composite loss function, which is a weighted sum of pixel-level reconstruction loss, perceptual loss, style loss, and total variation loss. The training employs a transfer learning strategy, including: In the first stage, the AFMNet inpainting model is pre-trained using the training dataset to learn general image features and inpainting priors. The loss function used in the first stage of training is a composite loss function. For pixel-level reconstruction loss applied to known regions, For pixel-level reconstruction loss applied to missing regions, To compute the perceptual loss in the feature space of the pre-trained network, For style loss based on feature map Gram matrix, To promote spatial smoothness, the total variation loss is λ1 to λ5, which are the weights of each loss component. In the second stage, after loading the weights of the pre-trained model obtained in the first stage, the AFMNet restoration model is tuned using a mural professional dataset to adapt the model to the semantic structure, artistic style and typical damage patterns of the murals.

6. The lightweight virtual restoration method for murals based on an inactivation-free generative model as described in claim 1, characterized in that, The multi-channel input tensor is a four-channel tensor, which is formed by stitching together the three-channel damaged image and the single-channel binary mask image along the channel dimension.

7. The lightweight virtual restoration method for murals based on an inactivation-free generative model as described in claim 5, characterized in that, Skip connections are used to pass multi-scale feature maps from the corresponding levels of the encoder and their corresponding updated masks; The decoder's upsampling module uses the nearest neighbor interpolation algorithm for upsampling; During the training of the AFMNet repair model, the weights of the composite loss function are configured as follows: λ1=1.00, λ2=6.00, λ3=0.05, λ4=120.00, λ5=0.

10.

8. A lightweight image inpainting device based on an activation-free generative model, characterized in that, include: A data receiving module is configured to receive input data, which includes a damaged image and a corresponding binary mask image. The data preprocessing module is configured to stack the damaged image with the binary mask image to form a multi-channel input tensor; The image inpainting module includes a trained AFMNet inpainting model configured to process the multi-channel input tensor to generate the inpainted complete image. The AFMNet repair model employs an encoder-bottleneck-decoder architecture, and: The encoder includes at least one multi-domain partial convolution module without activation function, used to simultaneously extract and fuse local texture information and global structural information based on frequency domain transformation of the input feature map, and to simultaneously update the input mask; The bottleneck includes a lightweight multi-head attention module for modeling the lowest-resolution feature map output by the encoder and its corresponding mask to capture long-range dependencies. The decoder includes at least one upsampling module for upsampling the feature map after the bottleneck processing and combining it with the feature map passed from the encoder via skip connections to reconstruct image details.

9. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the method of any one of claims 1-7.

10. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method of any one of claims 1-7.

Images