Image Inpainting Method Based on Wavelet Enhanced Mamba and Multi-Domain Feature Learning

Abstract

This invention relates to the field of image processing technology, specifically to an image inpainting method based on wavelet-enhanced Mamba and multi-domain feature learning. In this invention, Discrete Wavelet Transform (DWT) is used to decompose image features into low-frequency and high-frequency sub-bands, achieving collaborative restoration of structure and texture in different frequency domains. An LFSC-Mamba module is designed for the low-frequency sub-band, combining Mamba's linear sequence modeling capability with a channel-space dual attention mechanism to achieve global structure reconstruction while maintaining linear computational complexity. An HFDP module is designed for the high-frequency sub-band, employing a dynamic filtering kernel generation strategy in the Fourier domain and combining it with a multi-head attention mechanism to achieve fine-grained synthesis of directional and pattern textures, significantly improving the realism and diversity of texture generation.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an image inpainting method based on wavelet-enhanced Mamba and multi-domain feature learning. Background Technology

[0002] Image inpainting is a fundamental problem in the fields of digital image processing and computer vision. It aims to synthesize semantically consistent and visually plausible content to reconstruct missing or damaged areas in an image. Robust inpainting capabilities are crucial for maintaining the structural integrity and perceptual fidelity of visual content.

[0003] Currently, State-Space Models (SSMs) have attracted attention due to their ability to capture long-range dependencies in sequence modeling with linear time complexity. The representative Mamba model combines state-space theory with selective representations of input perception, significantly improving modeling efficiency and reducing computational overhead. Extending this to the vision domain, the VMamba architecture introduces a 2D Selective Scan Module (SS2D) to address the dimensionality mismatch problem from one-dimensional sequences to two-dimensional visual data. The MxT model is the first to jointly utilize Mamba's global receptive field and the nonlocal interaction of Transformer for inpainting learning. The SEM-Net model further combines snake-like bidirectional scanning with learnable location-enhanced Snake-Mamba blocks, enhancing spatial perception while maintaining linear time complexity, improving local texture fidelity and global structural consistency. Overall, Mamba-based methods alleviate the quadratic computation bottleneck of Transformer in large-scale image inpainting.

[0004] However, existing technologies still face challenges in achieving consistent texture continuity, structural consistency, and semantic coherence between the repaired region and the known region. This task focuses more on cross-regional non-local information interaction than on unidirectional sequential dependencies. Therefore, how to advance Mamba beyond one-dimensional sequence representation and effectively facilitate structural and texture reasoning in two-dimensional space is a key problem that urgently needs to be solved in the current intersection of efficient modeling and visual understanding. Summary of the Invention

[0005] To address at least one problem in the prior art, the present invention provides an image inpainting method based on wavelet-enhanced Mamba and multi-domain feature learning.

[0006] The first aspect of this invention provides an image inpainting method based on wavelet-enhanced Mamba and multi-domain feature learning, which includes the following steps: Obtain the image to be repaired; The multi-layer coded features of the image to be repaired are extracted using an encoder; The image is then decoded using a decoder to obtain the repaired image. The encoder and decoder both include an MDFL module. In the MDFL module, the input features are decomposed into a low-frequency sub-band and three high-frequency sub-bands by Discrete Wavelet Transform (DWT). The low-frequency sub-band is processed by the LFCS-Mamba module, the high-frequency sub-band is processed by the HFDP module, and then lossless reconstruction is performed by Inverse Wavelet Transform (IWT). In the LFCS-Mamba module, the low-frequency subband is processed by the Mamba module and the spatial attention module respectively, and the obtained channel enhancement features are multiplied element-wise with the spatial attention map, and then residually connected with the low-frequency subband. In the HFDP module, the high-frequency subband is filtered based on the Fast Fourier Transform, processed by the multi-head attention module, and then the resulting dynamic filter map and attention weights are multiplied element by element.

[0007] In some optional embodiments, in the MDFL module, the input features are first layer-normalized and processed by the DFP unit, and then residually connected with the input features to obtain an intermediate feature map; then the intermediate feature map is layer-normalized and processed by the feedforward neural network, and finally residually connected with the intermediate feature map to obtain the output features of the MDFL module; wherein, in the DFP unit, the input features of the DFP unit (obtained from the input features of the MDFL module after layer normalization) are decomposed into one low-frequency sub-band and three high-frequency sub-bands by the Discrete Wavelet Transform (DWT), the low-frequency sub-band is processed by the LFCS-Mamba module, the high-frequency sub-bands are processed by the HFDP module, and then lossless reconstruction is performed by the Inverse Wavelet Transform (IWT).

[0008] In some alternative embodiments, when extracting multi-layer coded features of the image to be repaired by the encoder and / or when decoding by the decoder, the encoder and / or decoder use multiple stacked MDFL modules per layer.

[0009] In some optional embodiments, processing the low-frequency subband using the Mamba module and the spatial attention module respectively includes: The low-frequency subbands are converted into sequences, layer normalized, processed by the Mamba module, and the sequences are restored through transpose and reshaping operations to obtain channel enhancement features. The low-frequency subband is subjected to global average pooling and global max pooling along the channel dimension. The two single-channel feature maps are then concatenated and activated by convolution and the Sigmoid function to obtain a spatial attention map.

[0010] In some optional embodiments, filtering the high-frequency subband based on the Fast Fourier Transform includes: The high-frequency subband is convolved, batch normalized, and activated by the ReLU function to decompose it into imaginary and real parts, and then transformed to the frequency domain by two-dimensional real fast Fourier transform (2D RFFT). Convolution, batch normalization, and ReLU activation are performed in the frequency domain to recombine the real and imaginary parts, and then the results are mapped back to the spatial domain through a two-dimensional real inverse fast Fourier transform (2D RIFFT). Finally, a dynamic filter map is generated through convolution.

[0011] In some optional embodiments, processing via the multi-head attention module includes: The high-frequency subband and dynamic filter map are divided into multiple attention heads along the channel dimension. Global average pooling is performed on the feature tensor of each attention head to obtain cross-space statistical features. The attention weights are obtained by mapping the cross-spatial statistical features through a shared fully connected layer.

[0012] In some optional embodiments, the method further includes a step of fusing multi-layer coded features and decoder output features through skip connection layers.

[0013] A second aspect of the present invention provides an electronic device comprising a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the image inpainting method based on wavelet-enhanced Mamba and multi-domain feature learning according to the computer program.

[0014] A third aspect of the present invention provides a computer-readable storage medium for storing a computer program; the computer program, when executed by a processor, implements the image inpainting method based on wavelet-enhanced Mamba and multi-domain feature learning.

[0015] The technical solutions of the embodiments of the present invention have the following beneficial effects: (1) The image features are decomposed into low-frequency sub-bands and high-frequency sub-bands by using Discrete Wavelet Transform (DWT) to achieve collaborative restoration of structure and texture in different frequency domains; (2) An LFSC-Mamba module was designed. For the low-frequency subband, this module combines Mamba’s linear sequence modeling capability with the channel-space dual attention mechanism to achieve global structure reconstruction while maintaining linear computational complexity. (3) An HFDP module was designed. For high-frequency subbands, the module adopts a dynamic filtering kernel generation strategy in the Fourier domain and combines it with a multi-head attention mechanism to achieve fine-grained synthesis of directional and pattern textures, which significantly improves the realism and diversity of texture generation. Attached Figure Description

[0016] Figure 1This is a schematic diagram of the overall architecture of the image restoration model WEMamba-MDFL in an embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram of the network structure of the MDFL module in an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of the network structure of the DFP unit in an embodiment of the present invention.

[0019] Figure 4 The images shown are DWT decomposition results of different original images in the embodiments of the present invention, wherein (a) represents different original images and (b) represents DWT decomposition results.

[0020] Figure 5 This is a schematic diagram of the network structure of the LFSC-Mamba module in an embodiment of the present invention.

[0021] Figure 6 This is a schematic diagram of the network structure of the Mamba module in an embodiment of the present invention.

[0022] Figure 7 This is a schematic diagram of the network structure of the HFDP module in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0024] To address the inherent problems of high computational complexity and difficulty in detail reconstruction in the field of image inpainting, this invention provides an image inpainting method based on wavelet-enhanced Mamba and multi-domain feature learning. An image inpainting model is trained and then used for image inpainting. For ease of representation, the image inpainting model in this invention is named WEMamba-MDFL.

[0025] WEMamba-MDFL comprises an encoder, skip connection layers, and a decoder. Specifically, the encoder receives the image to be restored (the damaged image) and extracts encoded features layer by layer through overlapping block embedding and a multi-domain feature learning module (MDFL). Skip connection layers are used to fuse encoded and decoded features from different layers. The decoder restores the image to its original spatial dimensions layer by layer through upsampling and the multi-domain feature learning module, ultimately outputting a restored image (the restored result) with the same output size.

[0026] like Figure 1 As shown, during the training phase of the image inpainting model, the input to WEMamba-MDFL can be of size [size missing]. binary mask With size The truth image First, generate the damaged image. ,in The damaged image Used to simulate the image to be repaired. Then, the damaged image... Its binary mask The layers are stitched together along the channel dimension, and then passed through a 3×3 embedded convolutional layer (denoted as...). Embedding (·) Adjust the channel dimension to C The generated size is initial feature map Next, the initial feature map. Passing through each layer Layer encoding, sequentially obtaining multiple feature maps As shown in the following formula: in, Indicates by Processed by stacked MDFL modules, Indicates the first The number of MDFL modules used during layer encoding (e.g.) ); This indicates downsampling, for example, using a 3×3 convolution with a stride of 2 to achieve downsampling; Indicates the coding layer index. Indicates the total number of coding layers (e.g.) ), ; Indicates the first The input feature map of the layer encoding, where Indicates batch size, , , These represent the number of channels, feature map height, and feature map width, respectively. Indicates the first The output feature map after layer processing. The dimensions of the output feature maps of the first, second, and third layers of the encoder are as follows: , , The encoder has a total of 10 layers. Decoder number layer( =0,1…, 2) After upsampling, compare the current feature map with the encoder's first... The feature maps input to the layer are concatenated along the channel dimension, and then convolved using a 3×3 method as the input to the decoder layer. =0 corresponds to the deepest layer of the decoder. k= This corresponds to the shallowest layer of the decoder. The decoder also uses multiple stacked MDFL modules to progressively reconstruct the complete image. It recovers image resolution and detail information gradually through upsampling operations (which can be implemented using 3×3 convolution and pixel shuffling) combined with skip connections, as shown in the following equation: in, Indicates the first Input feature map for layer decoding, This represents the output characteristics of the bottleneck layer; Indicates an upsampling operation; This indicates a splicing operation. Indicates by Processed by stacked MDFL modules, Indicates the first The number of MDFL modules used during layer decoding. After layer-by-layer upsampling by the decoder, the final output size is obtained through a single 3×3 convolution layer. Repaired images .

[0027] Each MDFL module decomposes the input features into one low-frequency sub-band and three high-frequency sub-bands using Discrete Wavelet Transform (DWT), and processes them using dedicated feature learning modules in different frequency spaces. For the low-frequency sub-bands carrying the main content and structural information, the Low-Frequency Spatial Channel Mamba module (LFCS-Mamba module) is used. The LFCS-Mamba module leverages Mamba's advantages in linear sequence modeling, incorporating channel attention mechanisms and combining them with spatial attention to construct a hybrid attention structure based on Mamba. This structure can achieve joint representation of spatial features while maintaining linear computational complexity, thereby effectively restoring the global structure of the image. For the high-frequency sub-bands containing edge and texture information, the High-Frequency Dynamic Perception module (HFDP module) is used. The HFDP module dynamically generates adaptive convolution kernels in the Fourier domain and introduces multi-head attention modulation to achieve fine synthesis of directional and regular textures, significantly improving the realism and richness of the textures.

[0028] like Figure 2 As shown, the core component of the MDFL module is the dual-channel frequency processing unit (DFP unit), and the calculation flow of the MDFL module is shown in the following formula: in, Presentation layer normalization operation, This indicates that the frequency processing unit is handling the signal. This represents the intermediate feature map obtained through residual connection; This indicates feedforward neural network processing. Represents the output feature map of the MDFL module; symbol This indicates element-wise addition and is used to characterize residual connections.

[0029] like Figure 3 As shown, in the DFP unit, the input feature map is decomposed into four sub-bands using Discrete Wavelet Transform (DWT): one low-frequency sub-band (LL) and three high-frequency sub-bands (HL, LH, HH). The low-frequency sub-band (LL) undergoes global structure modeling via the LFSC-Mamba module (Low-Frequency Channel-Spatial Attention Mamba Block), while the high-frequency sub-bands (HL, LH, HH) receive detail enhancement via the HFDP module. Finally, Inverse Wavelet Transform (IWT) is used to losslessly reconstruct the multi-sub-band information, thereby achieving effective structure and texture restoration.

[0030] DWT divides the image into four frequency sub-bands using orthogonal filters and can achieve lossless reconstruction using IWT, thus preserving the complete information required for multi-domain feature learning, as shown in the following equation: in, Indicates the low-frequency sub-band. , , These represent the high-frequency sub-bands in the horizontal, vertical, and diagonal directions, respectively. This represents the Discrete Wavelet Transform (DWT). To visually demonstrate the characteristics of the DWT, Figure 4 The DWT decomposition results for three different datasets are shown. The low-frequency subband (LL) effectively preserves the macroscopic structure and semantic content of the image, such as the posture of the crane in the mural, the architectural structure, and the mountain outline. In contrast, the three high-frequency subbands (LH, HL, HH) capture direction-sensitive edge and texture details. For example, in... Figure 4 In the second image of (b), horizontal features such as eaves and window frames show significant responses in LH, while vertical structures are more prominent in HL. Even in areas with complex textures (such as cracks in murals), DWT can still achieve effective decomposition. This visualization result shows that DWT can decouple images at the feature level into macroscopic structural features (LL) and microscopic texture features (LH, HL, HH), a characteristic that provides a key basis for differentiated modeling strategies.

[0031] like Figure 5As shown, in the LFSC-Mamba module, the input is the low-frequency subband obtained through Discrete Wavelet Transform (DWT). ,in Indicates batch size, Indicates the number of channels. and These represent the height and width of the feature map, respectively; first, Convert to sequence format As shown in the following formula: in, Indicates the sequence length. This means flattening the feature map into a two-dimensional matrix. This indicates swapping the specified dimensions to obtain the desired dimension. Subsequently, on Perform layer normalization operations to stabilize the training process and input the results into a Mamba module, whose output is... These are channel features enhanced by global context. Then, through transpose and reshaping operations, the feature sequence is restored to a two-dimensional spatial structure, resulting in... As shown in the following formula: in, Indicates channel enhancement features, This indicates that the dimension is changed from Adjusted to ; This represents the process of restoring one-dimensional sequence features to a two-dimensional feature map (satisfying...) ).like Figure 6 As shown, in the Mamba module, the input features after layer normalization are first divided into two branches (the first branch and the second branch) by linear layer projection. The first branch is processed sequentially by 3×3 convolution, SiLU activation, and the two-dimensional selective scan module (SS2D module) to obtain the output features of the first branch. The second branch is multiplied by the output features of the first branch after SiLU activation, and then multiplied by linear layer projection and added element-wise with the input features after layer normalization to obtain the output features of the Mamba module.

[0032] Meanwhile, to enhance focus on key spatial structures, the LFSC-Mamba module introduces a spatial attention branch. In this branch, the low-frequency subband of the input is first... Global average pooling and global max pooling are performed along the channel dimension to obtain two single-channel feature maps. These single-channel feature maps are then concatenated along the channel dimension, followed by 7×7 convolution and sigmoid activation to generate a spatial attention map. As shown in the following formula: in, This indicates that the Sigmoid function is activated. This represents a 7×7 convolution operation. This represents a splicing operation along the channel dimension. This indicates a global average pooling operation. This indicates global max pooling.

[0033] Finally, enhance the channel features Spatial attention map Element-wise multiplication, followed by residual concatenation with the original input (low-frequency subband) The LFCS-Mamba module is then merged to obtain its final output. As shown in the following formula: Here, ⊙ represents element-wise multiplication.

[0034] The LFSC-Mamba module preserves the original information while capturing global structural dependencies using the Mamba module and further enhances key regions through spatial attention. Therefore, the overall computational complexity of the model increases linearly with sequence length, effectively avoiding the inherent quadratic complexity problem of the Transformer.

[0035] like Figure 7 As shown, the High Frequency Dynamic Perception Module (HFDP module) dynamically generates adaptive filter kernels in the Fourier domain and introduces multi-head attention to assign weights to features, enabling the convolution operation to adaptively adjust parameters based on local content. For high-frequency feature maps... ( First, through the filter generator Generate a dynamic filter graph consistent with the input dimension. As shown in the following formula: in, Implemented based on Fast Fourier Transform (FFT). High-frequency feature map. First, the sample undergoes 1×1 convolution, batch normalization, and ReLU activation, then decomposes into imaginary and real parts, and transforms them to the frequency domain using a two-dimensional real fast Fourier transform (2DRFFT). Next, 1×1 convolution, batch normalization, and ReLU activation are performed in the frequency domain; the real and imaginary parts are then reassembled and mapped back to the spatial domain using a two-dimensional real inverse fast Fourier transform (2D RIFFT). Finally, a dynamic filter map is generated using 1×1 convolution. This process enables frequency-domain adaptive filtering, thereby effectively capturing the directional features of high-frequency textures.

[0036] To improve the selectivity of feature representation, a multi-head attention mechanism is further introduced. This involves processing high-frequency feature maps... With dynamic filtering diagram Divided along the channel dimension Size, the dimensions of each head satisfy As shown in the following formula: in, Indicates the number of heads of attention. The channel dimension representing a single attention head; and These represent the tensors after grouping the high-frequency feature map and the dynamic filtering map, respectively, both with dimension 1. Subsequently, the input features within a single attention head are... Perform global average pooling to compute cross-spatial statistical features. The obtained cross-spatial statistical characteristics Attention weights are obtained by sharing the fully connected layer mapping. And perform Softmax normalization along the channel dimension of each attention head; then calculate the resulting attention weights. Dynamic filtering diagram after equal distribution Element-wise multiplication is performed to achieve adaptive weighting across the head; the process is shown in the following equation: in, This indicates a global average pooling operation. To represent a linear transformation, This indicates Softmax normalization; This represents a weighted feature map.

[0037] In order to weight the feature map To restore it to its original size, first perform channel-wise convolution. Then perform pointwise convolution. Finally, the output of the HFDP module is obtained. As shown in the following formula: After processing the three high-frequency subbands through this module, the corresponding enhanced high-frequency features can be obtained. , and Ultimately, this will enhance low-frequency characteristics. The spatial features are reconstructed by fusing the high-frequency features with each feature via inverse wavelet transform (IWT). The definition is as follows: in, This represents the inverse wavelet transform.

[0038] The HFPD module effectively captures directional texture features, suppresses noise amplification, and significantly improves the fidelity of high-frequency detail restoration. By jointly modeling low-frequency global structure and high-frequency local texture, it can achieve image restoration results with coherent structural integrity and realistic, natural textures.

[0039] The model is trained using a generative adversarial strategy, in which a PatchGAN discriminator is employed. ), based on discriminator loss Update the discriminator The parameters are adjusted to enhance the ability to distinguish between real and generated images. Discriminator loss. The definition is shown in the following formula: in, This represents the discriminator loss value. This represents the expectation operation. Represents a real image. This represents the repaired image output by the generator. This represents the output of the PatchGAN discriminator.

[0040] Furthermore, based on multi-task loss Update WEMamba-MDFL parameters, multi-tasking loss The definition is shown in the following formula: in, , , , These represent the weighting hyperparameters used to balance the various losses; Indicates the losses incurred during reconstruction. Indicating resistance to loss, Indicates perceived loss. Style loss is represented by the following formula: in, Represents a real image. This represents a binary mask (1 represents a missing region); Represents generator The output of the repair result (repaired image); This indicates element-wise multiplication; This represents the repaired image output by the generator. Indicates the repaired image Find the expected value of the distribution it follows. This represents the PatchGAN discriminator; This indicates the pre-trained VGG-19 network's... Feature maps extracted from layers; The Gram matrix (obtained by flattening the feature map into a two-dimensional matrix and then calculating the inner product) represents, for example... ; express Norm.

[0041] To verify the effectiveness and generalization ability of the image inpainting model (WEMamba-MDFL) in this embodiment of the invention, experiments were conducted on the CelebA-HQ dataset, the Paris Street View dataset, the Places2 dataset, and the MuralDH dataset. The dataset partitioning strictly followed standard settings, as detailed in Table 1. In the experiments, all hyperparameter tuning was performed on the Places2 dataset.

[0042] Table 1 Dataset Description Places2 dataset: This dataset is a large-scale scene recognition benchmark dataset containing 365 categories, with 5000 images per category. To verify model performance, 20,000 images were selected from four subcategories: mountains, mountain roads, forests, and canyons. Specifically, 4900 images were randomly assigned to each subcategory for training, and 100 images were randomly selected from the remaining 400 images as the test set.

[0043] The CelebA-HQ dataset is a representative dataset of high-quality face images, containing 30,000 face images. 28,000 images were randomly selected as the training set, and the remaining 2,000 images were used as the test set.

[0044] Paris Street View dataset: This dataset contains 15,000 city street view images, of which 14,900 were used for training and the remaining 100 were divided into a test set.

[0045] MuralDH dataset: This dataset is a domain-specific dataset built for the task of restoring Dunhuang mural images, containing over 5000 pre-processed high-quality mural images. This experiment selected 760 images as the training set and 201 images as the independent test set.

[0046] The experiments were conducted using an NVIDIA GeForce RTX 4090 graphics card and the PyTorch deep learning framework. The model employed the AdamW optimizer for joint training of the generator and discriminator. The first-order moment decay coefficient of AdamW was used... β 1 is set to 0.9, the second-order moment attenuation coefficient. β 2 is set to 0.999. Multitasking loss. The weight hyperparameters are set as follows: =15、 =0.01、 =0.2、 =150. In the experiment, the input images were uniformly scaled to a resolution of 256×256 pixels, and the batch size was set to 4. Furthermore, a composite learning rate strategy combining warm-up learning and multi-step decay was adopted: in the first 10 training rounds, the learning rate was linearly increased from 0 to 0.0002; subsequently, through the multi-step learning rate scheduler (MultiStepLR), the current learning rate was decayed to 0.1 times its original value in the 100th and 150th rounds, respectively.

[0047] To validate the performance of WEMamba-MDFL, it was evaluated against six mainstream image inpainting models (CANet, Spa-former, APLRL, LSG-Net, TSBGNet, and SEM-Net) on three benchmark datasets: Places2, CelebA-HQ, and Paris Street View. Furthermore, it was evaluated against five mainstream image inpainting models (APLRL, SEM-Net, LSG-Net, HINT, and TSGDAM) on the domain-specific dataset MuralDH.

[0048] Table 2 shows the comparison results on the Places2 dataset. It can be seen that WEMamba-MDFL outperforms all comparison models across all six mask ratios. In terms of Peak Signal-to-Noise Ratio (PSNR), WEMamba-MDFL achieves an average of 27.81 dB, an improvement of 0.20 dB (approximately 0.72%) over APLRL, demonstrating its significant advantage in high-fidelity detail restoration. Regarding Structural Similarity (SSIM), WEMamba-MDFL achieves a value of 0.870, an improvement of 0.35% over APLRL, further validating its excellent ability to maintain structural consistency. In terms of Mean Absolute Error (MAE), WEMamba-MDFL's average value is 1.98%, a reduction of 5.71% compared to LSG-Net, fully demonstrating its superior performance in pixel-level accuracy.

[0049] Table 2 Comparison results on the Places2 dataset In this chart, ↑ indicates that higher values ​​result in better performance, ↓ indicates that lower values ​​result in better performance, and bolded items represent the optimal results.

[0050] Table 3 presents the comparison results on the Paris Street View dataset. It can be seen that WEMamba-MDFL achieves the best performance across all evaluation metrics, demonstrating its excellent generalization ability. Regarding Peak Signal-to-Noise Ratio (PSNR), WEMamba-MDFL achieves the highest value across six different mask ratios, with an average PSNR of 28.86 dB, indicating its stable advantage in street view image structure restoration. In terms of Structural Similarity (SSIM), WEMamba-MDFL also performs exceptionally well, with an average SSIM of 0.887, significantly outperforming other comparative models. Notably, both WEMamba-MDFL and APLRL achieve the best performance in the 0.1%–10% mask ratio range, indicating that APLRL also possesses high-fidelity restoration capabilities at low mask ratios; however, WEMamba-MDFL maintains excellent structural consistency even at high mask ratios. Regarding Mean Absolute Error (MAE), WEMamba-MDFL's average value is 1.83%, validating its pixel-level accuracy in building outline and fine detail restoration.

[0051] Table 3 Comparison results on the Paris Street View dataset Table 4 presents the quantitative comparison results on the CelebA-HQ dataset, showing that WEMamba-MDFL significantly outperforms all baseline methods. In terms of mean absolute error (MAE), WEMamba-MDFL achieves the lowest error across all six mask ratios, with an average MAE of 1.60%, indicating its significant effectiveness in restoring fine facial details. Regarding structural similarity (SSIM), WEMamba-MDFL also performs excellently, achieving the best results at all mask ratios except for specific intervals, with an average SSIM of 0.933. In terms of peak signal-to-noise ratio (PSNR), WEMamba-MDFL achieves the highest value except for the 50%-60% large missing content ratio. Specifically, TSBGNet achieves the best result (22.22 dB) at the 50%-60% mask ratio, slightly higher than WEMamba-MDFL (22.06 dB). This indicates that while TSBGNet demonstrates strong restoration capabilities in large-scale face image restoration, WEMamba-MDFL achieves a good balance between structural coherence and texture detail preservation, providing a robust solution for high-quality face image restoration.

[0052] Table 4 Comparison results on the CelebA HQ dataset Table 5 presents the comparative results on the MuralDH dataset. It can be seen that WEMamba-MDFL achieves superior performance on most evaluation metrics, demonstrating its strong adaptability. In terms of PSNR, WEMamba-MDFL achieves the best results across all masking ratios, with an average PSNR of 25.66 dB, 0.14 dB higher than SEM-Net. This indicates that WEMamba-MDFL maintains a significant advantage in handling texture details in the MuralDH dataset. Regarding MAE, WEMamba-MDFL consistently maintains the best performance across all masking ratios, with an average MAE of 2.64%, 4.35% lower than HINT, further validating its superiority. In terms of SSIM, WEMamba-MDFL achieves the highest value across masking ratios from 0.1% to 50%, with an average SSIM of 0.778 across the six ratios, significantly outperforming other comparative models. Overall, WEMamba-MDFL demonstrates its excellent effectiveness on the MuralDH dataset, effectively handling the texture and color information of mural images.

[0053] Table 5 Comparison results on the MuralDH dataset To verify the effectiveness of each core module in WEMamba-MDFL, a series of ablation experiments were conducted. Using the complete model as a baseline, four variants were designed by sequentially replacing key modules: (a) without wavelet modules (w / o Wavelet), removing all frequency domain decomposition structures; (b) without spatial attention (w / o SA), removing the spatial attention unit in the low-frequency branch LFCS-Mamba module; (c) without high-frequency dynamic Fourier convolution (w / o HFDP), replacing the dynamic Fourier convolution in the high-frequency branch with standard convolution; and (d) without channel attention (w / o CA), removing the channel attention unit in the low-frequency branch LFCS-Mamba module. All ablation experiments were performed on the Paris StreetView dataset, and the results are summarized in Table 6.

[0054] Table 6 Ablation experiment results on the Paris street view dataset As shown in Table 6, compared to WEMamba-MDFL, the model performance significantly decreased after removing the wavelet module (w / o Wavelet). For example, at mask ratios of 0.1% to 10%, the peak signal-to-noise ratio dropped from 38.20 dB to 37.02 dB, a decrease of 1.18 dB. This indicates that decomposing the image restoration task into the frequency domain using discrete wavelet transform is an important foundation for achieving high-quality restoration results. Replacing the high-frequency dynamic convolution module with standard convolution (w / o HFDP) also significantly reduced model performance. At mask ratios of 20% to 30%, the structural similarity dropped from 0.937 to 0.903. This result confirms that the high-frequency dynamic convolution module can generate adaptive filters in the frequency domain, synthesizing realistic and delicate high-frequency textures, and its effect is significantly better than traditional convolution with limited receptive field.

[0055] Furthermore, in the model variant without spatial attention (w / o SA), the model performance showed a significant decline, with the peak signal-to-noise ratio dropping from 29.24 to 28.90 dB. This indicates that although Mamba excels at capturing long dependencies in the channel dimension, introducing a spatial attention module can more effectively model both global and local information. In the model variant without channel attention (w / o CA), performance also dropped significantly, with structural similarity decreasing from 0.707 to 0.688 at high mask ratios. This phenomenon suggests that combining Mamba-based channel attention with spatial attention can generate more realistic repair results.

[0056] The computational performance of WEMamba-MDFL and various comparative models was evaluated on the CelebA-HQ dataset. The evaluation metrics included the number of parameters, floating-point operations, average inference time, and average training time per round. All experiments were conducted using a single NVIDIA GeForce RTX 4090 graphics card, and the input image resolution was uniformly set to 256×256. The relevant results are summarized in Table 7.

[0057] Table 7 Performance comparison on the CelebA-HQ dataset In terms of model lightweighting, WEMamba-MDFL has only 3.54M parameters, second only to APLRL among all compared models. Regarding training efficiency, WEMamba-MDFL takes only 1852.19 seconds per training round, effectively shortening the training cycle. Therefore, WEMamba-MDFL achieves efficient training and inference with extremely low parameter counts, fully validating the effectiveness of its lightweight design.

[0058] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any form or substance. It should be noted that those skilled in the art can make various improvements and additions without departing from the method of the present invention, and these improvements and additions should also be considered within the scope of protection of the present invention. Any modifications, alterations, and equivalent changes made by those skilled in the art based on the above-disclosed technical content without departing from the spirit and scope of the present invention are equivalent embodiments of the present invention. Furthermore, any modifications, alterations, and evolutions made to the above embodiments based on the essential technology of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. An image inpainting method based on wavelet-enhanced Mamba and multi-domain feature learning, characterized in that, Includes the following steps: Obtain the image to be repaired; The multi-layer coded features of the image to be repaired are extracted using an encoder; The image is then decoded using a decoder to obtain the repaired image. Both the encoder and decoder include an MDFL module. In the MDFL module, the input features are decomposed into one low-frequency sub-band and three high-frequency sub-bands through discrete wavelet transform. The low-frequency sub-band is processed by the LFCS-Mamba module, the high-frequency sub-band is processed by the HFDP module, and then lossless reconstruction is performed by inverse wavelet transform. In the LFCS-Mamba module, the low-frequency subband is processed by the Mamba module and the spatial attention module respectively, and the obtained channel enhancement features are multiplied element-wise with the spatial attention map, and then residually connected with the low-frequency subband. In the HFDP module, the high-frequency subband is filtered based on the Fast Fourier Transform, processed by the multi-head attention module, and then the resulting dynamic filter map and attention weights are multiplied element by element.

2. The method according to claim 1, characterized in that, In the MDFL module, the input features are first layer-normalized and processed by the DFP unit, then residually connected with the input features to obtain an intermediate feature map. The intermediate feature map is then layer-normalized and processed by a feedforward neural network, and finally residually connected with the intermediate feature map to obtain the output features of the MDFL module. Specifically, in the DFP unit, the input features are decomposed into one low-frequency sub-band and three high-frequency sub-bands using discrete wavelet transform. The low-frequency sub-band is processed by the LFCS-Mamba module, and the high-frequency sub-bands are processed by the HFDP module. Finally, lossless reconstruction is performed using inverse wavelet transform.

3. The method according to claim 1, characterized in that, When extracting multi-layer coded features of the image to be repaired by the encoder and / or when decoding by the decoder, the encoder and / or decoder use multiple stacked MDFL modules per layer.

4. The method according to claim 1, characterized in that, The low-frequency subband is processed by the Mamba module and the spatial attention module, respectively, including: The low-frequency subbands are converted into sequences, layer normalized, processed by the Mamba module, and the sequences are restored through transpose and reshaping operations to obtain channel enhancement features. The low-frequency subband is subjected to global average pooling and global max pooling along the channel dimension. The two single-channel feature maps are then concatenated and activated by convolution and the Sigmoid function to obtain a spatial attention map.

5. The method according to claim 1, characterized in that, Filtering the high-frequency subband based on Fast Fourier Transform includes: The high-frequency subband is convolved, batch normalized, and activated by the ReLU function to decompose it into imaginary and real parts, and then transformed to the frequency domain by a two-dimensional real fast Fourier transform. In the frequency domain, convolution, batch normalization, and ReLU activation are performed to recombine the real and imaginary parts, and then the results are mapped back to the spatial domain through a two-dimensional real inverse fast Fourier transform. Finally, a dynamic filter map is generated through convolution.

6. The method according to claim 1, characterized in that, Processing via the multi-head attention module includes: The high-frequency subband and dynamic filter map are divided into multiple attention heads along the channel dimension. Global average pooling is performed on the feature tensor of each attention head to obtain cross-space statistical features. The attention weights are obtained by mapping the cross-spatial statistical features through a shared fully connected layer.

7. The method according to claim 1, characterized in that, It also includes the step of fusing multi-layer coding features and decoder output features through skip connection layers.

8. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute, according to the computer program, the image inpainting method based on wavelet-enhanced Mamba and multi-domain feature learning as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, A computer-readable storage medium is used to store a computer program; the computer program, when executed by a processor, implements the image inpainting method based on wavelet-enhanced Mamba and multi-domain feature learning as described in any one of claims 1-7.

Images