An image inpainting method and system based on superpixel guided state space model

Abstract

The present application relates to the technical field of image inpainting, and particularly relates to an image inpainting method and system based on superpixel guided state space model, which comprises the following steps: obtaining a degraded image, and extracting shallow features of the degraded image through a feature embedding layer; constructing a semantic region-aware M module, which comprises a superpixel Mamba and a feature refinement feedforward network, and the superpixel Mamba is embedded with a superpixel guided selective scanning mechanism and a region-level gating mechanism; constructing a multi-level symmetric encoder-decoder architecture; inputting the shallow feature map into the multi-level symmetric encoder-decoder architecture for multi-scale feature learning; fusing the deep feature map and the shallow feature map to obtain a fused feature map; and performing feature reconstruction on the fused feature map through a reconstruction convolution layer to obtain a restored image. The present application has significant technical progress and practical application value in terms of image inpainting accuracy, computational efficiency, scene adaptability and the like.

Description

Technical Field

[0001] This invention relates to the field of image restoration technology, and in particular to an image restoration method and system based on a superpixel-guided state space model. Background Technology

[0002] All-in-one Image Restoration (AIR) aims to address complex weather degradation problems such as rain removal, snow removal, and fog removal through a single, unified framework. Existing convolutional neural network (CNN)-based methods are limited by their local receptive fields, making it difficult to capture large-scale spatial dependencies. Their restoration effectiveness is very limited when faced with degradation scenes involving extensive occlusion, such as long rain lines or dense fog. Transformer-based architectures achieve global modeling capabilities through self-attention mechanisms, effectively addressing this limitation; however, they face significant computational overhead in practical applications. Since their computational complexity increases quadratically with image resolution, they place extremely high demands on computational resources and GPU memory when processing high-resolution images.

[0003] State-space models (SSMs), exemplified by Mamba, have garnered significant attention due to their linear computational complexity and ability to model long-range dependencies. Existing visual state-space models typically rely on flattening two-dimensional image features into a one-dimensional sequence and transmitting spatial information through iterative state updates, thus achieving some efficiency improvements. Despite these advancements in computational efficiency, several core obstacles remain when handling complex image inpainting tasks. This is primarily because existing models often employ predefined, rigid geometric scanning trajectories. In practice, these content-irrelevant scanning mechanisms merely rearrange pixels geometrically, completely ignoring the semantic content of the image itself and the non-uniformity of degradation distribution. Such predefined trajectories inevitably lead to severe semantic breaks, mechanically forcibly connecting patches belonging to different semantic regions or with vastly different degrees of damage into the same one-dimensional sequence. This redundant and conflicting sequence connection makes it difficult for the model to distinguish between a clear background and large degradation noise such as raindrops and snowflakes, thus hindering the aggregation of effective features.

[0004] In complex weather scenarios, degradation phenomena often exhibit significant non-uniform distribution across different objects. Traditional content-independent geometric scanning, by ignoring physical boundaries, forcibly mixes multiple semantic regions with drastically different degradation characteristics into the same sequence, which can easily lead to cross-contamination of degradation features in the hidden states of Mamba. Summary of the Invention

[0005] This invention aims to address at least one of the technical problems existing in related technologies. To this end, this invention provides an image inpainting method and system based on a superpixel-guided state-space model, effectively solving key pain points of existing technologies when dealing with complex weather degradation problems, such as semantic fragmentation, feature cross-contamination, high computational overhead, and weak generalization ability. It demonstrates significant technical advancements and practical application value in terms of image inpainting accuracy, computational efficiency, and scene adaptability.

[0006] This invention provides an image inpainting method based on a superpixel-guided state-space model, comprising:

[0007] S1: Obtain the degraded image, extract shallow features of the degraded image through the feature embedding layer, and obtain a shallow feature map;

[0008] S2: Constructing semantic region awareness The M module, which is semantic region aware. The M module includes a superpixel Mamba and a feature refinement feedforward network. The superpixel Mamba embeds a superpixel-guided selective scanning mechanism and a region-level gating mechanism.

[0009] Semantic region awareness The working steps of module M include:

[0010] S21: Normalize the input feature map by layer normalization to obtain a normalized input feature map;

[0011] S22: Extract key features from the normalized input feature map using superpixel Mamba to obtain the superpixel guided feature map;

[0012] S23: Perform feature fusion on the normalized input feature map and the superpixel guided feature map to obtain a preliminary region-aware feature map;

[0013] S24: The initial region-aware feature map is processed by layer normalization and then input into the feature refinement feedforward network to obtain the enhanced feature map;

[0014] S25: Perform feature fusion on the preliminary region-aware feature map and the enhanced feature map to obtain the region-aware output feature map;

[0015] S3: Perceived through semantic regions The M module constructs a multi-level symmetric encoder-decoder architecture;

[0016] S4: Input the shallow feature map into the multi-level symmetric encoder-decoder architecture to perform multi-scale feature learning and obtain the deep feature map;

[0017] S5: Fuse the deep feature map and the shallow feature map to obtain a fused feature map; reconstruct the fused feature map by reconstructing the convolutional layer to obtain the repaired image.

[0018] Furthermore, the working steps of Superpixel Mamba include:

[0019] S221: Decouple the normalized input feature map in the channel dimension by linear projection and block layer to obtain local output feature map and semantically aware output feature map;

[0020] S222: The semantic perception output feature map is subjected to two-dimensional convolution to extract local spatial features and GELU activation, and then input into the superpixel generation module to obtain the superpixel label map;

[0021] S223: Extract key features from the superpixel label map through a superpixel-guided selective scanning mechanism to obtain a superpixel scan feature map;

[0022] S224: Perform regional feature calibration on the superpixel label map through a regional gating mechanism to obtain a gated feature map;

[0023] S225: Multiply the gated feature map and the superpixel scan feature map pixel by pixel, and then perform layer normalization to obtain the superpixel composite feature map;

[0024] S226: The superpixel integrated feature map, the GELU-activated local output feature map, and the semantic-aware output feature map are added element-wise, and then linear projection is used to obtain the superpixel guided feature map.

[0025] Furthermore, the superpixel generation module initializes cluster centers through gridded adaptive pooling, and then alternately performs soft allocation and cluster center updates based on feature distance calculation in the deep feature space. After multiple iterations, the feature map is adaptively divided into multiple perceptually consistent superpixel regions according to the inherent semantic features of the image.

[0026] Furthermore, step S223 includes:

[0027] The superpixel label map is rearranged using a permutation operator, and long-range features are extracted from the rearranged superpixel label map using a superpixel selective scanning mechanism. Finally, the superpixel scan feature map is obtained through inverse rearrangement.

[0028] Furthermore, the regional gating mechanism includes a series of stacked regional pooling layers, two linear layers, and a nonlinear activation layer.

[0029] Furthermore, in the multi-level symmetric encoder-decoder architecture, the encoder includes a semantic region-aware encoder. The M module and downsampling module, and the decoder includes semantic region awareness. The M module and the upsampling module fuse the multi-scale features extracted by the encoder with the corresponding upsampling features of the decoder through skip connections.

[0030] Furthermore, downsampling is performed using bilinear interpolation. Convolution is used for upsampling.

[0031] Furthermore, the feature embedding layer is Convolutional layer.

[0032] This invention also provides an image inpainting system based on a superpixel-guided state-space model, for performing the aforementioned image inpainting method based on a superpixel-guided state-space model, comprising:

[0033] A shallow feature extraction module acquires a degraded image and extracts shallow features of the degraded image through a feature embedding layer to obtain a shallow feature map.

[0034] The first construction module constructs semantic region awareness. The M module, which is semantic region aware. The M module includes a superpixel Mamba and a feature refinement feedforward network. The superpixel Mamba embeds a superpixel-guided selective scanning mechanism and a region-level gating mechanism.

[0035] Semantic region awareness The working steps of module M include:

[0036] S21: Normalize the input feature map by layer normalization to obtain a normalized input feature map;

[0037] S22: Extract key features from the normalized input feature map using superpixel Mamba to obtain the superpixel guided feature map;

[0038] S23: Perform feature fusion on the normalized input feature map and the superpixel guided feature map to obtain a preliminary region-aware feature map;

[0039] S24: The initial region-aware feature map is processed by layer normalization and then input into the feature refinement feedforward network to obtain the enhanced feature map;

[0040] S25: Perform feature fusion on the preliminary region-aware feature map and the enhanced feature map to obtain the region-aware output feature map;

[0041] The second building module, which is semantically region-aware... The M module constructs a multi-level symmetric encoder-decoder architecture;

[0042] The deep feature extraction module inputs the shallow feature map into a multi-level symmetric encoder-decoder architecture for multi-scale feature learning to obtain a deep feature map.

[0043] The repair module fuses deep and shallow feature maps to obtain a fused feature map; it then reconstructs the fused feature map using a reconstructed convolutional layer to obtain a repaired image.

[0044] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0045] This invention introduces a structure-aware scanning strategy, utilizing a superpixel algorithm to pre-perceive the inherent semantic structure of an image, clustering pixels belonging to the same semantic object or degradation feature. By strictly constraining the scanning trajectory using superpixel boundaries, this mechanism ensures a high degree of semantic continuity in the information flow entering the state space, achieving significant semantic protection; it effectively prevents information leakage between different semantic regions. The region-level gating mechanism perceives the specific degradation feature distribution within each independent semantic unit and adaptively modulates the response weights in the channel dimension, achieving fine-grained calibration of effective features, ensuring accurate restoration of consistent degradation distributions within the region, and ensuring that the model can accurately restore consistent degradation distributions within the region, thus exhibiting superior restoration accuracy and long-range modeling efficiency in complex weather scenarios. This invention effectively solves the key pain points of existing technologies in handling complex weather degradation problems, such as semantic breaks, feature cross-contamination, high computational overhead, and weak generalization ability, demonstrating significant technological advancements and practical application value in terms of image restoration accuracy, computational efficiency, and scene adaptability.

[0046] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating an image restoration method based on a superpixel-guided state-space model provided by the present invention.

[0049] Figure 2 This is a schematic diagram of the overall network of an image inpainting method based on a superpixel-guided state space model provided by the present invention.

[0050] Figure 3 This invention provides semantic region awareness. M-module structure diagram.

[0051] Figure 4 This is a visual comparison of image restoration in actual severe weather scenarios using the present invention.

[0052] Figure 5 This is a schematic diagram of the structure of an image restoration system based on a superpixel-guided state space model provided by the present invention.

[0053] Figure label:

[0054] 101. Shallow feature extraction module; 102. First construction module; 103. Second construction module; 104. Deep feature extraction module; 105. Repair module. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but cannot be used to limit the scope of this invention.

[0056] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0057] The following is combined Figures 1 to 5 This invention describes an image restoration method and system based on a superpixel-guided state-space model.

[0058] like Figure 1 As shown, an image inpainting method based on a superpixel-guided state-space model includes:

[0059] S1: Obtain the degraded image, extract shallow features of the degraded image through the feature embedding layer, and obtain a shallow feature map;

[0060] Overall network such as Figure 2 As shown, for a given degraded image , ,in The length of the input image. The width of the input image. For dimension Image set, shallow feature maps are extracted through the feature embedding layer. , ,in The number of feature channels, For dimension The image set has a feature embedding layer of 3×3 convolutional layer.

[0061] S2: Constructing semantic region awareness The M module, which is semantic region aware. The M module includes a superpixel Mamba (SP-SSM) and a feature refinement feedforward network, such as... Figure 3 As shown in Figure (a).

[0062] Addressing the core challenge of feature cross-contamination in Mamba hidden states caused by non-uniform degradation distribution under complex weather conditions, semantic region-aware... The M module incorporates superpixel Mamba and a feature refinement feedforward network (FFN). It utilizes the pre-perceived semantic boundaries of the image by the superpixels to strictly constrain the scanning trajectory, thereby capturing feature interactions within semantically consistent regions and effectively preventing information leakage between different degradation distribution regions.

[0063] Semantic region awareness The working steps of module M include:

[0064] S21: Normalize the input feature map by layer normalization to obtain a normalized input feature map;

[0065] S22: Extract key features from the normalized input feature map using superpixel Mamba to obtain the superpixel guided feature map;

[0066] SP-SSM incorporates a superpixel-guided selective scanning mechanism and a region-level gating (RGM) mechanism for key feature extraction and processing. Through a parallel dual-branch structure and semantic guidance strategy, SP-SSM achieves accurate processing of non-uniformly degenerate features.

[0067] like Figure 3 As shown in Figure (b), the working steps of superpixel Mamba (SP-SSM) include:

[0068] S221: Decouple the normalized input feature map in the channel dimension by linear projection and block layer to obtain local output feature map and semantically aware output feature map;

[0069] This step aims to decouple the high-dimensional input along the channel dimension, dividing it into two parallel processing paths: one serving as an auxiliary branch to preserve local details and information flow (local output feature map). The other branch serves as the main branch for core semantic perception and sequence modeling (semantic perception output feature map). ).

[0070] Local output feature map The input is fed into the upper branch and directly passes through a GELU activation function. This branch does not participate in complex spatial permutations; it mainly serves as a gating activation and information compensation function. Its output will be directly passed to the end of the module and fused with the final feature map.

[0071] S222: The semantic perception output feature map is subjected to two-dimensional convolution to extract local spatial features and GELU activation, and then input into the superpixel generation module to obtain the superpixel label map;

[0072] The superpixel generation module initializes cluster centers (including cluster 1, cluster 2, cluster 3, and cluster 4, etc.) through gridded adaptive pooling. Then, in the deep feature space, it alternately performs soft allocation based on feature distance calculation and cluster center update. After multiple iterations, the feature map is adaptively divided into multiple perceptually consistent superpixel regions according to the inherent semantic features of the image.

[0073] S223: Extract key features from the superpixel label map through a superpixel-guided selective scanning mechanism to obtain a superpixel scan feature map;

[0074] Existing scanning mechanisms often mechanically cross object boundaries, negatively impacting the final feature aggregation. To address this issue, this invention introduces a superpixel-guided selective scanning mechanism, which effectively prevents the model from learning non-discriminatory features from semantically conflicting regions, preserving crucial structural information.

[0075] The superpixel-guided selective scanning mechanism divides the spatial domain of the input image patch into superpixel generation modules. A region of consistent perception ,in For the first A region of consistent perception, each pixel A region label was assigned , To ensure that Mamba's state evolution is fully aligned with the semantic region, this invention defines a permutation operator. The flattened spatial markers are rearranged so that markers belonging to the same superpixel are arranged consecutively, forming a sequence for superpixel guided selective scanning (SSM).

[0076] The working steps of the superpixel-guided selective scanning mechanism include:

[0077] The superpixel label image is rearranged using a permutation operator, and the calculation expression is as follows:

[0078]

[0079] in, To rearrange the feature maps, For the permutation operator, For rearrangement operations, This is a superpixel label image.

[0080] Long-range features are extracted from the rearranged superpixel label map using a superpixel selective scanning mechanism, and then the superpixel scan feature map is obtained through inverse rearrangement. The calculation expression is as follows:

[0081]

[0082]

[0083] in, For key feature maps, This is a superpixel selective scanning operation. This is a superpixel scan feature map. It is the inverse permutation operator.

[0084] S224: Perform regional feature calibration on the superpixel label map through a regional gating mechanism to obtain a gated feature map;

[0085] The regional gating mechanism consists of a series of stacked regional pooling layers, two linear layers, and a nonlinear activation layer.

[0086] This mechanism achieves fine-grained calibration of effective features by perceiving the specific degradation feature distribution within each independent semantic unit and adaptively modulating the response weights in the channel dimension, ensuring accurate restoration of a consistent degradation distribution within the region.

[0087] To further overcome the burden caused by local anomalies (such as large raindrops or dense fog) in specific recovery tasks, this invention designs a region-level gating mechanism to calibrate features within the region along the channel dimension. A region pooling layer is applied to the superpixel label map. Inside this layer, for each perceptually consistent region, the mean and variance of all pixel features within that region along the channel dimension are calculated:

[0088]

[0089]

[0090] in, The mean, For variance, pixel coordinates Superpixel label image at the location, The number of regions with consistent perception. For the first A region with consistent perception;

[0091] To detect degradation features within each semantic unit, these statistics are concatenated as follows: , The vector space is 2C-dimensional real vector space, processed by a lightweight multilayer perceptron (MLP) containing nonlinear activation functions and having two fully connected layers and one nonlinear activation layer. This MLP generates a region-specific gating factor. , , Given a C-dimensional real vector space, the channel-level feature responses are adaptively modulated. For those belonging to... The A marker image , The gating update and subsequent state evolution formulas are as follows:

[0092]

[0093]

[0094] in, For the Sigmoid function, After weighting the gating, the first A marked image, For element-wise multiplication, For the first The hidden state of a marker graph, For the first The hidden state of a marker graph, The state transition weight matrix is... The input feature weight matrix is ​​used. Through this gating mechanism, the network can maximally suppress the influence of aberrant degradation features and provide content-adaptive state transitions.

[0095] S225: Multiply the gated feature map and the superpixel scan feature map pixel by pixel, and then perform layer normalization to obtain the superpixel composite feature map;

[0096] S226: The superpixel integrated feature map, the GELU-activated local output feature map, and the semantic-aware output feature map are added element-wise, and then linear projection is used to obtain the superpixel guided feature map.

[0097] S23: Perform feature fusion on the normalized input feature map and the superpixel guided feature map to obtain a preliminary region-aware feature map;

[0098] S24: The initial region-aware feature map is processed by layer normalization and then input into the feature refinement feedforward network to obtain the enhanced feature map;

[0099] The feature refinement feedforward network employs a Feed-Forward Network (FFN) to further enhance and refine the representation of features.

[0100] S25: Perform feature fusion on the preliminary region-aware feature map and the enhanced feature map to obtain the region-aware output feature map.

[0101] S3: Perceived through semantic regions The M module constructs a multi-level symmetric encoder-decoder architecture;

[0102] In a multi-level symmetric encoder-decoder architecture, the encoder includes a semantic region-aware encoder. The M module and downsampling module, and the decoder includes semantic region awareness. The M-module and upsampling module perform cross-layer fusion of multi-scale features extracted by the encoder with corresponding upsampled features from the decoder via skip connections, and downsampling is performed using bilinear interpolation. Convolution is used for upsampling.

[0103] In the multi-level symmetric encoder-decoder architecture, semantic region awareness is adopted at each level. The M module performs core feature processing.

[0104] S4: Input the shallow feature map into the multi-level symmetric encoder-decoder architecture to perform multi-scale feature learning and obtain the deep feature map;

[0105] S5: Fuse the deep feature map and the shallow feature map to obtain a fused feature map; reconstruct the fused feature map by reconstructing the convolutional layer to obtain the repaired image.

[0106] like Figure 5 As shown, an image inpainting system based on a superpixel-guided state-space model is used to execute the aforementioned image inpainting method based on a superpixel-guided state-space model, comprising:

[0107] The shallow feature extraction module 101 acquires the degraded image and extracts shallow features of the degraded image through the feature embedding layer to obtain a shallow feature map;

[0108] The first building module 102 constructs semantic region awareness. The M module, which is semantic region aware. The M module includes a superpixel Mamba and a feature refinement feedforward network. The superpixel Mamba embeds a superpixel-guided selective scanning mechanism and a region-level gating mechanism.

[0109] Semantic region awareness The working steps of module M include:

[0110] S21: Normalize the input feature map by layer normalization to obtain a normalized input feature map;

[0111] S22: Extract key features from the normalized input feature map using superpixel Mamba to obtain the superpixel guided feature map;

[0112] S23: Perform feature fusion on the normalized input feature map and the superpixel guided feature map to obtain a preliminary region-aware feature map;

[0113] S24: The initial region-aware feature map is processed by layer normalization and then input into the feature refinement feedforward network to obtain the enhanced feature map;

[0114] S25: Perform feature fusion on the preliminary region-aware feature map and the enhanced feature map to obtain the region-aware output feature map;

[0115] The second building module 103 uses semantic region awareness. The M module constructs a multi-level symmetric encoder-decoder architecture;

[0116] The deep feature extraction module 104 inputs the shallow feature map into the multi-level symmetric encoder-decoder architecture for multi-scale feature learning to obtain the deep feature map;

[0117] The repair module 105 fuses the deep feature map and the shallow feature map to obtain a fused feature map; it then reconstructs the fused feature map by using a reconstructed convolutional layer to obtain the repaired image.

[0118] Through the collaborative work of the aforementioned modules, this invention introduces a structure-aware scanning strategy, utilizing a superpixel algorithm to pre-perceive the inherent semantic structure of the image, clustering pixels belonging to the same semantic object or degradation feature into clusters. By strictly constraining the scanning trajectory using superpixel boundaries, this mechanism ensures a high degree of semantic continuity in the information flow entering the state space, achieving significant semantic protection; it effectively prevents information leakage between different semantic regions. The region-level gating mechanism perceives the specific degradation feature distribution within each independent semantic unit and adaptively modulates the response weights in the channel dimension, achieving fine-grained calibration of effective features, ensuring accurate restoration of consistent degradation distributions within the region, and ensuring that the model can accurately restore consistent degradation distributions within the region, thus exhibiting superior restoration accuracy and long-range modeling efficiency in complex weather scenarios. This invention effectively solves the key pain points of existing technologies in handling complex weather degradation problems, such as semantic breaks, feature cross-contamination, high computational overhead, and weak generalization ability, demonstrating significant technological advancements and practical application value in terms of image restoration accuracy, computational efficiency, and scene adaptability.

[0119] To verify the effectiveness and advancement of this invention in low-quality image restoration tasks, extensive experiments and quantitative performance comparisons were conducted on multiple publicly available evaluation datasets and real-world degraded images. Table 1 shows the performance of this invention on four mainstream publicly available evaluation datasets, including small-sample snow removal (Snow100K-S), large-sample snow removal (Snow100K-L), combined rain and haze removal (Outdoor), and raindrop removal (RainDrop). This invention uses Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) metrics to evaluate image restoration performance. Higher PSNR and SSIM values ​​indicate better image restoration results and a smaller difference between the reconstructed image and the original clear image.

[0120] Table 1 shows a comparison between this invention and 18 existing state-of-the-art image restoration methods. The results demonstrate that this invention outperforms all published results in image restoration across multiple tasks, achieving the highest average performance. The average PSNR reaches 34.15 dB, and the average SSIM reaches 0.9442. On the Outdoor dataset, which includes complex joint degradation scenarios, the PSNR reaches 33.22 dB, significantly exceeding the second-ranked CPL_PromptIR (TPAMI'26) method by 1.06 dB. This fully demonstrates the superior robustness and generalization ability of this invention.

[0121] Table 1: Quantitative comparison results of the present invention and existing advanced methods on public datasets.

[0122]

[0123] Table 2 illustrates the advantages of this invention in terms of model efficiency and computational complexity. While ensuring extremely high image restoration quality, this invention demonstrates excellent parameter economy. The number of model parameters in this invention is only 7.05M, the lowest among all compared methods. Compared to recent high-performance baseline methods, this invention improves restoration performance by 1.06dB while significantly reducing model size by approximately 80.4%. Furthermore, in terms of computational cost (FLOPs), this invention requires only 54.27G of computation, less than half that of some traditional large models, achieving a perfect balance between reconstruction capability and execution efficiency, which is beneficial for deployment and application in real-world industrial environments.

[0124] Table 2: Performance and complexity comparison on the Outdoor dataset

[0125]

[0126] Figure 4 This paper demonstrates the visual comparison of image restoration using the present invention in real-world severe weather scenarios (de-raindrop, combined defogging and de-raining, and de-snowing). Through comparison of the global image and magnified local details, it is clear that conventional methods often result in loss of detail or blurring when dealing with severe degradation and occlusion. The present invention, however, not only effectively eliminates visual artifacts such as dense snowflakes and severe water droplets, but also perfectly preserves sharp structural content. For example, in the combined defogging and de-raining task, the present invention successfully restored the extremely subtle "WAY OUT" characters on a blue sign; in the de-snowing task, it eliminated dense snowflakes without causing any blurring or loss of the edges of small objects.

[0127] Table 3 further illustrates the reference-free quality assessment results of this invention on real-world datasets (RainDrop-real and NTUrain-real). Since real-world degraded images lack standard, clear reference images, this invention employs the advanced multimodal evaluation metric Q-Align to assess the perceptual quality and aesthetic scores of the images. The results show that this invention surpasses existing methods across all real-world quality dimensions, further demonstrating its high practicality and effectiveness in complex and varied real-world scenarios.

[0128] Table 3: Comparison of parameter-free metric tests on real datasets

[0129]

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image inpainting method based on a superpixel-guided state-space model, characterized in that, include: S1: Obtain the degraded image, extract shallow features of the degraded image through the feature embedding layer, and obtain a shallow feature map; S2: Constructing semantic region awareness The M module, which is semantic region aware. The M module includes a superpixel Mamba and a feature refinement feedforward network. The superpixel Mamba embeds a superpixel-guided selective scanning mechanism and a region-level gating mechanism. Semantic region awareness The working steps of module M include: S21: Normalize the input feature map by layer normalization to obtain a normalized input feature map; S22: Extract key features from the normalized input feature map using superpixel Mamba to obtain the superpixel guided feature map; The working steps of Superpixel Mamba include: S221: Decouple the normalized input feature map in the channel dimension by linear projection and block layer to obtain local output feature map and semantically aware output feature map; S222: The semantic perception output feature map is subjected to two-dimensional convolution to extract local spatial features and GELU activation, and then input into the superpixel generation module to obtain the superpixel label map; S223: Extract key features from the superpixel label map through a superpixel-guided selective scanning mechanism to obtain a superpixel scan feature map; S224: Perform regional feature calibration on the superpixel label map through a regional gating mechanism to obtain a gated feature map; S225: Multiply the gated feature map and the superpixel scan feature map pixel by pixel, and then perform layer normalization to obtain the superpixel composite feature map; S226: The superpixel integrated feature map, the GELU-activated local output feature map, and the semantic-aware output feature map are added element-wise, and then linear projection is used to obtain the superpixel guided feature map. S23: Perform feature fusion on the normalized input feature map and the superpixel guided feature map to obtain a preliminary region-aware feature map; S24: The initial region-aware feature map is processed by layer normalization and then input into the feature refinement feedforward network to obtain the enhanced feature map; S25: Perform feature fusion on the preliminary region-aware feature map and the enhanced feature map to obtain the region-aware output feature map; S3: Perceived through semantic regions The M module constructs a multi-level symmetric encoder-decoder architecture; In a multi-level symmetric encoder-decoder architecture, Encoders include semantic region-aware encoders. The M module and downsampling module, and the decoder includes semantic region awareness. The M module and the upsampling module fuse the multi-scale features extracted by the encoder with the corresponding upsampling features of the decoder through skip connections. S4: Input the shallow feature map into the multi-level symmetric encoder-decoder architecture to perform multi-scale feature learning and obtain the deep feature map; S5: Fuse the deep feature map and the shallow feature map to obtain a fused feature map; reconstruct the fused feature map by reconstructing the convolutional layer to obtain the repaired image.

2. The image inpainting method based on a superpixel-guided state-space model according to claim 1, characterized in that, The superpixel generation module initializes cluster centers through gridded adaptive pooling, and then alternately performs soft allocation and cluster center updates based on feature distance calculation in the deep feature space. After multiple iterations, based on the inherent semantic features of the image, the feature map is adaptively divided into multiple perceptually consistent superpixel regions.

3. The image inpainting method based on a superpixel-guided state-space model according to claim 1, characterized in that, Step S223 includes: The superpixel label map is rearranged using a permutation operator, and long-range features are extracted from the rearranged superpixel label map using a superpixel selective scanning mechanism. Finally, the superpixel scan feature map is obtained through inverse rearrangement.

4. The image inpainting method based on a superpixel-guided state-space model according to claim 1, characterized in that, The regional gating mechanism consists of a series of stacked regional pooling layers, two linear layers, and a nonlinear activation layer.

5. The image inpainting method based on a superpixel-guided state-space model according to claim 1, characterized in that, Downsampling is performed using bilinear interpolation. Convolution is used for upsampling.

6. The image inpainting method based on a superpixel-guided state-space model according to claim 1, characterized in that, The feature embedding layer is Convolutional layer.

7. An image inpainting system based on a superpixel-guided state-space model, characterized in that, An image inpainting method based on a superpixel-guided state-space model as described in any one of claims 1 to 6, comprising: A shallow feature extraction module acquires a degraded image and extracts shallow features of the degraded image through a feature embedding layer to obtain a shallow feature map. The first construction module constructs semantic region awareness. The M module, which is semantic region aware. The M module includes a superpixel Mamba and a feature refinement feedforward network. The superpixel Mamba embeds a superpixel-guided selective scanning mechanism and a region-level gating mechanism. Semantic region awareness The working steps of module M include: S21: Normalize the input feature map by layer normalization to obtain a normalized input feature map; S22: Extract key features from the normalized input feature map using superpixel Mamba to obtain the superpixel guided feature map; S23: Perform feature fusion on the normalized input feature map and the superpixel guided feature map to obtain a preliminary region-aware feature map; S24: The initial region-aware feature map is processed by layer normalization and then input into the feature refinement feedforward network to obtain the enhanced feature map; S25: Perform feature fusion on the preliminary region-aware feature map and the enhanced feature map to obtain the region-aware output feature map; The second building module, which is semantically region-aware... The M module constructs a multi-level symmetric encoder-decoder architecture; The deep feature extraction module inputs the shallow feature map into a multi-level symmetric encoder-decoder architecture for multi-scale feature learning to obtain a deep feature map. The repair module fuses deep and shallow feature maps to obtain a fused feature map; it then reconstructs the fused feature map using a reconstructed convolutional layer to obtain a repaired image.

Images