Lightweight image inpainting method

Abstract

This invention discloses a lightweight image restoration method, relating to the fields of computer vision and image processing technology, comprising: S1, input preprocessing: inputting a damaged image and a size-matched binary mask, concatenating the two along the channel dimension to obtain an input tensor; S2, encoder feature extraction: configuring an encoder network composed of multiple downsampling blocks to extract multi-scale features; the downsampling block includes an LSConv module with improved LSNet convolution, convolutional layers, normalization layers, and activation functions. The LSConv module, with its separable convolutional structure based on multi-branch depth, simultaneously captures global structure and local detail features. Feature correction and fusion are completed through feature concatenation and channel adjustment combined with the SE attention mechanism. This invention innovatively designs a novel LSConv feature extraction module, employing a multi-branch parallel structure combined with multi-scale feature mining, breaking through the limitations of traditional single convolution, simultaneously capturing global and local features, and enhancing feature expression through channel optimization, thereby improving image restoration capabilities.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing technology, specifically to a lightweight image restoration method. Background Technology

[0002] With the rapid development of deep learning technology, image restoration technology based on convolutional neural networks has become the core means of image defect restoration. It is widely used in many scenarios such as old photo restoration, image defect repair, and visual content improvement. Most existing image restoration networks adopt conventional convolution stacked encoding and decoding structures to complete basic feature extraction and image restoration, and can achieve simple image restoration tasks.

[0003] Current traditional image restoration models still suffer from several common industry-wide shortcomings, limiting their practical application experience. First, traditional fixed convolutional structures have limited receptive fields, failing to flexibly consider both global structural information and subtle local textures. Their feature extraction methods are also singular and rigid, making them unsuitable for restoring complex damaged images. Second, existing network structures are redundant and complex, with a large number of parameters and computational overhead, resulting in insufficient model lightweighting and difficulty in deployment on low-computing-power devices such as mobile devices and embedded systems. Furthermore, conventional normalization methods have poor adaptability, easily leading to feature distribution drift during single-image inference and insufficient feature representation stability. The lack of a robust feature association and matching mechanism prevents the reasonable inference of missing content from intact image regions. The simple decoder design results in weak cross-layer feature fusion capabilities, easily leading to edge breaks, texture distortion, and visual artifacts after restoration, making the overall restoration effect insufficient for practical applications. To address these issues, we propose a lightweight image restoration method. Summary of the Invention

[0004] To address the aforementioned technical problems, a lightweight image restoration method is provided. This technical solution solves the problems of existing image restoration models being complex, slow, and having poor restoration effects.

[0005] To achieve the above objectives, the technical solution adopted by this invention is: a lightweight image restoration method, the specific restoration steps of which are as follows: S1. Input preprocessing: Input the damaged image and the size-matched binary mask, and concatenate the two in the channel dimension to obtain the input tensor; S2. Encoder Feature Extraction: Configure an encoder network consisting of multiple downsampling blocks to extract multi-scale features. The downsampling block contains an improved LSConv module of LSNet convolution, convolutional layers, normalization layers and activation functions. The LSConv module can separate the convolutional structure according to the multi-branch depth, and simultaneously capture global structure and local detail features. After feature concatenation and channel adjustment, feature fusion and aggregation are completed by combining the SE attention mechanism. S3, Deep Feature Completion: Configure an improved contextual attention module on the deepest feature map of the encoder, normalize the features of known and missing regions and calculate the similarity matrix, generate attention weights after normalization, and combine the weights to weighted aggregate the features of known regions. S4. Decoder Feature Reconstruction: Configure a decoder network consisting of multiple upsampling blocks to restore the original resolution of the image through step-by-step upsampling operations; S5. Multi-scale output fusion: Convolutional operations are used to integrate feature channels to generate a preliminary repaired image. A binary mask is then used to complete pixel-level fusion and output a high-quality repaired image.

[0006] Preferably, in step S1, the pixel values ​​of the binary mask are only 0 or 1, where pixel value 1 corresponds to the damaged or missing area of ​​the image, and pixel value 0 corresponds to the complete area of ​​the image. The acquired damaged image and the pre-matched binary mask are aligned in size and calibrated channel by channel, and then stitched together along the channel dimension direction to form a uniform input feature tensor according to the multi-dimensional channel fusion method. Preferably, in step S2, the downsampling block includes an improved LSNet convolution LSConv module, convolutional layers, normalization layers, and activation functions. The number of channels in the convolutional layers in the sampling block is the same as the number of channels in the input image, and the convolutional kernel stride is fixed at 2. Preferably, in step S2, the LSConv module is configured with three sets of parallel branch structures. Each branch employs convolutional methods with different receptive fields for multi-dimensional feature extraction. The first branch uses large-scale depthwise separable convolution to capture global contextual information and overall structural features of the image. After convolutional processing within the branch, batch normalization and non-linear activation are performed sequentially. The second branch uses small-scale depthwise separable convolution to extract local texture and detail features on the image surface, simultaneously performing batch normalization and non-linear activation operations. The third branch uses small-size standard convolution to uniformly adjust the dimensionality of the output features from multiple branches and to build residual connection channels. After extracting features at different scales from the three branches, features are uniformly spliced ​​and fused along the channel dimension. The number of feature channels is then adaptively adjusted through convolutional layers, and the feature weight parameters of each channel are dynamically calibrated using the SE attention mechanism to output multi-scale feature aggregation. Preferably, in step S3, the multi-scale aggregated feature map obtained by aggregation is subjected to channel-by-channel feature normalization to eliminate the numerical differences between features at different scales; then the normalized features are divided into damaged area features and intact area features, and the two types of features are matched one by one. By comparing the similarity of feature vectors, a similarity matrix representing the degree of correlation between damaged area and intact area features is calculated.

[0007] Preferably, in step S3, the feature similarity matrix is ​​globally normalized using a normalized exponential function, mapping the values ​​within the matrix to a specified range of values, thereby generating spatial attention weights for regional feature matching and content completion. Based on this attention weight, effective features of known complete regions in the image are weighted and allocated. Then, a weighted summation operation is performed on the allocated features, and the calculated target feature values ​​are filled into the corresponding damaged and missing regions of the image, thus completing the feature repair and completion of the missing regions. Preferably, in step S4, the deep feature map after feature completion is upsampled and reconstructed step by step through a decoder network to restore the original spatial resolution of the repaired image. The decoder contains multiple upsampling blocks, each of which includes an upsampling unit, an LSConv module, a skip connection unit, an instance normalization layer, and a ReLU activation function. The upsampling unit uses nearest neighbor upsampling or transposed convolution to enlarge the feature map size and restore the spatial dimension.

[0008] Preferably, in step S4, the decoder utilizes skip connection units to establish a feature transmission path between the encoder and decoder, performing deep concatenation and fusion of the upsampled and amplified high-level semantic features with the shallow detail features retained at the corresponding level of the encoder. The fused features are then extracted by the LSConv module. Preferably, in step S4, the decoder progressively amplifies the feature map resolution through each upsampling block. Each upsampling block first upsamples and increases the dimensionality of the output features from the previous level, then fuses the shallow texture and deep structural features of the corresponding level of the encoder through skip connections. After the fused features are refined and extracted by the LSConv module, feature regularization and nonlinear mapping transformation are sequentially performed through instance normalization layers and ReLU activation functions. This process iterates step-by-step until the feature map size is restored to the original size of the input damaged image, completing the full feature reconstruction.

[0009] Preferably, in step S5, the reconstructed features output by the decoder are integrated through channels to generate a preliminary repaired image with a complete structure; then, the binary mask from the input stage is retrieved, and according to the region division attributes of the mask, the pixels of the original complete region and the pixels of the algorithm repaired region are weighted and fused at the pixel level, and the repair boundary is smoothed; at the same time, global detail optimization is performed on the entire image to suppress noise and artifacts generated during the repair process, unify the overall texture and color style of the image, and finally output a high-quality repaired image.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention innovatively designs a novel LSConv feature extraction module, employing a multi-branch parallel structure combined with multi-scale feature mining to overcome the performance limitations of traditional single convolution. It can simultaneously capture global structural information and local texture details, and enhances feature expression through a channel optimization mechanism, improving the overall image restoration capability from the core network structure level. By simplifying the module structure and optimizing convolution operations, it significantly reduces the number of model parameters and computational consumption, effectively lowering the device's computing power threshold and adapting to deployment scenarios on mobile devices and resource-constrained devices, demonstrating significant lightweight advantages. It adds feature similarity calculation and attention-weighted filling mechanisms, reasonably filling in missing content based on known image region features, enhancing the semantic relevance of the restored region. Simultaneously, it uses a reasonable normalization strategy to ensure feature stability under single-image inference, improving the rationality and consistency of the restored content. The decoder adopts a fusion design combining multi-level upsampling blocks and skip connections, fully integrating multi-level encoding and decoding feature information, optimizing the image upsampling restoration effect, effectively weakening the problem of restoration boundary fragmentation, suppressing noise and artifact generation, and ultimately obtaining a high-quality restored image with complete structure and natural, coherent texture. Attached Figure Description

[0011] Figure 1 This is a flowchart of the steps of the present invention; Figure 2 This is a diagram of the core innovative module of the present invention. Detailed Implementation

[0012] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0013] Reference Figure 1 As shown, a lightweight image restoration method includes: S1. Input preprocessing: Input the damaged image and the size-matched binary mask, and concatenate the two in the channel dimension to obtain the input tensor; S2. Encoder Feature Extraction: Configure an encoder network consisting of multiple downsampling blocks to extract multi-scale features. The downsampling block contains an improved LSConv module of LSNet convolution, convolutional layers, normalization layers and activation functions. The LSConv module can separate the convolutional structure according to the multi-branch depth, and simultaneously capture global structure and local detail features. After feature concatenation and channel adjustment, feature fusion and aggregation are completed by combining the SE attention mechanism. S3, Deep Feature Completion: Configure an improved contextual attention module on the deepest feature map of the encoder, normalize the features of known and missing regions and calculate the similarity matrix, generate attention weights after normalization, and combine the weights to weighted aggregate the features of known regions. S4. Decoder Feature Reconstruction: Configure a decoder network consisting of multiple upsampling blocks to restore the original resolution of the image through step-by-step upsampling operations; S5. Multi-scale output fusion: Convolutional operations are used to integrate feature channels to generate a preliminary repaired image. A binary mask is then used to complete pixel-level fusion and output a high-quality repaired image.

[0014] This application discloses a lightweight image restoration method. It extracts multi-scale features by configuring an encoder with an improved LSConv module and optimizes feature representation using an SE attention mechanism. An improved contextual attention module is used to complete deep damage features. Feature reconstruction is achieved through a decoder with skip connections, and pixel-level feature fusion is realized using a binary mask. This invention addresses the problems of high complexity, slow restoration speed, unnatural connection between the restored and original regions, and susceptibility to noise artifacts in existing image restoration models. While reducing model parameters and improving restoration efficiency, it also improves the texture coherence and image quality uniformity of the restored image, achieving a balance between lightweight and high-quality restoration. It is suitable for various scenarios involving rapid restoration of damaged images.

[0015] In step S1, the pixel values ​​of the binary mask are only 0 or 1, where pixel value 1 corresponds to the damaged or missing area of ​​the image, and pixel value 0 corresponds to the complete area of ​​the image. The acquired damaged image and the pre-matched binary mask are aligned in size and calibrated dimensionally along the channel dimension. The images are then stitched together and processed along the channel dimension direction to form a uniform input feature tensor based on the multi-dimensional channel fusion method.

[0016] The damaged image to be repaired is acquired. The damaged image is an RGB three-channel color image or a single-channel grayscale image. The image resolution can be flexibly set according to the actual application scenario. In this embodiment, the image resolution is set to 256×256 pixels, and the pixel value range is normalized to the [0,1] interval. A binary mask that perfectly matches the size of the damaged image is generated. The pixel values ​​of the binary mask are only 0 or 1. The pixel value 1 precisely corresponds to the damaged and missing areas of the image (including scratches, stains, missing blocks, etc.), and the pixel value 0 corresponds to the intact and preserved areas of the image. The generation of the binary mask can be achieved by manual annotation or automatic detection algorithms (such as threshold segmentation and edge detection algorithms) to ensure that the mask accurately corresponds to the damaged areas of the damaged image. After preparing the damaged image and binary mask, they are first aligned in size and calibrated channel by channel to ensure that the width and height of the damaged image and the binary mask are completely consistent and that the channel dimensions match each other, avoiding dimensional misalignment during the stitching process. The damaged image and the binary mask are then stitched and processed along the channel dimension direction. The feature information of the two is integrated according to the multi-dimensional channel fusion method to form a uniform input feature tensor. The number of channels in the input feature tensor is the sum of the number of channels in the damaged image and the number of channels in the binary mask. This preprocessing operation provides standardized and uniform input data for subsequent encoder feature extraction. At the same time, channel fusion enables the model to quickly identify the location of the damaged area, improving the targeting of feature extraction.

[0017] In step S2, the downsampling block contains the LSConv module of the improved LSNet convolution, convolutional layers, normalization layers, and activation functions. The number of channels in the convolutional layers in the sampling block is the same as the number of channels in the input image, and the convolutional kernel stride is fixed at 2. The normalization layer specifically employs instance normalization, using a single damaged image as an independent processing sample to complete feature standardization correction, avoiding feature bias caused by batch statistics, and regularizing the feature distribution of damaged images; the activation function adopts the LeakyReLU activation function to perform nonlinear mapping transformation on the normalized feature data.

[0018] The encoder network consists of multiple downsampling blocks connected in series. In this embodiment, four downsampling blocks are preferably set. Each downsampling block sequentially extracts features and compresses the size of the input feature tensor, gradually obtaining multi-scale features of the image, shallow texture detail features and deep semantic structure features, thus completing the hierarchical extraction and aggregation of image features. Each downsampling block contains an improved LSConv module for LSNet convolution, convolutional layers, normalization layers, and activation functions, as follows: The convolutional layers in the downsampling block use 3×3 kernels, with the number of channels matching the number of channels in the input image. The kernel stride is fixed at 2. After convolutional processing, the feature map size is compressed, gradually increasing the feature abstraction level while controlling the computational load of the model. The normalization layer specifically adopts instance normalization, which differs from the batch statistical processing method of batch normalization. Instance normalization uses a single damaged image as an independent processing sample, performs feature standardization correction on each channel of each image separately, calculates the mean and variance of that channel of a single image, and normalizes the feature values ​​to a standard normal distribution. This can better regulate the feature distribution of the damaged image and improve the stability of feature extraction. The activation function used is LeakyReLU, which preserves gradient propagation in the negative region and performs nonlinear mapping transformation on the normalized feature data, enhancing the model's feature representation ability and enabling the model to better learn the feature association between damaged and intact images.

[0019] In step S2, the LSConv module sets up three sets of parallel branch structures; Each branch employs a different convolutional method with a different receptive field to extract multi-dimensional features. The first branch uses large-scale depthwise separable convolution to capture global contextual information and overall structural features of the image. After convolution processing within the branch, batch normalization and non-linear activation are performed sequentially. The second branch uses small-scale depthwise separable convolution to extract local texture and detail features of the image surface, and simultaneously performs batch normalization and non-linear activation operations. The third branch uses small-size standard convolution to uniformly adjust the dimensionality of the output features of multiple branches and to build residual connection channels. After extracting features at different scales from the three branches, the features are spliced ​​and fused uniformly in the channel dimension. Then, the number of feature channels is adaptively adjusted through the convolutional layer, and the feature weight parameters of each channel are dynamically calibrated by combining the SE attention mechanism to output multi-scale feature aggregation.

[0020] The first branch employs large-scale depthwise separable convolution with 7×7 kernels. This convolution method splits standard convolution into depthwise convolution and pointwise convolution, reducing the model parameter size while expanding the receptive field of feature extraction. This allows for the capture of global contextual information and overall contour structure features of the image over a wide range, reducing overall image structure distortion caused by local feature extraction. After the convolution operation is completed in this branch, batch normalization and nonlinear activation are performed sequentially. The activation function chosen is LeakyReLU, which is used to further regularize the feature distribution and enhance the nonlinear expression of features. The second branch employs small-scale depthwise separable convolution with a 3×3 kernel, also using a structure combining depthwise and pointwise convolutions. It focuses on accurately extracting local textures and subtle details from the image surface (such as texture lines and color gradations), compensating for the loss of detail features caused by the large-scale convolution in the first branch. This branch simultaneously performs batch normalization and non-linear activation operations, maintaining consistency with the processing flow of the first branch to ensure the consistent distribution of output features between the two branches. The third branch uses a small-size standard convolution with a 1×1 kernel. Its core function is to unify and adjust the dimensionality of the output features of the first two branches, making the multi-scale feature dimensions of the first and second branches consistent, which facilitates the splicing and fusion of subsequent channel dimensions. At the same time, this branch builds short-distance residual connection channels, directly performing residual fusion between the branch input features and the output features, reducing the loss of detailed information during multi-layer feature transmission and improving the integrity of feature transmission. Feature concatenation and fusion integrates the multi-scale features output from the three branches into a single feature map along the channel dimension. A 1×1 convolutional layer adaptively adjusts the number of feature channels to match the number of convolutional channels in the current downsampling block, ensuring consistent feature propagation. The SE attention mechanism dynamically calibrates the feature weights for each channel. First, global average pooling is performed on the concatenated feature map to obtain global feature statistics for each channel. Then, two fully connected layers construct channel attention weights to score the importance of each channel. Finally, the attention weights are multiplied channel-by-channel with the original feature map to enhance effective feature representation and suppress unwanted interference.

[0021] In step S3, the multi-scale aggregated feature map obtained by aggregation is subjected to channel-by-channel feature normalization to eliminate the numerical differences between features at different scales. Then, the normalized features are divided into damaged area features and intact area features. The two types of features are matched one by one. By comparing the numerical values ​​between feature vectors, a similarity matrix representing the degree of correlation between damaged area and intact area features is calculated.

[0022] The multi-scale aggregated feature map obtained through aggregation is subjected to channel-by-channel feature normalization. This normalization operation balances the numerical differences between features at different scales, ensuring a uniform and regular distribution of features across all channels, thus providing a stable feature foundation for subsequent feature matching and similarity calculation. Based on the generated binary mask, the normalized deep feature map is divided into two categories: damaged region features and intact region features. These two categories are then matched one-to-one. Each feature vector of the damaged region is compared one by one with all feature vectors of the complete region. By calculating the Euclidean distance between the feature vectors, the similarity between the two types of features is quantitatively characterized. Then, a similarity matrix is ​​generated to represent the degree of correlation between the features of the damaged region and the complete region. The dimension of the similarity matrix is ​​the number of pixels in the damaged region × the number of pixels in the complete region. Each element in the matrix corresponds to the feature similarity between a certain pixel in the damaged region and a certain pixel in the complete region. The larger the value, the more similar the features are.

[0023] In step S3, the feature similarity matrix is ​​globally normalized using a normalized exponential function, mapping the values ​​in the matrix to a specified range of values, and generating spatial attention weights for regional feature matching and content completion. Based on this attention weight, the effective features of the known complete region of the image are weighted and allocated, and then a weighted summation operation is performed on the allocated features. The target feature value obtained by the operation is filled into the corresponding damaged and missing region of the image, thus completing the feature repair and filling of the missing region. The effective features are texture features and semantic features that are not damaged in the intact region of the image and can be used for feature matching and feature filling. Redundant interference information is eliminated, providing a reliable reference for feature filling of the damaged region.

[0024] The feature similarity matrix is ​​globally normalized by a normalized exponential function (preferably the Softmax function in this embodiment), mapping all values ​​in the matrix to the [0,1] interval, with each row summing to 1, generating spatial attention weights for regional feature matching and content completion. These weights reflect the contribution of pixels in the complete region to the damaged region. Based on this spatial attention weight, the effective deep features of the complete region are assigned a targeted weight. The target feature value obtained by multiplying the feature vector of the complete region with the corresponding weight element by element and summing them is accurately filled into the damaged and missing position, thus completing the feature completion of the missing region and providing high-quality input for subsequent decoder feature reconstruction.

[0025] In step S4, the decoder network performs stepwise upsampling reconstruction on the deep feature map after feature completion to restore and repair the original spatial resolution of the image. The decoder contains multiple upsampling blocks, each of which includes an upsampling unit, an LSConv module, a skip connection unit, an instance normalization layer, and a ReLU activation function. The upsampling unit uses nearest neighbor upsampling or transposed convolution to enlarge the feature map size and restore the spatial dimension.

[0026] The specific settings of each component are as follows: The upsampling unit can selectively use either nearest neighbor upsampling or transposed convolution. In this embodiment, the first three upsampling blocks use transposed convolution with a kernel size of 4×4 and a stride of 2. The last upsampling block uses nearest neighbor upsampling to achieve the enlargement of the feature map size and restoration of spatial dimensions. After each upsampling block, the feature map size doubles and the number of channels is halved, gradually restoring the resolution and number of channels of the original image.

[0027] In step S4, the decoder uses a skip connection unit to establish a high-speed feature transmission path between the encoder and decoder. The upsampled and amplified high-level semantic features are deeply spliced ​​and fused with the shallow detail features retained by the corresponding level of the encoder. The LSConv module then performs secondary optimization and purification on the fused hybrid features.

[0028] The decoder establishes a high-speed feature transmission path between the encoder and decoder through a built-in skip connection unit, enabling cross-level feature interaction and reuse. The skip connection unit of each upsampling block is connected to the corresponding level feature of the encoder, and the high-level semantic features after upsampling by the decoder are deeply stitched and fused with the shallow detail features of the encoder along the channel dimension, taking into account both the overall structure and detail richness of the image. The fused hybrid features are input into the LSConv module for feature extraction. The LSConv module has three parallel branches and uses convolution with different receptive fields to extract and optimize the hybrid features, select effective features, suppress redundant features, strengthen multi-level feature associations, and improve the quality of fused features.

[0029] In step S4, the decoder progressively amplifies the feature map resolution through each upsampling block. Each upsampling block first upsamples and increases the dimensionality of the output features from the previous level, and then fuses the shallow texture and deep structural features of the corresponding level of the encoder through skip connections. After the LSConv module refines and extracts the fused features, the features are then normalized and nonlinearly mapped through instance normalization layers and ReLU activation functions in sequence. The above operations are iterated step by step until the feature map size is restored to the original size of the input damaged image, thus completing the complete feature reconstruction.

[0030] The specific workflow of each upsampling block is as follows: First, the deep features output from the previous level are upsampled and upgraded, increasing the feature map size and adjusting the number of channels to the corresponding specifications. Then, through skip connection units, the shallow texture information and deep structural semantic information output from the corresponding level of the encoder are fused and integrated to achieve complementarity between shallow and deep features. Subsequently, the LSConv module is used to refine and optimize the fused features, selecting effective features. Finally, the features pass through an instance normalization layer and a ReLU activation function in sequence. The instance normalization layer standardizes and regularizes the fused and optimized features, while the ReLU activation function introduces a nonlinear transformation to enhance the expressive power of the features and avoid the gradient vanishing problem. Repeat the above process, and after processing by 4 upsampling blocks, the full-dimensional feature reconstruction is completed. The feature map size and number of channels are restored to the original specifications, resulting in a reconstructed feature map with complete structure and coherent semantics.

[0031] In step S5, the reconstructed features output by the decoder are integrated through channels to generate a preliminary restored image with a complete structure. Then, the binary mask from the input stage is retrieved, and the pixels of the original complete region and the pixels of the algorithm-restored region are weighted and fused at the pixel level according to the region division attributes of the mask. The restoration boundary is also smoothed. At the same time, global detail optimization is performed on the entire image to suppress noise and artifacts generated during the restoration process, unify the overall texture and color style of the image, and finally output a high-quality restored image.

[0032] The reconstructed feature map output from the decoder is processed by a 1×1 convolutional layer for channel integration and dimension transformation to generate a structurally complete preliminary repaired image. A binary mask is retrieved, and the original complete region and the repaired region are distinguished according to the inherent pixel region division rules of the mask. Pixel-level weighted fusion is performed on the pixels of the two types of regions. The pixel weight of the original complete region is set to 0.9, and the pixel weight of the repaired region is set to 0.8. The transition region of the repair boundary is linearly weighted. This weight setting can take into account the authenticity of the original image and the naturalness of the repaired region, avoid the abruptness of the repaired region, and ensure that the repaired region and the original region are naturally connected. The initial restored image undergoes global detail unification optimization. Gaussian filtering is used to suppress noise, blur distortion, and edge artifacts that are prone to occur during the restoration inference process, thereby improving image clarity. Color equalization processing is also used to unify the texture and color style of the entire image, ensuring the overall consistency of the restored image. The final output is a high-quality, complete restored image with natural edge transitions, unified texture details, clear and uniform image quality, and a coherent and harmonious overall visual performance.

[0033] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A lightweight image restoration method, characterized in that, include: S1. Input preprocessing: Input the damaged image and the size-matched binary mask, and concatenate the two in the channel dimension to obtain the input tensor; S2. Encoder Feature Extraction: Configure an encoder network consisting of multiple downsampling blocks to extract multi-scale features; The downsampling block contains the LSConv module, which improves the LSNet convolution, convolutional layers, normalization layers, and activation functions. The LSConv module, which can separate the convolutional structure according to the multi-branch depth, simultaneously captures global structural and local detailed features. After feature concatenation and channel adjustment, it combines the SE attention mechanism to complete feature fusion and aggregation. S3, Deep Feature Completion: Configure an improved contextual attention module on the deepest feature map of the encoder, normalize the features of known and missing regions and calculate the similarity matrix, generate attention weights after normalization, and combine the weights to weighted aggregate the features of known regions. S4. Decoder Feature Reconstruction: Configure a decoder network consisting of multiple upsampling blocks to restore the original resolution of the image through step-by-step upsampling operations; S5. Multi-scale output fusion: Convolutional operations are used to integrate feature channels to generate a preliminary repaired image. A binary mask is then used to complete pixel-level fusion and output a high-quality repaired image.

2. The lightweight image restoration method according to claim 1, characterized in that: In step S1, the pixel values ​​of the binary mask are only 0 or 1, where pixel value 1 corresponds to the damaged or missing area of ​​the image, and pixel value 0 corresponds to the complete area of ​​the image. The acquired damaged image and the pre-matched binary mask are aligned in size and calibrated dimensionally along the channel dimension. The images are then stitched together and processed along the channel dimension direction to form a uniform input feature tensor based on the multi-dimensional channel fusion method.

3. The lightweight image restoration method according to claim 1, characterized in that: In step S2, the downsampling block contains the LSConv module of the improved LSNet convolution, convolutional layers, normalization layers, and activation functions. The number of channels in the convolutional layers in the sampling block is the same as the number of channels in the input image, and the convolutional kernel stride is fixed at 2. The normalization layer specifically employs instance normalization, using a single damaged image as an independent processing sample to complete feature standardization correction, avoiding feature bias caused by batch statistics, and regularizing the feature distribution of damaged images; the activation function adopts the LeakyReLU activation function to perform nonlinear mapping transformation on the normalized feature data.

4. The lightweight image restoration method according to claim 1, characterized in that: In step S2, the LSConv module sets up three sets of parallel branch structures; Each branch employs convolutional methods with different receptive fields for multi-dimensional feature extraction. The first branch uses large-scale depthwise separable convolution to capture global contextual information and overall structural features of the image. After convolutional processing within the branch, batch normalization and non-linear activation are performed sequentially. The second branch uses small-scale depthwise separable convolution to extract local texture and detail features on the image surface, simultaneously performing batch normalization and non-linear activation operations. The third branch uses small-size standard convolution to uniformly adjust the dimensionality of the output features from multiple branches and to build residual connection channels. After extracting features at different scales from the three branches, features are uniformly spliced ​​and fused along the channel dimension. The number of feature channels is then adaptively adjusted through convolutional layers, and the feature weight parameters of each channel are dynamically calibrated using the SE attention mechanism to output multi-scale feature aggregation.

5. A lightweight image restoration method according to claim 1, characterized in that: In step S3, the multi-scale aggregated feature map obtained by aggregation is subjected to channel-by-channel feature normalization to eliminate the numerical differences between features at different scales. Then, the normalized features are divided into damaged area features and intact area features. The two types of features are matched one by one. By comparing the similarity of feature vectors, a similarity matrix representing the degree of correlation between damaged area and intact area features is calculated.

6. The lightweight image restoration method according to claim 1, characterized in that: In step S3, the feature similarity matrix is ​​globally normalized using a normalized exponential function, mapping the values ​​in the matrix to a specified range of values, and generating spatial attention weights for regional feature matching and content completion. Based on this attention weight, the effective features of the known complete region of the image are weighted and allocated, and then a weighted summation operation is performed on the allocated features. The target feature value obtained from the operation is then filled into the corresponding damaged and missing region of the image, thus completing the feature repair and filling of the missing region.

7. A lightweight image restoration method according to claim 1, characterized in that: In step S4, the decoder network performs stepwise upsampling reconstruction on the deep feature map after feature completion to restore and repair the original spatial resolution of the image. The decoder contains multiple upsampling blocks, each of which includes an upsampling unit, an LSConv module, a skip connection unit, an instance normalization layer, and a ReLU activation function. The upsampling unit uses nearest neighbor upsampling or transposed convolution to enlarge the feature map size and restore the spatial dimension.

8. A lightweight image restoration method according to claim 1, characterized in that: In step S4, the decoder uses a skip connection unit to establish a feature transmission path between the encoder and decoder. It performs deep splicing and fusion of the upsampled and amplified high-level semantic features with the shallow detail features retained by the corresponding level of the encoder. The LSConv module then extracts features from the fused hybrid features.

9. A lightweight image restoration method according to claim 1, characterized in that: In step S4, the decoder progressively amplifies the feature map resolution through each upsampling block. Each upsampling block first upsamples and increases the dimensionality of the output features from the previous level, and then fuses the shallow texture and deep structural features of the corresponding level of the encoder through skip connections. After the LSConv module refines and extracts the fused features, the features are then passed through instance normalization layers and ReLU activation functions to complete feature regularization and nonlinear mapping transformation. This iterative operation is performed step by step until the feature map size is restored to the original size of the input damaged image, thus completing the complete feature reconstruction.

10. A lightweight image restoration method according to claim 1, characterized in that: In step S5, the reconstructed features output by the decoder are integrated through channels to generate a preliminary restored image with a complete structure. Then, the binary mask from the input stage is retrieved, and the pixels of the original complete region and the pixels of the algorithm-restored region are weighted and fused at the pixel level according to the region division attributes of the mask. The restoration boundary is also smoothed. At the same time, global detail optimization is performed on the entire image to suppress noise and artifacts generated during the restoration process, unify the overall texture and color style of the image, and finally output a high-quality restored image.

Images