An image inpainting method, system and device based on a lightweight large receptive field convolution kernel and a storage medium

Abstract

The application discloses a kind of image restoration method, system, equipment and storage medium based on light weight big receptive field convolution kernel, belong to image restoration technical field, the method includes: to true value image is applied mask generation broken image, constructs training set and test set;Image restoration model based on generative adversarial network is constructed, including generator and discriminator;The generator is integrated with light weight big receptive field convolution unit and includes multiple trainable low-rank factor tensor, and tensor contraction is constructed combined convolution operation by the multiple low-rank factor tensor in the process of forward propagation;The model is trained, and the difference between reconstructed image and true value image is constrained by jointly minimizing loss function, and iteration is updated to convergence;Broken image is input into model, and output is repaired complete image.The application keeps the ability of big receptive field feature modeling while significantly reducing the parameter quantity of image restoration model, improves large-area repair quality, and considers performance and deployability.

Description

Technical Field

[0001] This invention relates to the field of image restoration technology, and more specifically to an image restoration method, system, device, and storage medium based on a lightweight, large receptive field convolutional kernel. Background Technology

[0002] Image restoration is an important task in the field of computer vision. Its goal is to reasonably fill in unknown areas using known area information when an image is occluded, missing, or damaged. It is widely used in scenarios such as old photo restoration, digital preservation of cultural relics, and image editing.

[0003] Image inpainting methods based on convolutional neural networks have become mainstream. These methods typically employ an encoder-decoder structure to extract image features and reconstruct missing regions, and combine strategies such as generative adversarial networks, perceptual loss, and multi-scale feature fusion to improve inpainting quality. However, in scenarios with large-area damage, the network not only needs to restore local textures but also needs to understand the overall structural layout of the image and the semantic correspondences between distant regions. Therefore, expanding the receptive field of the network and aggregating global contextual information have become key technical challenges.

[0004] In existing technologies, to enhance the ability to model long-distance context, one approach is to introduce an attention mechanism or Transformer structure into the convolutional network to achieve long-distance feature interaction by calculating the correlation weights between any positions on the feature map; another approach is to directly increase the receptive field by expanding the convolutional kernel size or using dilated convolution, thereby covering a larger spatial range in a single convolutional operation. However, the above-mentioned solutions all have the following problems to varying degrees: Attention-based or Transformer-based methods require additional construction of query, key-value mapping and calculation of relevance matrices. Their computational complexity increases quadratically with the feature map size, significantly increasing the number of parameters, memory consumption, and inference latency, making it difficult to meet the lightweight deployment requirements of mobile terminals, embedded devices, and real-time scenarios; While using large-size convolutional kernels can expand the receptive field, the number of parameters and computational cost increases quadratically with the kernel size, which is also not conducive to building lightweight models; Although ordinary small-size convolutional kernels have fewer parameters and are computationally efficient, their local receptive field is limited, and deep stacking is required to gradually expand the receptive field, making it difficult to fully capture semantic relationships between distant regions, resulting in problems such as structural discontinuity, texture inconsistency, and inaccurate semantic recovery when repairing large areas. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides an image inpainting method, system, device, and storage medium based on lightweight large receptive field convolutional kernels. This scheme designs lightweight large receptive field convolutional units, sets multiple trainable low-rank factor tensors, and performs tensor shrunking on these low-rank factor tensors according to preset connection relationships during forward propagation to construct combined convolution operations. These combined convolution operations can be combined with conventional sampling, dilated sampling, or deformable sampling methods, achieving large-scale feature modeling capabilities without setting complete large-size convolutional kernel weight parameters, thereby reducing the number of model parameters and computational complexity. Simultaneously, this invention combines multi-scale feature extraction and fusion mechanisms to collaboratively model local texture information, mesoscale structural information, and global semantic context information. Without relying on computationally expensive attention mechanisms, it improves the structural recovery capability and texture detail generation quality of large-area missing regions, enhancing the structural rationality, content consistency, and overall visual effect of the restored image, and improving the deployment capability of the image inpainting model in mobile terminals, edge computing devices, and practical engineering scenarios.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, embodiments of the present invention provide an image inpainting method based on a lightweight, large receptive field convolutional kernel, comprising the following steps: Constructing the training dataset: Obtain the original image dataset, apply masking processing to the ground truth images in the dataset to generate corresponding damaged images, and use the ground truth images and their corresponding damaged images as training sample pairs to construct the training set and the test set. Model Construction: An image restoration model is constructed based on a generative adversarial network architecture, including a generator and a discriminator. The generator integrates a lightweight large receptive field convolutional unit. The lightweight large receptive field convolutional unit realizes the large receptive field feature modeling of the large-size convolutional kernel through a lightweight reconstruction method. The lightweight reconstruction method includes: using multiple trainable low-rank factor tensors to factorize the parameters of the large-size convolutional kernel, and constructing a combined convolution operation by tensor shrinking between the multiple low-rank factor tensors during the forward propagation process. Model training: The image restoration model is adversarially trained using the training dataset. The difference between the reconstructed image and the ground truth image is constrained by jointly minimizing the loss function, and the model parameters are iteratively updated until the model converges. Model inference: After preprocessing the damaged image to be repaired, it is input into the trained image repair model, and after forward propagation, the repaired complete image is output.

[0008] Furthermore, the lightweight large receptive field convolutional unit determines the sampling position of the input feature according to a preset sampling method, and performs a combined convolution operation at the sampling position by tensor shrinking between multiple low-rank factor tensors; the preset sampling method is any one of the following: a conventional sampling method that determines the sampling position according to a regular grid, an expanded sampling method that determines the sampling position according to a preset expansion rate, or a deformable sampling method that adjusts the sampling position according to the learned offset; The method of constructing combined convolution operations based on low-rank factor tensors specifically includes: setting multiple trainable factor tensors, the number, order, dimension, and connection relationship of the multiple trainable factor tensors being determined according to the number of input channels, the number of output channels, the convolution kernel space size, and a preset rank parameter; the input channel dimension, output channel dimension, and convolution kernel space dimension are distributed in one or more of the multiple trainable factor tensors, at least two of the multiple trainable factor tensors have corresponding rank dimensions, and trainable factor tensors with connection relationships have the same dimension in the corresponding rank dimension; during forward propagation, the multiple trainable factor tensors are tensor shrunk along the corresponding rank dimension according to the preset connection relationship, and combined convolution operations are performed on the input features based on the tensor shrunk result.

[0009] Furthermore, the preset connection relationship of the multiple low-rank factor tensors includes any of the following: a chain connection relationship in which adjacent factor tensors are connected in sequence; a ring connection relationship in which the first and last factor tensors are connected in a closed loop; a core-factor connection relationship in which the core tensor is connected to multiple factor tensors along the corresponding rank dimension; a parallel connection relationship in which multiple rank-one components are set in parallel and the tensor shrinkage result of each rank-one component is summed; a block connection relationship in which multiple low-rank block terms are set in parallel and the tensor shrinkage result of each low-rank block term is summed; or a two-factor connection relationship in which two matrix factors are connected along a common rank dimension.

[0010] Furthermore, the generator's restoration process for the damaged image to be repaired includes, in sequence: input mapping processing, main feature processing, and output mapping processing. The input mapping process uses standard convolution to perform feature mapping on the damaged image to be repaired, resulting in shallow input features. The main feature processing encodes and decodes the shallow input features to obtain the main recovery features; The output mapping process uses standard convolution to map the main recovery features to the pixel domain, resulting in the final restored complete image.

[0011] Furthermore, the main feature processing includes large receptive field feature encoding processing and residual decoding processing: The large receptive field feature encoding process is executed by cascading multiple levels of the lightweight large receptive field convolutional units, extracting the structural features and contextual semantic features of the repaired image step by step. The residual decoding process is performed by stacking multiple layers of lightweight, large receptive field convolutional units to decode and recover the encoded structural and contextual features, thereby obtaining the main recovered features.

[0012] Furthermore, the generator also performs skip connection passing processing during the encoding and decoding process, passing the features from the encoding stage to the decoding stage; The skip connection transitive processing uses standard convolution to adjust the channels, scales, or feature maps of the features output from the encoding stage, and then fuses them with the features of the corresponding layer in the decoding stage.

[0013] Furthermore, the generator employs a backbone network based on a deep feature extraction and restoration architecture to perform image inpainting. The deep feature extraction and restoration architecture can be implemented in any of the following ways: encoder-decoder architecture, U-Net architecture, lightweight architecture based on ResNet or MobileNet, multi-stage coarse-to-fine inpainting architecture, or hybrid architecture of convolution and Transformer.

[0014] In a second aspect, embodiments of the present invention provide an image inpainting system based on a lightweight, large receptive field convolutional kernel, applying the method described in the first aspect, the system comprising: A training dataset construction module is used to obtain the original image dataset, apply masking processing to the ground value images in the dataset to generate corresponding damaged images, and use the ground value images and their corresponding damaged images as training sample pairs to construct training and test sets. A model building module is used to build an image restoration model. The image restoration model is based on a generative adversarial network architecture and includes a generator and a discriminator. The generator integrates a lightweight large receptive field convolutional unit. The lightweight large receptive field convolutional unit realizes the large receptive field feature modeling of the large-size convolutional kernel through a lightweight reconstruction method. The lightweight reconstruction method includes: using multiple trainable low-rank factor tensors to factorize the parameters of the large-size convolutional kernel, and constructing a combined convolution operation by tensor shrinking between the multiple low-rank factor tensors during the forward propagation process. The model training module is used to perform adversarial training on the image restoration model using the training dataset. It combines minimizing the loss function to constrain the difference between the reconstructed image and the ground truth image, and iteratively updates the model parameters until the model converges. The model inference module is used to preprocess the damaged image to be repaired, input it into the trained image repair model, and output the repaired complete image after forward propagation.

[0015] Thirdly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the image inpainting method based on lightweight large receptive field convolutional kernels described in the first aspect of the present invention.

[0016] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image inpainting method based on lightweight large receptive field convolution kernels described in the first aspect of the present invention.

[0017] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following: 1. Address the issues of large parameter count and unfavorable lightweight deployment in existing image restoration methods. Existing image inpainting methods based on attention mechanisms typically require additional attention computations to achieve long-distance feature interaction and global context modeling, leading to problems such as large model parameter count, high computational cost, and slow inference speed. This invention addresses this by employing a lightweight, large receptive field convolutional kernel reconstruction method. This achieves a larger spatial receptive field and long-distance context modeling without relying on complex attention modules, thus enabling superior image inpainting results with fewer parameters and lower computational complexity.

[0018] 2. It can improve the global structure modeling capability in large-area defect image restoration tasks. For large-area damaged regions, the restoration result depends not only on local texture information but also on the structural relationships and semantic consistency between distant regions. While some existing methods can enhance the ability to model long-distance dependencies, they are costly and struggle to balance performance and efficiency in lightweight scenarios. This invention utilizes lightweight, large receptive field convolutions to model feature information over a wider range, enabling the network to effectively capture the contextual relationships between distant regions in an image, thereby improving the structural restoration capability of large-area damaged regions.

[0019] 3. This invention can achieve superior repair performance under lightweight conditions. This invention not only effectively reduces model complexity, but also maintains strong competitiveness in terms of restoration quality. It achieves a good balance between lightweight model and high-performance image restoration, and has high application and promotion value.

[0020] 4. The technical solution of this invention has strong scalability and a wide range of applications. The lightweight, large receptive field modeling method adopted in this invention does not rely on specific types of attention structures or complex context modules, and has good versatility. It can be flexibly applied to various tasks according to different task requirements, including but not limited to image inpainting, image denoising, image super-resolution reconstruction, image deblurring, object detection, and semantic segmentation. Therefore, it has strong promotion and application value. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0022] Figure 1 This is a flowchart of an image inpainting method based on a lightweight large receptive field convolutional kernel provided in an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of the overall structure of the image restoration model provided in an embodiment of the present invention.

[0024] Figure 3 This is a qualitative comparison result diagram on the CelebA-HQ dataset provided in this embodiment of the invention.

[0025] Figure 4 This is a graph showing the comparison of model parameter count and performance on the CelebA-HQ dataset provided in this embodiment of the invention.

[0026] Figure 5 This is a graph showing the comparison of model parameter count and performance on the Places2 dataset provided in this embodiment of the invention.

[0027] Figure 6 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Example 1: like Figure 1 As shown, this invention discloses an image inpainting method based on lightweight large receptive field convolution kernels, comprising the following steps: Constructing the training dataset: Obtain the original image dataset, apply masking processing to the ground truth images in the dataset to generate corresponding damaged images, and use the ground truth images and their corresponding damaged images as training sample pairs to construct the training set and the test set; the masking processing adopts either irregular masking or regular masking to simulate image damage areas of different shapes and sizes.

[0030] Model Construction: An image restoration model is constructed based on a generative adversarial network architecture, including a generator and a discriminator. The generator integrates lightweight large receptive field convolutional units. These units achieve large receptive field feature modeling of large convolutional kernels through a lightweight reconstruction method. This lightweight reconstruction method includes: using multiple trainable low-rank factor tensors to factorize the parameters of the large convolutional kernels, and constructing combined convolution operations during forward propagation by tensor shrinking between these low-rank factor tensors. The generator preferably uses an encoder-decoder architecture to perform step-by-step feature extraction and restoration of the input image.

[0031] Model training: The image restoration model is adversarially trained using the training dataset. The difference between the reconstructed image and the ground truth image is constrained by jointly minimizing the loss function. The model parameters are iteratively updated until the model converges. The discriminator and generator alternately update their parameters to make the generated image approximate the distribution of the real image.

[0032] Model inference: After preprocessing the damaged image to be repaired, it is input into the trained image restoration model, and the restored complete image is output after forward propagation. The preprocessing includes size normalization and pixel value normalization to make the input image conform to the input format required by the image restoration model.

[0033] This invention is particularly suitable for tasks such as image inpainting, which have high requirements for both local detail restoration and global structural modeling. By employing a lightweight, large receptive field modeling method based primarily on convolutional operations, this invention can achieve superior image inpainting results without relying on high-overhead attention mechanisms or complex Transformer structures, especially improving the restoration capability and result quality for large-area defect regions or complex structural regions. Simultaneously, this invention also balances model lightweighting with engineering deployability, thereby enhancing the application value of related technologies in mobile devices, edge computing, and real-world engineering scenarios.

[0034] The image inpainting method based on lightweight large receptive field convolution kernels of the present invention will be described in detail below: Step S1: Construct the training dataset: Obtain publicly available or self-built image datasets, such as the CelebA-HQ face dataset and the Places2 scene dataset.

[0035] For each complete ground truth image in the dataset, a random or predefined irregular binary mask is generated and overlaid on the ground truth image to obtain the corresponding damaged image. The damaged image and the ground truth image are used as training sample pairs to construct the training set and the test set.

[0036] Step S2: Constructing an image restoration model: The image restoration model of this invention is based on a generative adversarial network architecture, which mainly includes a generator and a discriminator.

[0037] Step S21: Construct lightweight convolutional units with large receptive fields: In large-area image inpainting tasks, the restoration of missing regions often relies on effective modeling of long-range contextual information. One direct way to expand the receptive field is to use large-size convolutional kernels, but as the kernel size increases, the number of parameters and computational overhead also rises significantly.

[0038] Therefore, this invention no longer uses the traditional method of directly stacking large kernel convolutions to expand the receptive field. Instead, it represents the weights of the large kernel convolution as a low-rank core tensor chain in the form of Tensor Train (TT), and implicitly constructs the convolution kernel by tensor shrinkage. While effectively reducing the number of parameters and computational complexity, it still maintains a large effective receptive field and strong global dependency modeling ability, providing a more efficient modeling method for structural inference and semantic completion of large-scale missing regions.

[0039] make Represents the weight tensor of the convolutional layer, where and These represent the number of input channels and the number of output channels, respectively. This indicates the size of the convolution kernel.

[0040] To reduce the parameter surge introduced by large receptive field convolution while maintaining the increase in receptive field of the convolution kernel, the higher-order weight tensor is decomposed using TT decomposition, which can be represented as:

[0041] The dimensions of the kernel tensor are respectively , , , , Indicates the first i The kernel tensor and the first j The connection rank between kernel tensors.

[0042] During training, the core tensor The weights are learned directly, rather than through explicitly storing weight tensors. During forward propagation, an approximate convolutional kernel can be obtained through tensor shrinking operations.

[0043] In the formula, This indicates the shrinking operation of the kernel tensor. This indicates that the result will be reverse-engineered and transformed into a standard convolutional weight tensor.

[0044] Given input and reconstructed convolutional kernels The output of the convolutional layer The following can be calculated using standard two-dimensional convolution operations: .

[0045] The lightweight large receptive field convolutional unit determines the sampling position of the input feature according to a preset sampling method, and performs a combined convolution operation at the sampling position by tensor shrinking between multiple low-rank factor tensors; the preset sampling method is any of the following: a conventional sampling method that determines the sampling position according to a regular grid, an expanded sampling method that determines the sampling position according to a preset expansion rate, or a deformable sampling method that adjusts the sampling position according to the learned offset. The method for constructing combined convolution operations based on low-rank factor tensors includes: setting up multiple trainable factor tensors, the number, order, dimension, and connection relationship of which are determined according to the number of input channels, the number of output channels, the convolution kernel space size, and a preset rank parameter; the input channel dimension, output channel dimension, and convolution kernel space dimension are distributed in one or more of the multiple trainable factor tensors, at least two of which have corresponding rank dimensions, and the trainable factor tensors with connection relationships have the same dimension in the corresponding rank dimension; during forward propagation, the multiple trainable factor tensors are tensor-merged along the corresponding rank dimension according to the preset connection relationship, and combined convolution operations are performed on the input features based on the tensor-merging result; The preset connection relationships of multiple low-rank factor tensors include any of the following: a chain connection relationship in which adjacent factor tensors are connected in sequence; a ring connection relationship in which the first and last factor tensors are connected in a closed loop; a core-factor connection relationship in which the core tensor is connected to multiple factor tensors along the corresponding rank dimension; a parallel connection relationship in which multiple rank-one components are set in parallel and the tensor shrinkage result of each rank-one component is summed; a block connection relationship in which multiple low-rank block terms are set in parallel and the tensor shrinkage result of each low-rank block term is summed; or a two-factor connection relationship in which two matrix factors are connected along a common rank dimension.

[0046] Step S22: Construct the generator backbone network: like Figure 2As shown, the generator adopts an encoder-decoder architecture. To achieve effective modeling of long-distance contextual dependencies and global consistency in the repair of large-area missing regions, this invention combines the proposed lightweight large receptive field convolution operator, utilizing one with a larger receptive field, for example... k The encoder-decoder architecture is designed using convolutional kernels of 5, 6, and 7 to enhance the network's global feature modeling capabilities while reducing the model parameter size. To compensate for the loss of detail information and training instability caused by the proposed lightweight large receptive field convolutional operator, the key convolutional layers of the generator are redesigned to further improve the restoration quality.

[0047] The network backbone structure is not limited to a specific encoder-decoder form; it can adopt lightweight structures such as U-Net, ResNet, and MobileNet, multi-stage coarse-to-fine repair structures, or hybrid structures of convolution and Transformer.

[0048] First, to avoid unstable responses during network training caused by the proposed lightweight large receptive field convolution operator extracting input features, the method described in this application first uses standard... convolution Input Mapping to the feature space yields shallow input features. : .

[0049] Secondly, in order to effectively expand the receptive field of the network and improve the network's ability to model interactions with large-scale information at a low parameter cost, a large receptive field is designed using the lightweight large receptive field convolution operator proposed in this invention. k =7) Convolutional module, followed by a cascade of multiple lightweight large receptive field convolutional modules to construct a large receptive field feature encoding network. Shallow input features pass This yields coding features with stronger structural representation capabilities. : .

[0050] To further extract high-level semantic information from deep features and enhance structural recovery capabilities during the decoding stage, this invention combines the large receptive field modeling ability and efficient parameter representation advantages of the proposed lightweight large receptive field convolution to design a residual decoding network composed of multiple layers of lightweight large receptive field convolutions stacked together. Encode features Send in Obtain decoding features : .

[0051] Furthermore, in order to reduce the loss of shallow texture information during downsampling and deep transformation in the encoding stage, this invention designs a cross-layer feature transfer mechanism that combines convolutional mapping and skip connections. The corresponding layer features at the encoding end are transferred to the decoding end after being adjusted by standard convolution and fused with deep decoding features, so that the constructed network can simultaneously take into account global structure inference, deep semantic completion and local texture detail restoration.

[0052] Finally, this invention is based on standards. convolution As an output mapping, the backbone decoding features are... Mapping back to the pixel domain yields the final reconstruction result. : .

[0053] Step S23: Construct a dual-resolution discriminator: In the adversarial learning phase, to simultaneously ensure the consistency of the global image structure and the realism of local details, this invention uses a dual-resolution discriminator. For the output reconstructed image and their corresponding real images The high-resolution branch performs discrimination directly in the original resolution space to strengthen the constraints on texture details; the low-resolution branch constructs a low-resolution semantic representation by performing average pooling on the input to improve the discrimination ability of the overall structure. Finally, the two branches jointly output an adversarial loss, which is then compared with the generator's output. The alternating game of power generation gradually brings the generated results closer to the distribution of the real image, thereby effectively improving the naturalness and visual realism of the reconstruction results.

[0054] S3. Model Training and Loss Function Optimization: Loss function: The loss function can be selected from one or more of the following, such as reconstruction loss, perceptual loss, style loss, and adversarial loss, depending on the application scenario. The core is a loss function that achieves large receptive field context modeling through lightweight convolutional reconstruction; all of these can be considered equivalent alternatives to this invention.

[0055] In this embodiment, in order to obtain a visually realistic and semantically coherent repair result, the present invention utilizes reconstruction loss. and combat losses To minimize the reconstruction results With the corresponding truth value The differences between them.

[0056]

[0057] in, The parameters for weighing the losses during reconstruction.

[0058] Reconstruction loss: To measure the reconstruction outcome With the corresponding truth value Pixel-level differences between them, reconstruction loss The calculation is as follows: .

[0059] Adversarial Loss: To further improve the realism of the generated results, the method described in this application introduces adversarial training between the discriminator and the generator. The discriminator learns to distinguish between real samples and generated samples, thereby forcing the generator to produce outputs that are closer to the real distribution. Its adversarial loss can be expressed as: .

[0060] Step S4: Model Inference: The damaged image to be repaired undergoes the same preprocessing as the training set, i.e., it is adjusted to a uniform size. It is then input into the trained image restoration model generator, propagated forward once, and the output RGB image is the complete image after restoration.

[0061] The image inpainting method based on lightweight large receptive field convolution kernels of the present invention will be illustrated by the following embodiment: Suppose there is a 512×512 pixel color (3-channel) image with a 100×100 pixel white irregular missing area in the upper left corner.

[0062] S1 Data Preparation: Scale and crop the image to 256×256 pixels, and generate a corresponding binary mask. Missing regions are represented by 1, and known regions by 0. No additional ground truth values ​​are needed; no ground truth values ​​are available during inference.

[0063] S2 Model Building: Loads pre-trained model weights. This generator contains an input Std-Conv, an 8-level lightweight large receptive field convolutional encoder (K=7), an 8-level lightweight large receptive field convolutional decoder, skip connections with Std-Conv, and an output Std-Conv.

[0064] S3 model inference: Input a 256×256×3 damaged image tensor into the model. The encoder downsamples to extract deep semantics, the decoder upsamples to restore details, and edge textures are supplemented through skip connections.

[0065] S4 output: The model outputs a 256×256×3 tensor, in which the original missing areas have been filled in with pixels that conform to the overall scene and have natural textures. Scaling the tensor back to 512×512 yields the final repaired image.

[0066] Experimental details: 1. Datasets: The experiments were conducted on three different well-known public datasets. The datasets are divided as shown in Table 1.

[0067] Table 1. Training and test set partitioning on the three datasets.

[0068] During the training and testing of the network described in this application, all images and their corresponding masks were adjusted to 256×256 pixels. For the training phase, randomly generated irregular masks were used; while for the testing phase, the irregular mask dataset provided by Liu et al. was used.

[0069] 2. Comparison Methods: The method described in this application will be compared with mainstream methods such as CA (2018 CVPR), MC (2018 NIPS), EC (2019 ICCV), MEDFE (2020 ECCV), GDS (2021 CVPR), CTSDG (2021 ICCV), TFill (2022 CVPR), AOT (2023 TVCG), GraphFill (2024 WACV), PUT (2024 TPAMI), and FDM (2025 WACV).

[0070] Subjective and objective evaluation: 1. Qualitative comparison: like Figure 3 As shown, qualitative comparative experiments on different types of datasets demonstrate that the method proposed in this invention outperforms several state-of-the-art comparative methods in structural reconstruction. Figure 3 As shown, on the CelebA-HQ dataset, the method described in this application can more accurately complete the missing facial geometric relationships and symmetry in multiple sets of samples, and effectively suppress artifacts. Compared with CA, GMCNN, and GDS, the method described in this application overcomes the problems of unstable facial structure or obvious texture artifacts in large-area missing regions. Compared with EC, the method described in this application further improves the structural consistency and semantic coordination of local regions while maintaining the rationality of the geometric structure, such as inconsistent eye sizes. Compared with AOT and GraphFill, the method described in this application can more accurately recover key facial features and obtain clearer generation results under conditions of large local defects.

[0071] 2. Quantitative comparison: Compared with state-of-the-art benchmark methods, the method described in this application achieves the highest Structural Similarity Index Measure (SSIM) values ​​for all six mask ratios, and achieves the best Peak Signal-to-Noise Ratio (PSNR) and Learned Perceptual Image Patch Similarity (LPIPS) values ​​for most mask ratios.

[0072] 3. Parameter comparison: like Figure 4 and Figure 5 As shown, the method described in this application achieves superior objective evaluation metrics while maintaining a very small number of parameters (0.98M). In contrast, methods with similar model size or larger number of parameters have failed to achieve sustained improvements in reconstruction quality.

[0073] like Figure 4 As shown, a) compares the parameter count and average PSNR of the method described in this application and the comparative method on the CelebA-HQ dataset, and b) compares the parameter count and average SSIM of the method described in this application and the comparative method on the CelebA-HQ dataset. The method described in this application achieved the highest PSNR and SSIM (28.5137 dB / 0.9199). Specifically, the parameter count of the method described in this application is only about 1.63% of TFill (≈60M) and 1.03% of SF (≈95M), respectively, but its reconstruction performance is still significantly better than these two methods, indicating that the improvement in detail recovery and structural consistency of the method described in this application does not depend on the increase in the number of parameters.

[0074] like Figure 5 As shown, a) compares the parameter count and average PSNR of the method described in this application and the comparison method on the Places2 dataset, and b) compares the parameter count and average SSIM of the method described in this application and the comparison method on the Places2 dataset. The method described in this application also achieves the highest PSNR and SSIM (25.9024dB / 0.8615) with a very small parameter count. Compared with the second-best method MADF (≈85M), the parameter count of the method described in this application is only about 1.15% of it, but it still maintains the leading performance, further verifying the high parameter efficiency and cross-dataset robustness of the method described in this application in complex scenarios.

[0075] Example 2: Based on the same inventive concept, embodiments of the present invention also provide an image inpainting system based on a lightweight large receptive field convolutional kernel, the system comprising: A training dataset construction module is used to obtain the original image dataset, apply masking processing to the ground value images in the dataset to generate corresponding damaged images, and use the ground value images and their corresponding damaged images as training sample pairs to construct training and test sets. A model building module is used to build an image restoration model. The image restoration model is based on a generative adversarial network architecture and includes a generator and a discriminator. The generator integrates a lightweight large receptive field convolutional unit. The lightweight large receptive field convolutional unit realizes the large receptive field feature modeling of the large-size convolutional kernel through a lightweight reconstruction method. The lightweight reconstruction method includes: using multiple trainable low-rank factor tensors to factorize the parameters of the large-size convolutional kernel, and constructing a combined convolution operation by tensor shrinking between the multiple low-rank factor tensors during the forward propagation process. The model training module is used to perform adversarial training on the image restoration model using the training dataset. It combines minimizing the loss function to constrain the difference between the reconstructed image and the ground truth image, and iteratively updates the model parameters until the model converges. The model inference module is used to preprocess the damaged image to be repaired, input it into the trained image repair model, and output the repaired complete image after forward propagation.

[0076] Since these systems and the principles they address are similar to the aforementioned image inpainting methods based on lightweight large receptive field convolutional kernels, the implementation of this system can be found in the implementation of the aforementioned methods, and the repetitions will not be repeated.

[0077] Example 3: Based on the same inventive concept, the present invention also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When the processor executes a program stored in memory, it is able to implement the image inpainting method based on a lightweight large receptive field convolutional kernel as described in any one of Embodiments 1.

[0078] like Figure 6 As shown, the electronic device may include: a processor 10, a communication interface 20, a memory 30, and a communication bus 40, wherein the processor 10, the communication interface 20, and the memory 30 communicate with each other via the communication bus 40. The processor 10 can call logical instructions in the memory 30 to execute an image inpainting method based on a lightweight large receptive field convolutional kernel, the method including: Constructing the training dataset: Obtain the original image dataset, apply masking processing to the ground truth images in the dataset to generate corresponding damaged images, and use the ground truth images and their corresponding damaged images as training sample pairs to construct the training set and the test set. Model Construction: An image restoration model is constructed based on a generative adversarial network architecture, including a generator and a discriminator. The generator integrates a lightweight large receptive field convolutional unit. The lightweight large receptive field convolutional unit realizes the large receptive field feature modeling of the large-size convolutional kernel through a lightweight reconstruction method. The lightweight reconstruction method includes: using multiple trainable low-rank factor tensors to factorize the parameters of the large-size convolutional kernel, and constructing a combined convolution operation by tensor shrinking between the multiple low-rank factor tensors during the forward propagation process. Model training: The image restoration model is adversarially trained using the training dataset. The difference between the reconstructed image and the ground truth image is constrained by jointly minimizing the loss function, and the model parameters are iteratively updated until the model converges. Model inference: After preprocessing the damaged image to be repaired, it is input into the trained image repair model, and after forward propagation, the repaired complete image is output.

[0079] Example 4: This invention also provides a computer-readable storage medium containing a program for executing the image inpainting method based on lightweight large receptive field convolution kernels described in Embodiment 1 above. This program can be executed on a processor.

[0080] Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0081] The program stored on this medium is loaded into the processor's memory and executed to perform various functions. This storage medium, connected to hardware devices, enables the computer to perform the steps of Embodiment 1 described above.

[0082] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0083] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in this invention may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image inpainting method based on lightweight large receptive field convolution kernels, characterized in that, Includes the following steps: Constructing the training dataset: Obtain the original image dataset, apply masking processing to the ground truth images in the dataset to generate corresponding damaged images, and use the ground truth images and their corresponding damaged images as training sample pairs to construct the training set and the test set. Model Construction: An image restoration model is constructed based on a generative adversarial network architecture, including a generator and a discriminator. The generator integrates a lightweight large receptive field convolutional unit. The lightweight large receptive field convolutional unit realizes the large receptive field feature modeling of the large-size convolutional kernel through a lightweight reconstruction method. The lightweight reconstruction method includes: using multiple trainable low-rank factor tensors to factorize the parameters of the large-size convolutional kernel, and constructing a combined convolution operation by tensor shrinking between the multiple low-rank factor tensors during the forward propagation process. Model training: The image restoration model is adversarially trained using the training dataset. The difference between the reconstructed image and the ground truth image is constrained by jointly minimizing the loss function, and the model parameters are iteratively updated until the model converges. Model inference: After preprocessing the damaged image to be repaired, it is input into the trained image repair model, and after forward propagation, the repaired complete image is output.

2. The method as described in claim 1, characterized in that, The lightweight large receptive field convolutional unit determines the sampling position of the input feature according to a preset sampling method, and performs a combined convolution operation at the sampling position by tensor shrinking between multiple low-rank factor tensors; the preset sampling method is any one of the following: a conventional sampling method that determines the sampling position according to a regular grid, an expanded sampling method that determines the sampling position according to a preset expansion rate, or a deformable sampling method that adjusts the sampling position according to the learned offset. The method of constructing combined convolution operations based on low-rank factor tensors specifically includes: setting multiple trainable factor tensors, wherein the number, order, dimension, and connection relationship of the multiple trainable factor tensors are determined according to the number of input channels, the number of output channels, the convolution kernel space size, and a preset rank parameter; the input channel dimension, the output channel dimension, and the convolution kernel space dimension are distributed in one or more of the multiple trainable factor tensors, at least two of the multiple trainable factor tensors have corresponding rank dimensions, and the trainable factor tensors with connection relationships have the same dimension in the corresponding rank dimension; During the forward propagation process, the tensors of the multiple trainable factors are shrunk along the corresponding rank dimension according to the preset connection relationship, and the input features are subjected to combined convolution operation based on the tensor shrunk result.

3. The method as described in claim 2, characterized in that, The preset connection relationships of the multiple low-rank factor tensors include any of the following: a chain connection relationship in which adjacent factor tensors are connected in sequence; a ring connection relationship in which the first and last factor tensors are connected in a closed loop; a core-factor connection relationship in which the core tensor is connected to multiple factor tensors along the corresponding rank dimension; a parallel connection relationship in which multiple rank-one components are set in parallel and the tensor shrinkage result of each rank-one component is summed; a block connection relationship in which multiple low-rank block terms are set in parallel and the tensor shrinkage result of each low-rank block term is summed; or a two-factor connection relationship in which two matrix factors are connected along a common rank dimension.

4. The method as described in claim 1, characterized in that, The generator's repair processing of the damaged image to be repaired includes, in sequence: input mapping processing, main feature processing, and output mapping processing. The input mapping process uses standard convolution to perform feature mapping on the damaged image to be repaired, resulting in shallow input features. The main feature processing encodes and decodes the shallow input features to obtain the main recovery features; The output mapping process uses standard convolution to map the main recovery features to the pixel domain, resulting in the final restored complete image.

5. The method as described in claim 4, characterized in that, The main feature processing includes large receptive field feature encoding processing and residual decoding processing: The large receptive field feature encoding process is executed by cascading multiple levels of the lightweight large receptive field convolutional units, extracting the structural features and contextual semantic features of the repaired image step by step. The residual decoding process is performed by stacking multiple layers of lightweight, large receptive field convolutional units to decode and recover the encoded structural and contextual features, thereby obtaining the main recovered features.

6. The method as described in claim 1, characterized in that, The generator also performs skip connection passing processing during the encoding and decoding process, passing the features from the encoding stage to the decoding stage; The skip connection transitive processing uses standard convolution to adjust the channels, scales, or feature maps of the features output from the encoding stage, and then fuses them with the features of the corresponding layer in the decoding stage.

7. The method as described in claim 1, characterized in that, The generator uses a backbone network based on a deep feature extraction and restoration architecture to perform image restoration. The deep feature extraction and restoration architecture can be implemented in any of the following ways: encoder-decoder architecture, U-Net architecture, lightweight architecture based on ResNet or MobileNet, multi-stage coarse-to-fine restoration architecture, or hybrid architecture of convolution and Transformer.

8. An image inpainting system based on lightweight large receptive field convolution kernels, characterized in that, The system comprises: A training dataset construction module is used to obtain the original image dataset, apply masking processing to the ground value images in the dataset to generate corresponding damaged images, and use the ground value images and their corresponding damaged images as training sample pairs to construct training and test sets. A model building module is used to build an image restoration model. The image restoration model is based on a generative adversarial network architecture and includes a generator and a discriminator. The generator integrates a lightweight large receptive field convolutional unit. The lightweight large receptive field convolutional unit realizes the large receptive field feature modeling of the large-size convolutional kernel through a lightweight reconstruction method. The lightweight reconstruction method includes: using multiple trainable low-rank factor tensors to factorize the parameters of the large-size convolutional kernel, and constructing a combined convolution operation by tensor shrinking between the multiple low-rank factor tensors during the forward propagation process. The model training module is used to perform adversarial training on the image restoration model using the training dataset. It combines minimizing the loss function to constrain the difference between the reconstructed image and the ground truth image, and iteratively updates the model parameters until the model converges. The model inference module is used to preprocess the damaged image to be repaired, input it into the trained image repair model, and output the repaired complete image after forward propagation.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image inpainting method based on lightweight large receptive field convolution kernels as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the image inpainting method based on lightweight large receptive field convolution kernels as described in any one of claims 1 to 7.

Images