Image restoration method and system based on semi-supervised semantic feedback correction

Abstract

The invention discloses an image restoration method and system based on semi-supervised semantic feedback correction. The method comprises the following steps: firstly, processing a to-be-restored image with a mask through an initial restoration module to generate a coarse restoration image; performing semantic segmentation on the coarse repair image by adopting a repair image semantic segmentation model optimized by a semi-supervised semantic feedback correction module to generate a corrected semantic graph; and finally, performing fine repair on the generated coarse repair image by using a fine repair module to generate a final high-quality repair image. According to the method, an interactive feedback closed loop between image restoration and semantic segmentation is established through an original three-stage framework of'initial restoration-semi-supervised semantic feedback correction-fine restoration '. The framework can actively identify and correct semantic deviation generated in an initial repair stage, effectively prevent error propagation and accumulation, ensure reasonability of a final repair result in semantic logic and coordination of a final repair result in structure, and remarkably improve repair quality when a complex scene and a large defect area are processed.

Description

Technical Field

[0001] This invention belongs to the field of digital image restoration, specifically relating to an image restoration method and system based on semi-supervised semantic feedback correction. Background Technology

[0002] The core objective of image inpainting is to fill in missing regions in an image, resulting in a visually realistic image with coherent semantic content. This technique has performed exceptionally well in various applications, including object removal, visual photo editing, and the restoration of damaged images. Recent research has trended towards utilizing semantic structure (such as semantic segmentation) as guidance for image inpainting tasks. By providing semantic structural information about the missing regions, this approach offers prior knowledge for inferring the texture of missing images. This allows for better consistency with the known semantic structure when restoring the texture of missing regions.

[0003] One of the main challenges of semantically guided image inpainting is the difficulty in obtaining accurate and complete semantic structure guidance before beginning the restoration of image texture information. To address this, single-stage guided methods typically employ a cascaded architecture, first generating a semantic map of the complete semantic structure, and then using this map to guide the inpainting process. However, this single-stage guided mechanism lacks a feedback loop, meaning that errors in the initial semantic prediction stage are directly propagated to subsequent inpainting steps, leading to a gradual accumulation of errors. Furthermore, progressive guided methods employ hierarchical semantic prediction to achieve dynamic correction, and their core design usually involves establishing an iterative optimization mechanism between semantic prediction and texture inpainting. However, if semantic errors exist in the texture information, this dependency can exacerbate these errors, affecting the final inpainting outcome. Summary of the Invention

[0004] This invention addresses the lack of a feedback mechanism between restoration and semantics in single-semantic guided image restoration and the problem that progressive semantic guided image restoration amplifies potential semantic biases in the initial restoration. It provides an image restoration method and system based on semi-supervised semantic feedback correction, which establishes an interactive feedback mechanism between image restoration and semantic segmentation by combining semi-supervised learning and semantic consistency criteria. This effectively corrects semantic biases while reducing the cost of manual annotation.

[0005] Firstly, embodiments of this application provide an image inpainting method based on semi-supervised semantic feedback correction. This method achieves bidirectional interaction and collaborative optimization between image inpainting and semantic segmentation through a three-stage integrated process comprising initial inpainting, semi-supervised semantic feedback correction, and fine-tuning. Specifically, the technical solution provided by this invention is as follows:

[0006] Step 1: The initial repair module processes the masked image to be repaired to generate a coarse repaired image. The initial repair module adopts a generative model architecture, which inputs the masked image to be repaired into the generative network of the initial repair module to generate a coarse repaired image.

[0007] Step 2: Use the semantic segmentation model of the repaired image optimized by the semi-supervised semantic feedback correction module to perform semantic segmentation on the generated coarse repaired image to generate a corrected semantic map.

[0008] The semi-supervised semantic feedback correction module establishes a feedback loop connecting image restoration and semantic segmentation. This feedback loop is used to optimize a final image restoration semantic segmentation model through collaborative training.

[0009] The process of establishing and utilizing the feedback loop includes: after supervised initial training of the original image semantic segmentation model and the repaired image semantic segmentation model (both using existing semantic segmentation models with the same structure), generating pseudo-labels for unlabeled images using the original image semantic segmentation model, and performing quality evaluation and screening of the pseudo-labels based on a preset semantic consistency criterion; adopting a differentiated optimization strategy for the pseudo-labeled images according to the evaluation results, that is, using high-quality pseudo-labels to perform joint semi-supervised optimization of the two semantic segmentation models, while placing the images corresponding to low-quality pseudo-labels in an iterative refinement process, using the optimized original image semantic segmentation model to generate new pseudo-labels for the final convergence training of the repaired image semantic segmentation model.

[0010] Step 3: Based on the corrected semantic map, a fine restoration module is applied to refine the coarse restoration image generated in Step 1, thereby generating the final high-quality restored image. The fine restoration module receives the coarse restoration image and the corrected semantic map, and optimizes the coarse restoration image according to the guidance of the corrected semantic map.

[0011] In one possible implementation, the generative model architecture used by the initial repair module is a Generative Adversarial Network (GAN), which includes a generator network (as a generator) and a discriminator; the specific operation is as follows:

[0012] Given an image to be repaired and a corresponding binary mask, the initial repair module uses this information to generate a coarse repaired image. The generator employs an encoder-decoder structure. During the generation process, pixels in the missing regions are filled with features learned from the image by the encoder-decoder structure. The encoder extracts global and local features from the image to be repaired, while the decoder uses these features to reconstruct the pixels in the missing regions.

[0013] In one possible implementation, the semi-supervised semantic feedback correction module operates as follows:

[0014] First, the task dataset is obtained according to the requirements. This dataset is divided into a "labeled image set" containing real semantic labels and an "unlabeled image set". The labeled and unlabeled image sets are derived from the same data distribution in terms of content and style to ensure the effectiveness of subsequent model training.

[0015] (1) Supervised initial training

[0016] 2-1. Using the labeled image set with real semantic labels, supervised training is performed on the original image semantic segmentation model. The original image semantic segmentation model adopts an existing semantic segmentation model.

[0017] 2-2. Apply a random mask to the labeled image set with real semantic labels, and process it using the initial repair module in step 1 to generate a coarse repair result; then use the coarse repair result and its corresponding real semantic labels to conduct supervised training on the repaired image semantic segmentation model, the structure of the repaired image segmentation model being consistent with the original image semantic segmentation model.

[0018] (2) Pseudo-tag generation and filtering

[0019] 2-3. Using the unlabeled image set, input it into the original image semantic segmentation model trained in step 2-1 to generate pseudo-labels. Subsequently, the generated pseudo-labels are corrected based on a preset semantic consistency criterion, and a qualified, high-quality subset of pseudo-labeled images is selected.

[0020] (3) Semi-supervised iterative optimization

[0021] 2-4. Merge the labeled images with real semantic labels from step 2-1 with the high-quality qualified pseudo-labeled images selected from step 2-3 to form an enhanced training set. Use this enhanced training set to retrain the original image semantic segmentation model and the repaired image segmentation model to complete one round of iterative optimization.

[0022] 2-5. For the low-quality pseudo-labeled images that failed the screening in step 2-3, new pseudo-labels are regenerated using the optimized original image semantic segmentation model from step 2-4. These new pseudo-labels are then combined with the qualified pseudo-labeled image subset from step 2-3 to form an updated, complete pseudo-labeled image set. The repair image semantic segmentation model is further trained using both the labeled images with real semantic labels and the pseudo-labeled image set until the model performance converges. Finally, the fully optimized repair image semantic segmentation model can perform semantic segmentation on the coarsely repaired image to complete semantic correction.

[0023] In one possible implementation, the fine-tuning module employs a generative network integrating a "semantic-guided normalization module." Here, the "semantic-guided normalization module" is defined as a processing unit capable of using the corrected semantic map generated in the preceding steps as high-level structured guidance information, and adaptively modulating the feature maps of one or more intermediate layers in the generative network accordingly. Its core mechanism lies in the fact that the "semantic-guided normalization module" does not use globally uniform modulation parameters, but rather dynamically generates modulation parameters for different units (e.g., different spatial locations, feature channels, or combinations thereof) on the feature map based on the input semantic map, ultimately achieving final optimization of the image's structure, texture, and details under the guidance of precise semantic layout.

[0024] In one possible implementation, the specific structure of the fine repair module is as follows:

[0025] The fine-tuning module utilizes the corrected, precise semantic information to guide the final image content generation. Building upon semantic labels, the fine-tuning module extends the generator network of the initial inpainting module by integrating a Spatially-Adaptive Denormalization (SPADE) module as a semantically guided normalization module. The SPADE module receives the feature map output from the corresponding intermediate layer of the generator network in the fine-tuning module and the sampled corrected semantic map, and dynamically generates affine transformation parameters (scaling factors) based on the semantic information at each spatial location. and offset The SPADE module normalizes the feature map. It can dynamically adjust the feature map of the generator network based on the semantic information of each pixel position, making the details of the generated image more consistent with the semantic labels of the input, enhancing the semantic consistency of the generator network, and improving the quality of the final restored image.

[0026] Secondly, embodiments of this application provide an image inpainting system based on semi-supervised semantic feedback correction, the system comprising:

[0027] Coarse Repair Module: The initial repair module processes the masked image to be repaired to generate a coarse repaired image. The initial repair module adopts a generative model architecture, inputting the masked image to be repaired into the generative network of the initial repair module to generate the coarse repaired image.

[0028] Semi-supervised semantic correction module: The semantic segmentation model of the repaired image optimized by the semi-supervised semantic feedback correction module is used to perform semantic segmentation on the generated coarse repaired image to generate a corrected semantic map.

[0029] Fine-tuning module: Based on the semantic map corrected by the semantic correction module, a fine-tuning module is applied to refine the coarse-tuned image generated by the coarse-tuning module, thereby generating a final high-quality restored image. The fine-tuning module receives the coarse-tuned image and the corrected semantic map, and optimizes the coarse-tuned image according to the guidance of the corrected semantic map.

[0030] In one possible implementation, the semi-supervised semantic feedback correction module establishes a feedback loop connecting image inpainting and semantic segmentation, and uses this feedback loop to optimize a final inpainted image semantic segmentation model through co-training.

[0031] The process of establishing and utilizing the feedback loop includes: after supervised initial training of the original image semantic segmentation model and the repaired image semantic segmentation model (both using existing semantic segmentation models with the same structure), generating pseudo-labels for unlabeled images using the original image semantic segmentation model, and performing quality evaluation and screening of the pseudo-labels based on a preset semantic consistency criterion; adopting a differentiated optimization strategy for the pseudo-labeled images according to the evaluation results, that is, using high-quality pseudo-labels to perform joint semi-supervised optimization of the two semantic segmentation models, while placing the images corresponding to low-quality pseudo-labels in an iterative refinement process, using the optimized original image semantic segmentation model to generate new pseudo-labels for the final convergence training of the repaired image semantic segmentation model.

[0032] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory;

[0033] The memory is used to store computer programs.

[0034] When the processor executes the program stored in the memory, it implements any of the image restoration methods described in this application.

[0035] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the image restoration methods described in this application.

[0036] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the image restoration methods described in this application.

[0037] The beneficial effects of this invention are as follows:

[0038] To address the issues of error accumulation in single-stage guidance methods and the potential amplification of initial semantic errors in progressive guidance methods, this invention establishes an interactive feedback loop between image inpainting and semantic segmentation through a unique three-stage architecture: "initial inpainting - semi-supervised semantic feedback correction - fine inpainting." This architecture proactively identifies and corrects semantic deviations generated in the initial inpainting stage, effectively preventing the propagation and accumulation of errors. It ensures the semantic logical rationality and structural consistency of the final inpainting result, significantly improving the inpainting quality when handling complex scenes and large defect areas.

[0039] This invention innovatively integrates a semantic-guided normalization module (such as SPADE) into the fine-tuning stage, establishing a technical pathway to efficiently couple the structured information of the corrected semantic map to the image texture generation process. This module adaptively injects high-level semantic information directly into and modulates the intermediate layer feature maps of the generative network, achieving fine-grained control over image content generation. This design ensures that the accurate semantic layout obtained from the second-stage correction can be faithfully converted into the final visual details, achieving optimal results in terms of structure, texture, and boundary clarity, perfectly solving the key technical problem of how to fully utilize the corrected semantic information.

[0040] This invention employs a semi-supervised learning framework, which can effectively utilize a large amount of unlabeled data to optimize and generalize semantic segmentation models. This greatly reduces the reliance on expensive and time-consuming manually labeled data, lowers the training cost of the algorithm, and enhances the practicality and scalability of the method.

[0041] The optimized semantic segmentation model trained by this invention is specifically optimized for the characteristics of "restored images". Compared with the use of general segmentation models, it can better understand and handle problems such as blurring, artifacts and discontinuities that may occur during the restoration process, and shows stronger task adaptability and robustness. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the overall process of the image restoration method based on semi-supervised semantic feedback correction according to an embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of the network architecture generated by the initial repair module in an embodiment of the present invention.

[0044] Figure 3 This is a schematic diagram of the semi-supervised semantic feedback correction module according to an embodiment of the present invention.

[0045] Figure 4 This is a schematic diagram illustrating the concept of pseudo-label correction based on cross-image consistency in an embodiment of the present invention.

[0046] Figure 5This is a schematic diagram of the network architecture generated by the fine repair module in an embodiment of the present invention.

[0047] Figure 6 The images show a comparison of image restoration results obtained by using the method of this invention and those obtained by using a semi-inventive method. Detailed Implementation

[0048] The invention will be further described below with reference to the accompanying drawings.

[0049] In view of the shortcomings of existing technologies, this invention provides an image inpainting method based on semi-supervised semantic feedback correction. This method addresses potential semantic biases in semantically guided methods by using semantic consistency constraints for semantic correction, and then selects reliable samples to feed back to the semantic segmentation model for re-correction, forming a closed loop of "generation-selection-re-optimization".

[0050] This application provides an image inpainting method based on semi-supervised semantic feedback correction. The specific scheme is as follows, and a schematic diagram is shown. Figure 1 As shown:

[0051] Step 1: The initial repair module processes the masked image to be repaired to generate a coarse repaired image. The initial repair module adopts a generative model architecture. The masked image to be repaired is input into the generative network of the initial repair module to generate the coarse repaired image.

[0052] Step 2: Use the semantic segmentation model of the repaired image optimized by the semi-supervised semantic feedback correction module to perform semantic segmentation on the coarse repaired image to generate a corrected semantic map.

[0053] The core innovation of the semi-supervised semantic feedback correction module lies in establishing a feedback loop connecting image restoration and semantic segmentation. This feedback loop is used to optimize a final image restoration semantic segmentation model through a collaborative training approach.

[0054] The process of establishing and utilizing the feedback loop includes: after supervised initial training of the original image semantic segmentation model and the repaired image semantic segmentation model (both using existing semantic segmentation models with the same structure), generating pseudo-labels for unlabeled images using the original image semantic segmentation model, and performing quality evaluation and screening of the pseudo-labels based on a preset semantic consistency criterion; adopting a differentiated optimization strategy for the pseudo-labeled images according to the evaluation results, that is, using high-quality pseudo-labels to perform joint semi-supervised optimization of the two semantic segmentation models, while placing the images corresponding to low-quality pseudo-labels in an iterative refinement process, using the optimized original image semantic segmentation model to generate new pseudo-labels for the final convergence training of the repaired image semantic segmentation model.

[0055] Step 3: Based on the corrected semantic map, apply a fine repair module to perform fine repair on the coarse repair image generated in Step 1 to generate the final high-quality repair image; the fine repair module is used to receive the coarse repair image and the corrected semantic map, and optimize the coarse repair image according to the guidance of the corrected semantic map.

[0056] Any other type of generative network capable of receiving a masked image and outputting preliminary repair results, such as, but not limited to, standard encoder-decoder networks, generators in generative adversarial networks (GANs), or variational autoencoders (VAEs), can be used as specific implementations of the initial repair module of this invention. In a preferred embodiment of this invention, the initial repair module is specifically implemented using a generative adversarial network (GAN) architecture, and the generator network of the initial repair module adopts a U-Net-based encoder-decoder architecture. The initial repair module is specifically as follows:

[0057] The core objective of this module is to perform image inpainting tasks using preliminary pixel content generated for the missing regions. Given an image to be inpainted... and the corresponding binary mask (For example, where pixel value 1 represents the area to be repaired, and pixel value 0 represents the known area), the initial repair module uses the above information to generate a coarse repair image. .like Figure 2 As shown, the generator network of the initial inpainting module adopts a U-Net-based encoder-decoder structure. The encoder extracts multi-scale features of the image through multi-layer downsampling convolutions, while the decoder fuses these features through upsampling and skip connections to reconstruct the image. The discriminator can adopt a PatchGAN structure to distinguish between real and fake local patches in the inpainted image. Through adversarial training and pixel-level loss (such as L1 loss) optimization, the initial inpainting module can generate a coarsely inpainted image containing basic structure and texture. The main task of the initial inpainting is to restore missing pixels and reconstruct basic texture information. Although the inpainting at this stage is not perfect, it provides an important foundation for subsequent semi-supervised semantic correction and fine inpainting.

[0058] In one possible implementation, the semantic segmentation model of the repaired image, optimized by a semi-supervised semantic feedback correction module, is used to process the coarsely repaired image. The system identifies and corrects potential semantic errors. The specific process of the semi-supervised semantic feedback correction module is as follows: Figure 3 As shown, it includes the following steps:

[0059] First, to execute the semi-supervised learning process of this invention, a task dataset needs to be obtained according to requirements. This dataset is divided into a "labeled image set" containing real semantic labels and an "unlabeled image set" without labels. The labeled and unlabeled image sets originate from the same data distribution in terms of content and style to ensure the effectiveness of subsequent model training.

[0060] (1) Supervised training

[0061] 2-1. Using the labeled image set with real semantic labels, a source image semantic segmentation model is trained in a supervised manner. The source image semantic segmentation model adopts an existing semantic segmentation model, including but not limited to FCN, SegNet, DeepLabv3+, etc.

[0062] 2-2. Apply a random mask to the labeled image set with real semantic labels, and process it using the initial repair module in step 1 to generate a coarse repair result; then use the coarse repair result and its corresponding real semantic labels to conduct supervised training on a repaired image semantic segmentation model, the structure of which is consistent with the original image semantic segmentation model.

[0063] (2) Pseudo-tag generation and filtering

[0064] 2-3. Using the unlabeled image set, input it into the original image semantic segmentation model trained in step 2-1. Pseudo-labels are generated. Subsequently, the generated pseudo-labels are corrected based on a preset semantic consistency criterion (using a cross-image consistency method in this case), and a subset of qualified, high-quality pseudo-labeled images is selected.

[0065] (3) Semi-supervised iterative optimization

[0066] 2-4. Merge the labeled images with real semantic labels from step 2-1 with the high-quality, qualified pseudo-labeled images selected from step 2-3 to form an enhanced training set. Use this enhanced training set to simultaneously improve the semantic segmentation model of the original image. With repair image segmentation model Retraining yielded the following results: and This is to complete one round of iterative optimization.

[0067] 2-5. For the low-quality pseudo-labeled images that failed the screening in step 2-3, the optimized original image semantic segmentation model from step 2-4 can be used. Regenerate pseudo-labels for the images and combine them with the qualified pseudo-labeled image subset from steps 2-3 to form an updated and complete pseudo-labeled image set. Use the labeled images with real semantic labels and this pseudo-labeled image set together to improve the image semantic segmentation model. Further training will be required. This process continues until the model performance converges. The final result is a fully optimized inpainted image semantic segmentation model. It can then be used for subsequent semantic correction tasks.

[0068] In one possible implementation, the "preset semantic consistency criterion" mentioned in steps 2-3 is implemented as a comprehensive evaluation and correction mechanism based on cross-image semantic consistency. Its specific implementation is as follows:

[0069] First, construct the "category anchor vector" as the basis for all subsequent evaluations: traverse the labeled image set with real labels, extract pixel-level deep features belonging to each semantic category, and average all features belonging to the same category to calculate a "category anchor vector" for each semantic category as a global, standardized feature benchmark.

[0070] Secondly, the aforementioned "category anchor vector" is applied to perform the two processes of pseudo-label correction and filtering:

[0071] I. Pixel-level correction process for pseudo-tags:

[0072] 1.1) Selection of Labeled Reference Set: This step aims to select the highest quality reference examples for the subsequent correction process. The process is as follows: For each semantic category, the reliability of all labeled images containing that category is independently calculated (defined as a predefined loss value between the image features and the corresponding "category anchor vector." This loss function can be specifically implemented as Binary Cross-Entropy Loss, or it can be replaced by other known functions in the field used to measure similarity or distance (e.g., L1 or L2 distance). The smaller the loss, the more reliable the image). Subsequently, the reliability of each category is sorted from high to low, and images with the highest reliability from the sorted results are selected according to a predetermined proportion or number (e.g., the top 50%) to form the "optimal reference set" for that category.

[0073] 1.2) Pixel-level Correction of Pseudo-labels: This step uses the previously selected reference set to correct the initial pseudo-labels of the unlabeled image. The process is as follows: For an unlabeled image to be corrected, firstly, a semantic category in its initial pseudo-label is obtained; then, features are extracted from the "optimal reference set" of that category, and a more locally representative "category support vector" is constructed by performing local-specific weighting and fusion on the high-level features of these reference set images (in a specific implementation, this support vector can be generated by performing element-wise multiplication (Hadamard Product) on the high-level features of the reference set images and their corresponding category masks); next, a cross-image semantic consistency map is generated by calculating the pixel-level cosine similarity between the pixel features of the image to be corrected and the "category support vector"; finally, this map serves as a highly reliable confidence map to confirm or adjust the pixel attribution of the corresponding category in the initial pseudo-label, thereby completing the correction of the pseudo-labels.

[0074] II. The process of selecting unlabeled image sets for semi-supervised learning:

[0075] 2.1) Calculation of overall image reliability: For an unlabeled image, its overall reliability is defined as a cumulative loss value. This value is calculated as follows: First, obtain all semantic categories existing in the image pseudo-label; second, for each existing semantic category, independently calculate the loss value between its image features and the corresponding "category anchor vector"; finally, sum the loss values ​​of all categories calculated for the image to obtain a single scalar value representing the overall quality of the image pseudo-label.

[0076] 2.2) Sorting and partitioning: Based on the calculated reliability scalar values ​​of all unlabeled images, the unlabeled image set is sorted in ascending order, and the sorted image set is divided into "reliable set" and "unreliable set" by a preset partitioning criterion (e.g., dividing it in half at a fixed ratio, or setting an absolute loss threshold) for subsequent phased optimization training.

[0077] Finally, those skilled in the art will understand that the cross-image semantic consistency strategy described in detail above is merely a preferred example for implementing the "pseudo-label evaluation and correction" function of this invention. Any other strategy capable of evaluating and optimizing pseudo-label quality based on consistency criteria between or within images falls within the scope of protection claimed by this invention.

[0078] In a preferred embodiment, the fine repair module employs a generator network integrating a Spatial Adaptive Normalization (SPADE) module, i.e., a fine repair generator. To effectively optimize and train the generator network, the fine repair module further includes a fine repair discriminator. The network structure, function, and training method of the fine repair discriminator can be the same as or substantially the same as the discriminator of the initial repair module described in step 1.

[0079] The following will be combined with the appendix Figure 5 The internal working mechanism of the fine repair generator, the core component of the fine repair module, is explained in detail:

[0080] The fine restoration generator receives the coarse restoration image generated in step 1 as its backbone input. One or more intermediate layers within its network (e.g., in the upsampling path of the decoder) integrate a Spatial Adaptive Normalization (SPADE) module. Figure 5 The SPADE module (represented as the semantic propagation function module) serves as the semantic guidance normalization module. During the forward propagation of the network, the SPADE module, located in a specific intermediate layer of the network, is configured to receive two inputs: one is the feature map to be processed generated by the intermediate layer itself; the other is the corrected semantic map after being downsampled according to the corresponding spatial dimension to match the size of the feature map to be processed. Based on the corrected semantic map it receives, the SPADE module dynamically generates affine transformation parameters (scaling factors) for each spatial location in the feature map to be processed. and offset The system then uses these parameters to normalize and modulate the feature map to be processed. By deploying the SPADE module in the intermediate layer of the network and performing the above operations, the precise structure and layout guidance from the semantic map is effectively and on demand applied to the image reconstruction process. This makes the details of the generated image more consistent with the input semantic labels, enhances the semantic consistency of the generative model, and improves the final restored image. The quality.

[0081] A qualitative comparison of image inpainting detection results based on semi-supervised semantic feedback correction and results obtained without the method of this invention, for example... Figure 6 As shown in the figure, it is clear that the advantages of this invention are more prominent when dealing with challenging scenarios such as distant buildings (first and fourth rows) and object boundaries (walls and roads in the third row). Other algorithms generally suffer from structural distortion, blurred boundaries, and loss of detail. For example, in the first row, only this invention clearly restored the outline of the distant buildings. This fully demonstrates that in scenarios with complex and varied semantic information, the "initial repair-semantic correction-fine repair" framework proposed in this invention can effectively identify and correct semantic deviations in the initial stage, thereby avoiding error accumulation and generating repair results that are highly reasonable in both structure and content.

[0082] To further objectively verify the technical effects of the present invention, quantitative comparative experiments were also conducted on publicly available datasets (such as Cityscapes). Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) were used as evaluation metrics. Higher PSNR and SSIM values ​​indicate better image fidelity and structural similarity; lower LPIPS values ​​indicate closer resemblance to human perception and higher restoration quality.

[0083] Table 1. Quantitative comparison of different image restoration methods on the Cityscapes dataset.

[0084]

[0085] As shown in Table 1, the method proposed in this invention is compared with several existing mainstream methods (such as RFR, CTSDG, MMT, LG-Net, W-Net, and MDTG). The data in the table shows that compared with MDTG, a state-of-the-art method that also utilizes semantic information, this method comprehensively outperforms other methods in terms of average PSNR, SSIM, and LPIPS, achieving a relative improvement of 1.58% in average SSIM and a relative reduction of 6.15% in average LPIPS. The Cityscapes dataset is complex, with diverse semantic categories, making it highly susceptible to semantic errors in the initial restoration process. The significant advantages of this invention on this dataset strongly demonstrate the effectiveness of our proposed semi-supervised semantic feedback correction module. This module can proactively identify and correct semantic biases introduced by the initial restoration, providing high-quality semantic priors for the final fine restoration, thereby achieving more accurate structure restoration and more realistic texture generation in complex urban scenes.

[0086] This application also provides an image inpainting system based on semi-supervised semantic feedback correction, including the following modules:

[0087] The coarse restoration module is used to process the masked image to be restored and generate a coarse restored image. The initial restoration module adopts a generative model architecture, which inputs the masked image to be restored into the generative network of the initial restoration module to generate a coarse restored image.

[0088] Semi-supervised semantic correction module: The semantic segmentation model of the repaired image optimized by the semi-supervised semantic feedback correction module is used to perform semantic segmentation on the generated coarse repaired image to generate a corrected semantic map.

[0089] The semi-supervised semantic feedback correction module establishes a feedback loop connecting image restoration and semantic segmentation. This feedback loop is used to optimize a final image restoration semantic segmentation model through collaborative training.

[0090] The process of establishing and utilizing the feedback loop includes: after supervised initial training of the original image semantic segmentation model and the repaired image semantic segmentation model (both using existing semantic segmentation models with the same structure), generating pseudo-labels for unlabeled images using the original image semantic segmentation model, and performing quality evaluation and screening of the pseudo-labels based on a preset semantic consistency criterion; adopting a differentiated optimization strategy for the pseudo-labeled images according to the evaluation results, that is, using high-quality pseudo-labels to perform joint semi-supervised optimization of the two semantic segmentation models, while placing the images corresponding to low-quality pseudo-labels in an iterative refinement process, using the optimized original image semantic segmentation model to generate new pseudo-labels for the final convergence training of the repaired image semantic segmentation model.

[0091] Fine-tuning module: Based on the semantic map corrected by the semantic correction module, a fine-tuning module is applied to refine the coarse-tuned image generated by the coarse-tuning module, thereby generating a final high-quality restored image. The fine-tuning module receives the coarse-tuned image and the corrected semantic map, and optimizes the coarse-tuned image according to the guidance of the corrected semantic map.

[0092] In one possible implementation, the generative model architecture used by the initial repair module is a Generative Adversarial Network (GAN), which includes a generator network (as a generator) and a discriminator; the specific operation is as follows:

[0093] The core objective of this module is to perform image inpainting tasks using preliminary pixel content generated for the missing regions. Given an image to be inpainted... and the corresponding binary mask The initial repair module uses the above information to generate a coarse repair image. The generator employs an encoder-decoder structure. During the generation process, pixels in the missing regions are filled with features learned from the image by the encoder-decoder structure. The encoder is used to learn features from the image to be repaired. Global and local features are extracted, and the decoder uses these features to reconstruct pixels in the missing regions. The main task of the initial inpainting is to recover the missing pixels and reconstruct basic texture information. Although the inpainting at this stage is not perfect, it provides an important foundation for subsequent semi-supervised semantic correction and fine-tuning.

[0094] In one possible implementation, the semi-supervised semantic feedback correction module operates as follows:

[0095] First, the task dataset is obtained according to the requirements. This dataset is divided into a "labeled image set" containing real semantic labels and an "unlabeled image set". The labeled and unlabeled image sets are derived from the same data distribution in terms of content and style to ensure the effectiveness of subsequent model training.

[0096] (1) Supervised initial training

[0097] 2-1. Using the labeled image set with real semantic labels, supervised training is performed on the original image semantic segmentation model. The original image semantic segmentation model adopts an existing semantic segmentation model.

[0098] 2-2. Apply a random mask to the labeled image set with real semantic labels, and process it using the initial repair module to generate a coarse repair result; then use the coarse repair result and its corresponding real semantic labels to conduct supervised training on the repaired image semantic segmentation model, the structure of the repaired image segmentation model being consistent with the original image semantic segmentation model.

[0099] (2) Pseudo-tag generation and filtering

[0100] 2-3. Using the unlabeled image set, input it into the original image semantic segmentation model trained in step 2-1 to generate pseudo-labels. Subsequently, the generated pseudo-labels are corrected based on a preset semantic consistency criterion, and a qualified, high-quality subset of pseudo-labeled images is selected.

[0101] (3) Semi-supervised iterative optimization

[0102] 2-4. Merge the labeled images with real semantic labels from step 2-1 with the high-quality qualified pseudo-labeled images selected from step 2-3 to form an enhanced training set. Use this enhanced training set to retrain the original image semantic segmentation model and the repaired image segmentation model to complete one round of iterative optimization.

[0103] 2-5. For the low-quality pseudo-labeled images that failed the screening in step 2-3, new pseudo-labels are regenerated using the optimized original image semantic segmentation model from step 2-4. These new pseudo-labels are then combined with the qualified pseudo-labeled image subset from step 2-3 to form an updated, complete pseudo-labeled image set. The repair image semantic segmentation model is further trained using both the labeled images with real semantic labels and the pseudo-labeled image set until the model performance converges. Finally, the fully optimized repair image semantic segmentation model can perform semantic segmentation on the coarsely repaired image to complete semantic correction.

[0104] In one possible implementation, the fine-tuning module employs a generative network integrating a "semantic-guided normalization module." Here, the "semantic-guided normalization module" is defined as a processing unit capable of using the corrected semantic map generated in the preceding steps as high-level structured guidance information, and adaptively modulating the feature maps of one or more intermediate layers in the generative network accordingly. Its core mechanism lies in the fact that the "semantic-guided normalization module" does not use globally uniform modulation parameters, but rather dynamically generates modulation parameters for different units (e.g., different spatial locations, feature channels, or combinations thereof) on the feature map based on the input semantic map, ultimately achieving final optimization of the image's structure, texture, and details under the guidance of precise semantic layout.

[0105] In one possible implementation, the specific structure of the fine repair module is as follows:

[0106] The fine-tuning module utilizes the corrected, precise semantic information to guide the final image content generation. Building upon semantic labels, the fine-tuning module extends the generator network of the initial inpainting module by integrating a Spatially-Adaptive Denormalization (SPADE) module as a semantically guided normalization module. The SPADE module receives feature maps output from the corresponding intermediate layers of the generator network in the fine-tuning module and the sampled corrected semantic map. And based on the semantic information of each spatial location, dynamically generate affine transformation parameters (scaling factors). and offset The SPADE module normalizes the feature maps. It dynamically adjusts the feature maps of the generator network based on the semantic information of each pixel location, making the details of the generated image more consistent with the input semantic labels, enhancing the semantic consistency of the generator model, and improving the final restored image. The quality.

[0107] This application also provides an electronic device, including a processor and a memory.

[0108] The memory is used to store computer programs.

[0109] When the processor executes a program stored in the memory, it implements any of the methods described in this application.

[0110] In one possible implementation, the electronic device of this application embodiment further includes a communication interface and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0111] The communication bus mentioned in the aforementioned electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0112] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0113] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0114] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements any of the methods described in this application.

[0115] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the methods described in this application.

[0116] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0117] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0118] The various embodiments in this specification are described in a related manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.

[0119] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. An image inpainting method based on semi-supervised semantic feedback correction, characterized in that, The steps include the following: Step 1: The initial repair module processes the masked image to be repaired to generate a coarse repaired image. The initial repair module adopts a generative model architecture, which inputs the masked image to be repaired into the generative network of the initial repair module to generate a coarse repaired image. Step 2: Use the semantic segmentation model of the repaired image optimized by the semi-supervised semantic feedback correction module to perform semantic segmentation on the generated coarse repaired image to generate a corrected semantic map; The semi-supervised semantic feedback correction module establishes a feedback loop connecting image restoration and semantic segmentation. This feedback loop is used to optimize a final restored image semantic segmentation model through collaborative training. Step 3: Based on the corrected semantic map, apply a fine restoration module to perform fine restoration on the coarse restoration image generated in Step 1 to generate the final high-quality restoration image; The fine restoration module is used to receive the coarse restoration image and the corrected semantic map, and optimize the coarse restoration image according to the guidance of the corrected semantic map.

2. The image inpainting method based on semi-supervised semantic feedback correction according to claim 1, characterized in that, The initial repair module employs a generative model architecture called a generative adversarial network, which includes a generator and a discriminator; the specific operation is as follows: Given an image to be repaired and a corresponding binary mask, the initial repair module uses the above information to generate a coarse repaired image. The generator adopts an encoder-decoder structure. During the generation process, the pixels in the missing area are filled by the features of the image learned by the encoder-decoder structure. The encoder is used to extract global and local features from the image to be repaired, and the decoder uses these features to reconstruct the pixels in the missing area.

3. The image inpainting method based on semi-supervised semantic feedback correction according to claim 1, characterized in that, The process of establishing and utilizing the feedback loop includes: after supervised initial training of the original image semantic segmentation model and the repaired image semantic segmentation model, generating pseudo-labels for unlabeled images using the original image semantic segmentation model, and performing quality evaluation and screening of the pseudo-labels based on a preset semantic consistency criterion; adopting a differentiated optimization strategy for the pseudo-labeled images based on the evaluation results, that is, using high-quality pseudo-labels to perform joint semi-supervised optimization of the two semantic segmentation models, while placing the images corresponding to low-quality pseudo-labels in an iterative refinement process, using the optimized original image semantic segmentation model to generate new pseudo-labels for the final convergence training of the repaired image semantic segmentation model.

4. The image inpainting method based on semi-supervised semantic feedback correction according to claim 3, characterized in that, The semi-supervised semantic feedback correction module operates as follows: First, the task dataset is obtained according to the requirements. This dataset is divided into a "labeled image set" containing real semantic labels and an "unlabeled image set" without labels. The labeled and unlabeled image sets are derived from the same data distribution in terms of content and style. (1) Supervised initial training 2-1. Using the labeled image set with real semantic labels, supervised training is performed on the original image semantic segmentation model using the image set; the original image semantic segmentation model adopts an existing semantic segmentation model; 2-2. Apply a random mask to the labeled image set with real semantic labels, and process it using the initial repair module to generate a coarse repair result; then use the coarse repair result and its corresponding real semantic labels to conduct supervised training on the repaired image semantic segmentation model, the structure of the repaired image segmentation model is consistent with the original image semantic segmentation model; (2) Pseudo-tag generation and filtering 2-3. Using the unlabeled image set, input it into the original image semantic segmentation model trained in step 2-1 to generate pseudo-labels; Subsequently, the generated pseudo-labels are corrected based on the preset semantic consistency criteria, and a qualified, high-quality subset of pseudo-label images is selected. (3) Semi-supervised iterative optimization 2-4. Merge the labeled images with real semantic labels from step 2-1 with the high-quality qualified pseudo-label images selected in step 2-3 to form an enhanced training set. Using this enhanced training set, the original image semantic segmentation model and the repaired image segmentation model are retrained to complete one round of iterative optimization; 2-5. For the low-quality pseudo-labeled images that failed the screening in step 2-3, use the optimized original image semantic segmentation model from step 2-4 to regenerate new pseudo-labels. Combine the new pseudo-labels with the qualified pseudo-labeled image subset from step 2-3 to form an updated and complete pseudo-labeled image set. Further train the repair image semantic segmentation model by combining the labeled images with real semantic labels and the pseudo-labeled image set until the model performance converges. Finally, the fully optimized repair image semantic segmentation model can be used to perform semantic segmentation on the coarse repair image to complete semantic correction.

5. The image inpainting method based on semi-supervised semantic feedback correction according to claim 1, characterized in that, The fine repair module employs a generative network that integrates a "semantic guidance normalization module". Here, the "semantic guidance normalization module" is defined as a processing unit that can use the corrected semantic map generated in the aforementioned steps as high-level structured guidance information, and accordingly perform adaptive modulation on the feature maps of one or more intermediate layers of the generative network. Its core mechanism is that the "semantic-guided normalization module" does not use globally unified modulation parameters, but dynamically generates modulation parameters for different units on the feature map based on the input semantic map, and finally optimizes the structure, texture and details of the image under the guidance of precise semantic layout.

6. The image inpainting method based on semi-supervised semantic feedback correction according to claim 5, characterized in that, The specific structure of the fine repair module is as follows: Based on semantic labels, the fine repair module extends the generator network of the initial repair module and integrates the spatial adaptive normalization SPADE module as a semantic-guided normalization module in the network structure. The SPADE module receives the feature map output by the corresponding intermediate layer of the generator network in the fine repair module and the sampled corrected semantic map, and dynamically generates affine transformation parameters according to the semantic information of each spatial location to normalize the feature map.

7. An image inpainting system based on semi-supervised semantic feedback correction, characterized in that, The system includes: Coarse Repair Module: The initial repair module processes the masked image to be repaired to generate a coarse repaired image. The initial repair module adopts a generative model architecture, inputting the masked image to be repaired into the generative network of the initial repair module to generate the coarse repaired image. Semi-supervised semantic correction module: The semantic segmentation model of the repaired image optimized by the semi-supervised semantic feedback correction module is used to perform semantic segmentation on the generated coarse repaired image to generate a corrected semantic map; Fine-tuning module: Based on the semantic map corrected by the semantic correction module, a fine-tuning module is applied to refine the coarse-tuned image generated by the coarse-tuning module to generate the final high-quality restored image; the fine-tuning module is used to receive the coarse-tuned image and the corrected semantic map, and optimize the coarse-tuned image according to the guidance of the corrected semantic map.

8. The image inpainting method based on semi-supervised semantic feedback correction according to claim 7, characterized in that, The semi-supervised semantic feedback correction module establishes a feedback loop connecting image restoration and semantic segmentation. This feedback loop is used to optimize a final image restoration semantic segmentation model through collaborative training. The process of establishing and utilizing the feedback loop includes: after supervised initial training of the original image semantic segmentation model and the repaired image semantic segmentation model, generating pseudo-labels for unlabeled images using the original image semantic segmentation model, and performing quality evaluation and screening of the pseudo-labels based on a preset semantic consistency criterion; adopting a differentiated optimization strategy for the pseudo-labeled images based on the evaluation results, that is, using high-quality pseudo-labels to perform joint semi-supervised optimization of the two semantic segmentation models, while placing the images corresponding to low-quality pseudo-labels in an iterative refinement process, using the optimized original image semantic segmentation model to generate new pseudo-labels for the final convergence training of the repaired image semantic segmentation model.

9. An electronic device, characterized in that, Including processor and memory; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the image restoration method according to any one of claims 1-6.

10. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the image restoration method according to any one of claims 1-6.

Images