An Image Blind Restoration Method Based on Semantic Inconsistency Detection

Abstract

The present invention discloses a blind image restoration method based on semantic inconsistency detection. The method comprises: pre-processing a noise-contaminated image as input; using a mask prediction network constructed with annular residual blocks to amplify the semantic difference between the contaminated area and the background, and roughly locating the degraded area in the contaminated image; then utilizing the texture similarity between regions of different categories to obtain a refined prediction mask through a mask refinement network; jointly inputting the damaged image and the prediction mask into an image restoration network, iteratively using information from valid areas to complete the content of the damaged area based on mask confidence considerations; simultaneously using a contextual attention aggregation module at different scales to improve structural consistency; and fusing multiple feature information and then decoding and restoring it into an image, thereby achieving blind restoration of the degraded image. The present invention can accurately detect noise contamination in real damaged images, meeting the requirements for robust restoration of various degraded images.

Description

Technical Field

[0001] The invention belongs to the field of computer graphics and image processing, and relates to an image blind restoration method based on semantic inconsistency detection. Background Art

[0002] With the development of computer and multimedia technologies, digital images have become important information carriers. Over time and due to unforeseen circumstances, photographs can experience various degradations during their preservation process, such as ink stains, creases, damage, mold, and fading. Furthermore, recording moments can also be subject to unexpected events, such as someone grabbing the spotlight while taking a photo or a camera lens becoming stained. These factors can significantly impact the expression of image content. Consequently, image inpainting techniques, which aim to complete image content and improve image quality, have developed rapidly in recent years and are widely used in fields such as image editing, object removal, biomedical image processing, and criminal investigation. Image inpainting technology has achieved numerous important research results over the years. Currently, widely used processing methods, such as Photoshop's inpainting tool, employ traditional inpainting methods, leveraging the redundancy of image information to fill damaged areas with pixels from known regions. These methods can effectively inpaint scenes with repetitive textures, but due to their lack of understanding of image semantics, they cannot generate new content.

[0003] As a major research hotspot in computer vision, researchers have recently attempted to introduce deep learning methods into the field of image restoration. While these models can infer missing content from the valid pixels provided, they all assume that blank areas in the image are damaged and explicitly require a binary mask for calibration. These methods are effective in training models to infer the content of missing areas. However, the degradation patterns and locations of real-life damaged images are often unknown, making it difficult to provide accurate masks to guide the repaired areas in advance. This significantly limits the applicability of these methods in real-world scenarios. Therefore, identifying and repairing damaged content in an image based solely on the damaged image has become a pressing challenge. Summary of the Invention

[0004] In order to overcome the above-mentioned defects, the present invention proposes a blind image restoration method based on semantic inconsistency detection. The specific steps of the present invention are as follows:

[0005] S1, input a damaged image I m , including clean pixel areas and polluted pixel areas;

[0006] S2, builds a mask prediction network through multi-layer residual blocks to generate a single-channel rough prediction soft mask for locating the damaged area

[0007] S3: The rough prediction mask obtained in S2 and the damaged image are input into the mask refinement network again. The prediction accuracy of the boundary area is improved through reinforcement learning to obtain a fine prediction mask of the damaged area.

[0008] S4, takes the refined prediction mask obtained in S3 as prior information and inputs it into the shared encoder together with the damaged image. It extracts the features of valid pixels according to the guidance of the mask and propagates them to the damaged area.

[0009] In S5, the deep feature map extracted by the encoder network is input into the multi-task parallel decoding branch. After passing through multiple layers of convolutional blocks, the content of the missing area is inferred and the context information is used to ensure global semantic consistency.

[0010] S6, fuses the features extracted by different branches in S5, decodes them through the decoder network, and restores them to an image;

[0011] S7, using the refined prediction mask in S3, intercepts the pixels at the damaged area position in the result of S6 and splices them with the valid pixels in the damaged image to output the final repaired image.

[0012] The technical features and improvements of the present invention are:

[0013] For step S1, the present invention first defines the damaged image. Unlike existing research that simply uses blank pixels to represent the area to be repaired, the present invention considers that the damaged image should be composed of clean valid pixels and different types of degraded and contaminated pixels. Since there is currently no dataset dedicated to blind repair research, the present invention first synthesizes batch training data based on the above idea for model training. The mathematical expression is as follows:

[0014]

[0015] In formula (1), I m represents the damaged image of the splicing, I gt In order to improve the robustness of the method of the present invention, N simulates graffiti, folds, text occlusion, and any interception of other images to splice into I. gt Generate a corrupted image I containing various types of pollution and degradation m .

[0016] Preferably, in step S1, in order to make the contamination noise and the original image merge more naturally, the present invention uses a smoothing Gaussian function for smoothing, which is expressed as follows:

[0017] I=I m *G σ (2)

[0018] In formula (2), I represents the smoothed damaged image, I m represents the directly stitched damaged image, G σ represents a two-dimensional Gaussian kernel with standard deviation σ.

[0019] For step S2, the present invention uses a modified annular residual convolution block as a feature extractor, which amplifies the difference between the valid pixel area and the contaminated area, compares the inherent properties of different image regions, and then locates the damaged area. Among them, the annular residual block used in the present invention includes three steps. Its design is derived from the memory and consolidation mechanism of the human brain and is implemented through the propagation and feedback process of the residual in the CNN. The first stage is the forward residual propagation, which solves the gradient degradation problem in the deeper network by recalling the input feature information. The formula definition can be expressed as:

[0020] y f =F(x, {W i})+W s *x (3)

[0021] In formula (3), x represents the input feature map, y f Denotes the learned residual mapping. F(x, {W i}) represents the learned residual map, whose structure includes two convolutional layers and activation function ELU, W s This is a 1×1 convolution. Residual propagation is similar to the memory mechanism of the human brain. As the model learns new knowledge, it may forget previous knowledge, so a recall mechanism is needed to help recall those fuzzy memories.

[0022] To further enhance the distinction between the properties of damaged and valid content, the second stage uses residual feedback to integrate input feature information. A simple gating mechanism is used to learn the nonlinear relationship between discernible feature channels, preventing the diffusion of feature information. The response values are then superimposed on the input features through an activation function, amplifying the essential attribute differences between noisy and valid areas of the image. The formula is defined as follows:

[0023] y b =(s(G(y f ))+1)*x (4)

[0024] In formula (4), x is the residual mapping feature, y bis the residual feedback feature, G(·) is a linear mapping, and s is the activation function, here using the sigmoid function. Unlike the recall mechanism simulated by residual propagation, residual feedback appears to mimic the human brain's process of consolidating knowledge, gaining a new understanding of the features. The third stage repeats the first stage, performing residual propagation on the new features to further learn the amplified feature differences. The two forward residual propagations and one reverse residual feedback form a circular residual structure.

[0025] For step S3, the present invention introduces an attention mechanism to refine the coarse prediction mask, and improves the recognition results of details such as contours by paying attention to similar textures across the entire image. Specifically, if a low-confidence area predicted to be damaged shares a similar texture with a high-confidence area, the low-confidence area should be modified. To this end, it is necessary to extract the key features of the damaged content from the high-confidence area and use them as global visual features of this class. The present invention calculates the cosine similarity of the coarse prediction mask as a new bias, and then reduces the score map of the predicted area through Softmax. After reducing the score, the area that remains highlighted can be considered to have sufficiently significant features, so these areas can be selected to extract key features as global features of the damaged area. The calculation formula is as follows:

[0026] CosSim(x′ sem )=X∈R c×c

[0027]

[0028]

[0029] In formula (5), CosSim(·) represents the improved cosine similarity calculation function, x′ sem Represents the prediction weight matrix, i and j represent the prediction categories, which can be divided into damaged areas and non-damaged areas, X i，j represents the cosine similarity between two pixels with different predicted categories, is x′ sem The i-th channel of X represents the prediction result of each pixel belonging to a certain class. i，j The closer it is to 1, and The more similar the activation results are, the less credible the position prediction is. By setting the deviation of the same type of pixels to 0 and the deviation of different types of pixels to a similarity score X i，j , so that the area that still maintains high activation value in the classification is the key feature, and the whole process is called key feature pooling.

[0030] Preferably, in step S3, the present invention uses the prediction weight matrix x′ sem and feature map x fCalculate the weighted sum to get the key feature v k , as follows:

[0031]

[0032] Where i represents the predicted category. k As the key, the feature x f Treat it as a query and highlight the key features v k Similar areas are obtained by an AttentionMap, which is then convolved with the original image to predict the final refined prediction mask.

[0033] For step S4, the present invention introduces a gated convolution mechanism to improve the residual convolution block, identifies damaged areas through learning, and dynamically selects valid pixel content in the image, so that the convolution result depends only on valid pixels, replacing the traditional residual convolution structure to extract and integrate features of the valid area. The output of the gated convolution is calculated as:

[0034] Gating y，x =∑∑W g I

[0035] Feature y，x =∑∑W f ·I (7)

[0036]

[0037] In formula (7), I represents the input feature, W g and W f Represents two different convolution kernels, φ represents the use of LeakyReLU activation function, σ represents the sigmoid function, and all values are restricted to [0,1] to indicate the importance of each local area. Indicates element-by-element multiplication, O y，x represents the soft-gated weighted output features.

[0038] Preferably, in step S4, in order to avoid the influence of error accumulation of the prediction mask on the image restoration result, the present invention uses a new probabilistic context normalization (PCN) at the end of the improved residual block to transfer statistical information, and propagates statistical information such as the mean and variance of the valid pixel area to the damaged area to ensure that the distribution of features inside and outside the hole is consistent. The public representation is as follows:

[0039]

[0040] In formula (8), X represents the output of the last convolution layer in the gated residual block, and H represents the predicted mask Sampling to the same size as X, β is a learnable channel attention weight, "indicates information transfer, specifically:

[0041]

[0042] In formula (9), X P and X Q where represents the polluted area and the valid pixel area, μ(·) represents the regional mean, and σ(·) represents the regional variance. For images, the feature mean is related to global semantics, while the variance is related to local texture features. This paper uses PCN to transfer feature statistics, which is beneficial for integrating contextual information to generate missing content.

[0043] For step S5, the present invention uses a multi-scale contextual attention aggregation branch to obtain image context information, where the contextual similarity calculation cosine similarity measures the similarity between patches inside and outside the missing area, and finds the content with the highest similarity in the valid area for the patch to be completed, assigning a higher reference weight so that the completed content is consistent with the context in terms of semantics and texture. The similarity measurement formula is as follows:

[0044]

[0045] In formula (10), p i and p j They represent the feature patches of the valid area and the missing area respectively, and then pass a softmax function to get the attention score of each patch:

[0046]

[0047] Where N is the number of patches in the valid region. After calculation, each patch in the missing region finds a more noteworthy area within the valid pixels, which will be given a higher reference weight in the subsequent feature fusion.

[0048] Preferably, in step S5, in order to reduce the amount of calculation and improve the inference speed, the present invention uses context information transfer to propagate the calculated inter-patch attention similarity score. Specifically, the present invention calculates the similarity score once on the feature map of the deep layer of the network with a size of 32×32, and then uses context attention transfer to propagate the attention score to lower layers of different scales for feature weighting, in the following way:

[0049]

[0050] In formula (12), l represents different shallow layers of the network, Indicates the missing area patch at different scales, Indicates the effective area of the same size, s i，j represents the attention score, and N represents the number of patches in the background. Since feature map sizes vary layer by layer, the patch size should also vary accordingly. This is done by comparing the current feature map size with the attention score map and zooming in on the mapped area. For example, in a 128×128 feature map, every four adjacent pixels share an attention score. This sharing of attention scores not only achieves better global semantic consistency in model inference results, but also significantly improves storage and computational efficiency.

[0051] The blind image restoration method based on semantic inconsistency detection of the present invention solves the problem that existing technologies cannot repair damaged images with multiple degradation modes in real scenes and that calibration masks are difficult to obtain directly. It has the following advantages:

[0052] (1) The method of the present invention analyzes and designs an end-to-end network model. Compared with existing repair methods, it does not require a mask to calibrate the damaged area. It automatically identifies the contaminated and damaged areas in the image and repairs them to obtain semantically consistent and visually complete results. It can repair various damage patterns in real images with robustness and authenticity.

[0053] (2) The method of the present invention can be easily extended to other research fields of image processing, such as target removal, highlight removal, image rain and fog removal, etc., and has good portability and applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flowchart of the blind image restoration based on semantic inconsistency detection in the present invention.

[0055] Figure 2 Schematic diagram of the structure of the prediction mask refinement module in the present invention.

[0056] Figure 3 Schematic diagram of the structure of the probabilistic context content aggregation convolution block in the present invention. DETAILED DESCRIPTION

[0057] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] A blind image restoration method based on semantic inconsistency detection, such as Figure 1 FIG. 1 is a flowchart of a blind image restoration method based on semantic inconsistency detection according to the present invention, which comprises:

[0059] S1, data preprocessing, read a damaged image I with noise pollution mThe images are resized to 256×256 and then normalized before being fed into the network model. During the training phase, various degradation patterns found in real scenes are simulated to create corrupted images, which are then processed using Gaussian smoothing to make them appear more realistic and natural.

[0060] S2, coarse prediction of damaged areas, inputs the processed degraded image into a coarse mask prediction network constructed by a six-layer ring residual block. Its overall structure is a codec network that integrates image context information through convolution to learn the inherent properties of the image. The ring structure of residual propagation and residual feedback alternately calculates to amplify the difference between the valid pixel area and the damaged area, and generates a single-channel coarse damaged area prediction mask. When calculating the loss during the training phase, since we only need to determine whether each position belongs to the valid area or the damaged area, we use binary cross entropy loss as the loss function, which is expressed as follows:

[0061]

[0062] In formula (13), T is the adaptive weight, p∈{p|M p =1} represents the real damaged area, q∈{q|M q =0} indicates the real effective area.

[0063] S3, prediction mask refinement, inputs the rough prediction mask generated by S2 and the damaged image into the mask refinement network, such as Figure 2 As shown in the figure, a simple encoder is first used to extract image features, and the cosine similarity between pixels predicted to be of different categories is calculated. Then a softmax function is used to limit the value to [0, 1]. The closer the value is to 1, the less credible the predicted category of the area is. After this, the key features of the damaged area with high confidence are selected as the key. The query method of the attention mechanism is used to traverse the entire image feature Query to obtain the global attention weight. Finally, the updated feature information is integrated through deconvolution and the image is restored to obtain a refined prediction mask with clearer and more accurate detail contours.

[0064] S4, content feature extraction, the damaged image is input into the encoder. In order to avoid the influence of the accumulation of prediction mask errors, the predicted refined mask is scaled to the same size as the feature map and then input into each layer of the encoder to guide the extraction of valid pixel information and its transfer to the damaged area. The encoder consists of four layers of gated residual convolution blocks newly designed by this invention, and its structure is as follows: Figure 3As shown, the outputs of two standard convolutional layers for different tasks are element-wise multiplied. One layer is followed by a LeayReLU function, and the other is followed by a sigmoid function. This automatically learns to update the soft mask from the input in a learnable manner, restricting the convolution operation to the valid pixel region. In addition, probabilistic context normalization is used instead of batch normalization to transfer image statistics and ensure consistent distribution of feature information inside and outside the mask.

[0065] S5, missing area content inference, this paper proposes a multi-task parallel framework, and designs two parallel decoding branches for feature inference and content propagation. Figure 1 As shown in the figure, the upstream branch consists of multi-layer dilation convolutions with dilation rates of 2, 4, and 8. Different dilation rates are used to expand the receptive field and capture multi-scale contextual information. The downstream branch uses a multi-scale contextual attention integration module to calculate the attention scores between different patches on the feature map of the deep network with a size of 32×32. Through the contextual attention transfer module, feature weighting is performed on the shallow layers of the network at different scales to ensure the global structure and semantic consistency of the features.

[0066] S6, feature decoding and image restoration: The feature maps extracted from different branches in S5 are concatenated channel by channel and then fed into the decoder network for decoding. The decoder's structure is symmetrical to the encoder's. It fuses features through four layers of gated residual convolution blocks alternating with upsampling, and finally restores the predicted inpainted image through a layer of 3×3 ordinary convolution.

[0067] S7 outputs the final restoration result. To ensure a clearer result, the prediction mask is used to select the valid content of the input image and the content of the prediction result for splicing. After smoothing, a clean restoration result with complete structure and consistent semantics is output.

[0068] In summary, the image blind restoration method based on semantic inconsistency detection of the present invention is suitable for repairing real damaged images in real life. It does not require additional binary masks to mark the damaged areas. It achieves high-quality restoration of degraded images through an end-to-end network, ensuring that the restoration results have visual integrity and structural rationality. It can robustly solve various image degradation and pollution faced in different real scenes, and has wide application value.

[0069] Although the present invention has been described in detail through the above preferred embodiments, it should be understood that the above description is not intended to limit the present invention. After reading the above description, various modifications and substitutions of the present invention will become apparent to those skilled in the art. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A blind image restoration method based on semantic inconsistency detection, its characteristics and The specific steps are as follows: S1, input a damaged image I m , including clean pixel areas and damaged pixel areas; S2, builds a mask prediction network through multi-layer residual blocks to generate a single-channel rough prediction soft mask for locating the damaged area S3: The rough prediction mask obtained in S2 and the damaged image are input into the mask refinement network again. The prediction accuracy of the boundary area is improved through reinforcement learning to obtain a fine prediction mask of the damaged area. S4, takes the refined prediction mask obtained in S3 as prior information and inputs it into the shared encoder together with the damaged image. It extracts the features of valid pixels according to the guidance of the mask and propagates them to the damaged area. In S5, the deep feature map extracted by the encoder network is input into the multi-task parallel decoding branch. After passing through multiple layers of convolutional blocks, the content of the missing area is inferred and the context information is used to ensure global semantic consistency. S6, fuses the features extracted by different branches in S5, decodes them through the decoder network, and restores them to an image; S7, using the refined prediction mask in S3, intercepts the pixels at the damaged area position in the result of S6 and splices them with the valid pixels in the damaged image to output the final repaired image.

2. The method for blind image restoration based on semantic inconsistency detection according to claim 1, characterized in that: For step S1, the damaged image is first defined. The damaged image consists of clean valid pixels and different types of degraded and polluted pixels. The synthetic batch training data is used for model training. The mathematical expression is as follows: In formula (1), I m represents the damaged image of the splicing, I gt represents a completely clean image, N represents the contaminated noise content, and M is a binary mask; N simulates graffiti, creases, text occlusion, and any interception of other images to be spliced into I gt Generate a corrupted image I containing various types of pollution and degradation m ; In step S1, in order to make the contamination noise merge with the original image more naturally, a smooth Gaussian function is used for smoothing, which is expressed as follows: I=I m *G σ (2) In formula (2), I represents the smoothed damaged image, I m represents the directly stitched damaged image, G σ represents a two-dimensional Gaussian kernel with standard deviation σ; For step S2, a modified annular residual convolution block is used as a feature extractor. By amplifying the difference between the valid pixel area and the polluted area, the inherent properties of different image regions are compared to locate the damaged area. The annular residual block used includes three steps, which are implemented through the propagation and feedback process of the residual in the CNN. The first stage is the forward residual propagation, which solves the gradient degradation problem in the deeper network by recalling the input feature information. The formula definition is expressed as: y f =F(x,{W i })+W s *x (3) In formula (3), x represents the input feature map, y f represents the learned residual mapping; F(x,{W i }) represents the learned residual map, whose structure includes two convolutional layers and activation function ELU, W s It is a 1×1 convolution; To further enhance the difference between the attributes of damaged content and valid content, the second stage uses residual feedback to integrate input feature information. A simple gating mechanism is used to learn the nonlinear relationship between discriminable feature channels to avoid the diffusion of feature information. The response value is then superimposed on the input feature through an activation function to amplify the difference in the essential image attributes between the noisy and valid areas. The formula is defined as follows: and b =(s(G(y f ))+1)*x (4) In formula (4), x is the residual mapping feature, y b is the residual feedback feature, G(·) is a linear mapping, s is the activation function, and the sigmoid function is used here. The third stage repeats the first stage operation and performs residual propagation on the new features. Two forward residual propagations and one reverse residual feedback are combined to form a ring residual structure. For step S3, an attention mechanism is introduced to refine the coarse prediction results, focusing on similar textures across the entire image to improve the recognition results of details such as contours. Specifically, if the low-confidence region predicted to be damaged shares similar textures with the high-confidence region, the low-confidence region is modified. To this end, the key features of the damaged content need to be extracted from the high-confidence region and used as the global visual features of this class. The cosine similarity of the coarse prediction mask is calculated as a new bias, and then the score map of the predicted area is reduced through Softmax. The areas that remain highlighted after the score reduction are considered to have sufficiently significant features, so these areas are selected to extract key features as the global features of the damaged area. The calculation formula is as follows: In formula (5), CosSim(·) represents the improved cosine similarity calculation function, x′ sem represents the prediction weight matrix, i and j represent the prediction categories, which are divided into damaged areas and non-damaged areas, X i,j represents the cosine similarity between two pixels with different predicted categories, is x′ sem The i-th channel of represents the prediction result of each pixel belonging to a certain class; X i,j The closer it is to 1, and The more similar the activation results are, the less credible the position prediction is; by setting the deviation of the same type of pixels to 0 and the deviation of different types of pixels to a similarity score X i,j , the area that still maintains high activation value in the classification is the key feature, and the whole process is called key feature pooling; In step S3, the prediction weight matrix x' is used sem and feature map x f Calculate the weighted sum to get the key feature v k , as follows: Where i represents the predicted category; the key feature v k As the key, the feature x f Treat it as a query and highlight the key features v k Similar areas, get an Attention Map, and then perform convolution operation with the original image to predict the final refined prediction mask For step S4, a gated convolution mechanism is introduced to improve the residual convolution block. The damaged area is identified by learning, and the valid pixel content in the image is dynamically selected so that the convolution result depends only on the valid pixels. This replaces the traditional residual convolution structure to extract and integrate the features of the valid area. The output of the gated convolution is calculated as: In formula (7), I represents the input feature, W g and W f Represents two different convolution kernels, φ represents the use of LeakyReLU activation function, σ represents the sigmoid function, and all values are restricted to [0,1] to indicate the importance of each local area. Indicates element-by-element multiplication, O y,x represents the output features of soft-gating weights; In step S4, in order to avoid the accumulation of errors in the predicted mask affecting the image restoration results, a new probabilistic context normalization (PCN) is used at the end of the improved residual block to transfer statistical information. The statistical information such as the mean and variance of the valid pixel area is propagated to the damaged area to ensure that the distribution of features inside and outside the hole is consistent. The public representation is as follows: In formula (8), X represents the output of the last convolution layer in the gated residual block, and H represents the predicted mask Sampling to the same size as X, β is a learnable channel attention weight, Γ represents information transfer, specifically: In formula (9), X P and X Q They represent the polluted area and the valid pixel area respectively, μ(·) represents the regional mean, and σ(·) represents the regional variance. For images, the feature mean is related to the global semantics, and the variance is related to the local texture features. The statistical information transmission of features through PCN is conducive to the fusion of context information to generate missing content. For step S5, a multi-scale contextual attention aggregation branch is used to obtain image context information. The contextual similarity calculation cosine similarity measures the similarity between patches inside and outside the missing area. The patch of the area to be completed is found to have the highest similarity in the valid area and is assigned a higher reference weight so that the completed content is consistent with the context in terms of semantics and texture. The similarity measurement formula is as follows: In formula (10), p i and p j They represent the feature patches of the valid area and the missing area respectively, and then pass a softmax function to get the attention score of each patch: Where N represents the number of patches divided into valid areas. After calculation, each patch in the missing area finds an area within the valid pixels that deserves more attention, and will be given a higher reference weight in the subsequent feature fusion. In step S5, in order to reduce the amount of calculation and improve the inference speed, the calculated inter-patch attention similarity score is propagated using context information transfer. Specifically, the similarity score is calculated once on the feature map of the deep layer of the network with a size of 32×32, and then the attention score is propagated to the lower layers of different scales using context attention transfer for feature weighting, as follows: In formula (12), l represents different shallow layers of the network, Indicates the missing area patch at different scales, Indicates the effective area of the same size, s i,j Represents the attention score, and N represents the number of patches in the background. Since the size of the feature map changes in layers, the size of the patch should also change accordingly. The specific operation is to compare the current feature map size with the attention score map, enlarge the mapping area, and every four adjacent pixels in the 128×128 feature map share an attention score value.

Images