Image blind restoration method based on self-information and prediction mask enhancement

Abstract

The application discloses an image blind restoration method based on self-information and prediction mask enhancement, and belongs to the technical field of computer vision. The method comprises the following steps: converting a damaged image into a two-dimensional feature; using a TSBlock coding module to code the two-dimensional feature to obtain a coded feature; predicting a damaged area mask of the damaged image, and using the predicted mask to generate an information enhancement feature which only retains an undamaged area; using an information enhancement IETSBlock coding module to code the coded feature and the information enhancement feature to obtain an information enhancement coded feature; reusing the TSBlock coding module to recode the information enhancement coded feature; using a scale recovery and refinement module to recover the recoded feature into a multi-channel feature which is consistent in size with the damaged image; and using channel recovery to perform channel filtering on the multi-channel feature to obtain a restored image. The application solves the problem that image restoration is prone to failure.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and more particularly to an image blind restoration method based on self-information and prediction mask enhancement. BACKGROUND

[0002] With the development of current artificial intelligence and computer vision technology, image processing technologies such as Artificial Intelligence Generated Content (AIGC) gradually show great development potential and are applied to education, medical treatment and other fields. The development of such technologies relies on a large number of complete image data sets. However, images collected from various channels inevitably have degradation phenomena such as damage, which in turn negatively affects the performance of the model. At the same time, the poor visual effect of damaged images also affects the practicality of the images themselves (for example: damaged medical images, damaged traffic monitoring snapshot images, etc.).

[0003] Traditional image restoration methods mainly rely on specific patterns to restore images, have weak generalization ability, and cannot adapt to the restoration of complex damaged images. Therefore, in recent years, with the development of deep learning, using digital image restoration technology to restore damaged images has become a current research hotspot and has obtained certain application. Digital image restoration technology can quickly restore relatively complex damaged images such as natural scenery, murals, and old photographs.

[0004] The current mainstream digital image restoration method is mainly divided into two categories: non-blind restoration method and blind restoration method. The non-blind damaged image restoration method assumes that the mask is known, and good image restoration results have been achieved in damaged image restoration. However, the non-blind restoration method relies heavily on the mask, so this method needs to first label the damaged area of the image during testing, and the application is greatly limited.

[0005] Based on the above problems, in recent years, researchers have begun to focus on image blind restoration methods that do not require masks. The current blind restoration method mainly includes two paradigms: direct blind restoration method and multi-stage blind restoration method based on prediction. The direct blind restoration method is usually a single-stage network, and this method is easily affected by the content of the damaged area due to the lack of guidance for the damaged area and the undamaged area, resulting in color artifact residual phenomenon. The multi-stage blind restoration method based on prediction is mainly composed of several sub-networks trained in multiple stages in series, and this method is easily affected by the prediction results of the front network, resulting in failure of the subsequent network restoration.

[0006] Therefore, how to solve the problems of the direct blind restoration method lacking damaged area guidance and the multi-stage blind restoration method being easily affected by the prediction results of the sub-networks is an urgent problem for those skilled in the art. SUMMARY

[0007] In view of this, the present invention provides a method and system for blind image inpainting based on self-information and predictive mask enhancement, which is used to at least solve some of the technical problems in the background art.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] This invention first discloses a blind image inpainting method based on self-information and predictive mask enhancement, comprising the following steps:

[0010] Convert damaged images into feature tokens;

[0011] The feature token is encoded using a first encoding module composed of Transformer Blocks to obtain the encoded feature;

[0012] Perform mask prediction on the damaged image, and use the predicted mask and the damaged image to generate information enhancement features that retain only the pixels in the undamaged areas;

[0013] The second coding module, composed of information-enhanced Transformer Blocks, encodes the coding features and information-enhanced features to obtain information-enhanced coding features;

[0014] The information enhancement coding features are recoded using a third coding module composed of Transformer Blocks to obtain recoded features;

[0015] The recoded features are restored to multi-channel features of the same size as the damaged image through the scale restoration and thinning module;

[0016] The multi-channel features are then subjected to channel filtering through channel restoration to obtain the repaired image.

[0017] In one specific embodiment, converting a damaged image into a feature token specifically includes:

[0018] The damaged image I with dimensions h*w*c d Divided into several image blocks of the same size I' d Each image patch has a size of p*p*c, and h represents the damaged image I. d The height of w indicates the damaged image I. d The width; p represents the height or width of the image block; c represents the number of channels;

[0019] Each image block I' d Flattened to p 2 *c is a one-dimensional feature vector;

[0020] The flattened one-dimensional feature vector is projected into a set D-dimensional space using a linear layer to obtain each image block I' d The corresponding Patch Embedding sequence;

[0021] All image blocks I' d The Patch Embedding sequence is combined to obtain the Patch Embeddings sequence, i.e., the embedding sequence of the damaged image I d .

[0022] The embedding sequence of the damaged image I d is added with a class token.

[0023] The embedding sequence with the added class token is positionally encoded to obtain the feature token of the damaged image I d .

[0024] Further, in the step of encoding the feature token using the first encoding module composed of TransformerBlock in the application, the first encoding module includes two TransformerBlock units with the same structure and mutual superposition, denoted as the first TransformerBlock unit and the second TransformerBlock unit.

[0025] The first TransformerBlock unit is used to perform primary encoding on the feature token.

[0026] Further, in the embodiment of the application, the first TransformerBlock unit performs primary encoding on the feature token, which specifically includes the following encoding process:

[0027] F T1 =TSBlock(F embedding )

[0028] =MLP(Norm(MHA(Norm(F embedding ))+F embedding ))

[0029] Wherein, F T1 represents the primary encoding result obtained by the first TransformerBlock performing primary encoding on the feature token, TSBlock() represents the TransformerBlock unit processing operation, F embedding represents the feature token obtained by converting the damaged image, Norm() represents the normalization operation, MHA() represents the multi-head attention operation, and MLP() represents the multi-layer perceptron operation.

[0030] Further, in the embodiment of the present application, the damaged image is subjected to damaged area mask prediction, and an information enhancement feature retaining only the undamaged area is generated by using the predicted mask and the damaged image, specifically including:

[0031] The damaged image is subjected to mask prediction by using the trained deep learning model, and a predicted mask of the damaged area is obtained;

[0032] According to the predicted mask of the damaged area, the damaged area and the undamaged area in the damaged image are obtained;

[0033] The pixel values of the undamaged area in the damaged image are retained, and the pixel values of the damaged area are replaced with a fixed value, to obtain an information enhancement feature retaining only the pixel values of the undamaged area in the damaged image.

[0034] Further, in the embodiment of the present application, in the encoding step of the encoding feature and the information enhancement feature by using the second encoding module composed of information enhancement TransformerBlock, the second encoding module includes two information enhancement TransformerBlock units with the same structure and mutual superposition, which are denoted as a first information enhancement TransformerBlock unit and a second information enhancement TransformerBlock unit.

[0035] The first information enhancement TransformerBlock unit is used for primary encoding of the encoding feature and the information enhancement feature, and the second information enhancement TransformerBlock unit is used for secondary encoding of the primary encoding result of the first information enhancement TransformerBlock unit and the information enhancement feature.

[0036] Further, in the embodiment of the present application, the first information enhancement TransformerBlock unit performs primary encoding on the encoding feature and the information enhancement feature, specifically including the following encoding process:

[0037] F temp1 =MHA(Norm(F CT1 ))+F CT1 ;

[0038] F IET1 =Linear(Cat(Norm(MLP(Norm(F temp1 ))+F temp1 ),F enhanced ));

[0039] Wherein, F temp1 represents the intermediate encoding feature, F IET1 represents the primary encoding result, and F CT1F represents the encoding feature obtained by the first encoding module enhanced Norm() represents a normalization operation, MHA() represents a multi-head attention operation, MLP() represents a multi-layer perception operation, Cat() represents a concatenation operation, and Linear() represents a linear mapping.

[0040] Further, in the embodiment of the present application, in the step of re-encoding the information-enhanced encoding feature by using the third encoding module composed of the Transformer Block, the third encoding module comprises five Transformer Block units which are the same in structure and stacked with each other.

[0041] Further, in the embodiment of the present application, in the step of restoring the re-encoded feature to a multi-channel feature consistent with the size of the damaged image by using the scale restoration and refinement module, the scale restoration and refinement module comprises two scale restoration and refinement units which are the same in structure and stacked with each other, denoted as a first scale restoration and refinement unit and a second scale restoration and refinement unit.

[0042] The first scale restoration and refinement unit is configured to perform preliminary dimension lifting and refinement on the re-encoded feature output by the third encoding module to obtain a preliminary dimension lifting and refinement result.

[0043] Further, in the embodiment of the present application, the first scale restoration and refinement unit performs preliminary dimension lifting and refinement on the re-encoded feature output by the third encoding module, and specifically includes the following operation process:

[0044] F recovery1 = Relu(BatchNorm(Conv(Upsampling(F encoder ))))

[0045] F recovery1 represents the preliminary dimension lifting and refinement result, and F encoder represents the re-encoded feature obtained by the third encoding module, Upsampling() represents an up-sampling operation, Conv() represents a convolution operation, BatchNorm() represents batch normalization, and Relu() represents a Relu activation operation.

[0046] The present application also discloses an image blind restoration system based on self-information and prediction mask enhancement, comprising a computer system, which can implement the image blind restoration method based on self-information and prediction mask enhancement according to any one of the present application when executed.

[0047] Compared with the prior art, the image blind restoration method and system based on self-information and prediction mask enhancement provided by the technical solution have the following beneficial effects:

[0048] The application provides an image blind restoration method based on self-information and prediction mask enhancement.

[0049] The image blind restoration method has better robustness, and the restoration result of the damaged image has better visual effect. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description only constitute the embodiments of the application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0051] Figure 1 The application provides an image blind restoration method based on self-information and prediction mask enhancement.

[0052] Figure 2 The application provides an image blind restoration method based on self-information and prediction mask enhancement.

[0053] Figure 3 The application provides an image blind restoration method based on self-information and prediction mask enhancement.

[0054] Figure 4 The application provides an image blind restoration method based on self-information and prediction mask enhancement.

[0055] Figure 5 The application provides an image blind restoration method based on self-information and prediction mask enhancement. DETAILED DESCRIPTION

[0056] With reference to the drawings and embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0057] The embodiment of the present application discloses an image blind repair method based on self-information and prediction mask enhancement, as shown in the formula (1), which specifically comprises the following steps: Figure 1

[0058] The damaged image is converted into a feature token;

[0059] The first encoding module composed of the Transformer Block is used to encode the feature token to obtain an encoded feature;

[0060] The mask prediction is performed on the damaged image, and the information enhancement feature which only retains the pixels of the undamaged area is generated by using the prediction mask and the damaged image;

[0061] The second encoding module composed of the information enhancement Transformer Block is used to encode the encoded feature and the information enhancement feature to obtain an information enhancement encoded feature;

[0062] The third encoding module composed of the Transformer Block is used to re-encode the information enhancement encoded feature to obtain a re-encoded feature;

[0063] The re-encoded feature is restored to a multi-channel feature with the same size as the damaged image through the scale restoration and refinement module;

[0064] The multi-channel feature is channel filtered through the channel restoration to obtain a repaired image.

[0065] Each step of the present application will be described in detail below.

[0066] (1) First, in the embodiment of the present application, the feature conversion needs to be performed on the damaged image to be repaired, and the damaged image to be repaired is converted into a two-dimensional feature token which can be directly used by the model.

[0067] Suppose the damaged image is where c represents the number of channels, h and w represent the height and width of the image respectively. The image is dimensionally transformed to obtain Then, the encoded feature is obtained through linear mapping and other steps where e represents the encoding dimension, The specific process of this step is as follows:​

[0068] The damaged image I with dimensions h*w*c d Divided into several image blocks of the same size I' d Each image patch has a size of p*p*c, and h represents the damaged image I. d The height w represents the damaged image I. d p represents the width of the image block; c represents the number of channels.

[0069] Each image block I' d Flattened to p 2 *c is a one-dimensional feature vector; in a specific embodiment, the damaged image can be divided into image blocks with a size of 4*4*3, where 3 represents the number of channels c is 3. Then, the one-dimensional feature vector of each image block after flattening is a one-dimensional feature vector of 16*3.

[0070] A linear layer is used to project the flattened one-dimensional feature vectors onto a defined D-dimensional space to obtain each image patch I'. d The corresponding Patch Embedding sequence.

[0071] All image blocks I' d The Patch Embedding sequences are combined to obtain the Patch Embeddings sequence, which is the damaged image I. d The embedded sequence.

[0072] For damaged image I d A category token is added to the embedded sequence; position encoding is performed on the embedded sequence with the added category token to obtain the damaged image I. d The characteristic token.

[0073] (2) Next, the two-dimensional feature token obtained in the previous step is used as input, and the two-dimensional feature token is encoded by the first encoding module composed of Transformer Block to obtain the encoded feature.

[0074] In this embodiment of the invention, the first encoding module is composed of two identical and superimposed TransformerBlock units, which can be referred to as the first TransformerBlock unit and the second TransformerBlock unit, respectively. The first TransformerBlock unit is used to perform initial encoding on the feature token, and the second TransformerBlock unit is used to re-encode the initial encoding result of the first TransformerBlock unit to obtain the encoded features of the first encoding module.

[0075] The structure of the TransformerBlock unit in the present application is the same, as shown in Figure 2 The encoding process of the first TransformerBlock (TSBlock) unit for the initial encoding of the feature tokens is as follows:

[0076] F T1 =TSBlock(F embedding )

[0077] =MLP(Norm(MHA(Norm(F embedding ))+F embedding ))

[0078] wherein F T1 represents the initial encoding result obtained by the first TransformerBlock for the initial encoding of the feature tokens, TSBlock() represents the TransformerBlock unit processing operation, F embedding represents the feature tokens obtained by converting the damaged image, Norm() represents the normalization operation, MHA() represents the multi-head attention operation, and MLP() represents the multi-layer perception operation.

[0079] In the embodiments of the present application, the number of attention heads in the multi-head attention mechanism in the Transformer Block unit can be two or more.

[0080] (3) While encoding the feature tokens using the first encoding module, the damaged area mask prediction is performed with the damaged image to be repaired as the input, and the information enhancement feature retaining only the undamaged area is generated using the predicted mask and the damaged image.

[0081] The implementation of this process is as follows:

[0082] The trained deep learning model such as Mask-RCNN, U-Net, ConvMAE, etc. is used to perform mask prediction on the damaged image, and the predicted mask of the damaged area is obtained; the damaged area and the undamaged area in the damaged image are obtained according to the predicted mask of the damaged area; the pixel values of the undamaged area in the damaged image are retained, and the pixel values of the damaged area are replaced with a fixed value to obtain the information enhancement feature retaining only the pixel values of the undamaged area in the damaged image.

[0083] Specifically, with as the input, the deep learning model can first expand the input channel number through channel convolution, secondly encode the input feature through four cascaded downsampling operations, then recover the mask prediction feature through four upsampling operations, further reduce the channel number through channel convolution, and finally obtain the predicted mask M preUltimately, the information enhancement feature F is obtained, which consists only of the features (pixel values) of the undamaged areas of the image itself. enhanced The formula is as follows:

[0084] F enhanced = (1-M) pre )×I d +M pre (4)

[0085] (4) The second encoding module, composed of information-enhanced TransformerBlock, is used to encode the feature F obtained by the first encoding module in step (2). CT1 The information enhancement feature F obtained in step (3) enhanced Encode the information to obtain information-enhanced coding features.

[0086] In this embodiment of the invention, the second encoding module composed of information-enhanced TransformerBlocks is composed of two structurally identical and superimposed information-enhanced TransformerBlock units, respectively denoted as the first information-enhanced TransformerBlock unit and the second information-enhanced TransformerBlock unit; Reference Figure 3 The first information enhancement TransformerBlock (TSBlock) unit encodes the coded feature F output by the first encoding module. CT1 Enhanced feature F by retaining only pixel values ​​from undamaged areas enhanced Perform initial encoding to obtain the initial encoding result F. IET1 Then, the second information-enhancing TransformerBlock (TSBlock) unit encodes the initial encoding result F obtained by the first information-enhancing TransformerBlock unit. IET1 and information enhancement feature F enhanced The second encoding module outputs the information-enhanced coding feature F. CT2 .

[0087] refer to Figure 3 The first information enhancement TransformerBlock unit encodes feature F. CT1 and information enhancement features F enhanced The initial encoding process can specifically include the following encoding steps:

[0088] F temp1 =MHA(Norm(F CT1 ))+F CT1 ;

[0089] F IET1= Linear(Cat(Norm(MLP(Norm(F temp1 ))+ F temp1 ), F enhanced ));

[0090] wherein, F temp1 represents the intermediate encoded feature, F IET1 represents the primary encoded result, F CT1 represents the encoded feature obtained by the first encoding module, F enhanced represents the information enhanced feature, Norm() represents the normalization operation, MHA() represents the multi-head attention operation, MLP() represents the multi-layer perception operation, Cat() represents the connection operation, and Linear() represents the linear mapping.

[0091] (5) After obtaining the information enhanced encoded feature F CT2 output by the second encoding module in step (4), a third encoding module composed of a TransformerBlock is used to re-encode the encoded feature with enhanced information to obtain a re-encoded feature F encoder .

[0092] In the embodiments of the present application, the third encoding module is composed of five TransformerBlock units which are the same in structure and stacked with each other; specifically, the first Transformer Block unit receives the encoded feature with enhanced information F CT2 output by the second encoding module; the output result of the last Transformer Block unit is taken as the output of the third encoding module to obtain the re-encoded feature F encoder ; in the middle Transformer Block units, the output result of the former Transformer Block unit is input into the latter Transformer Block unit. The specific structure and encoding process of each Transformer Block unit can refer to the first Transformer Block unit in the first encoding module, and only the input and output are different.

[0093] (6) After obtaining the re-encoded feature F encoder , a scale restoration and refinement module is used to restore the re-encoded feature to a multi-channel feature with the same size as the to-be-repaired damaged image.

[0094] The scale recovery and refinement module package in the embodiment of the application is composed of two scale recovery and refinement units which are the same in structure and are superimposed on each other, and are denoted as a first scale recovery and refinement unit and a second scale recovery and refinement unit respectively; wherein the first scale recovery and refinement unit is used to perform preliminary dimension lifting and refinement on the re-encoded features output by the third encoding module to obtain a preliminary dimension lifting and refinement result, and then the second scale recovery and refinement unit performs again dimension lifting and refinement on the preliminary dimension lifting and refinement result output by the first scale recovery and refinement unit to obtain the multi-channel features consistent with the size of the damaged mural image.

[0095] In the embodiment of the application, as shown in Figure 4 , the first scale recovery and refinement unit performs preliminary dimension lifting and refinement on the re-encoded features output by the third encoding module, and specifically includes the following operation process:

[0096] F recovery1 = Relu(BatchNorm(Conv(Upsampling(F encoder )))

[0097] Wherein, F recovery1 represents the preliminary dimension lifting and refinement result, F encoder represents the re-encoded features obtained by the third encoding module, Upsampling() represents the up-sampling operation, Conv() represents the convolution operation, BatchNorm() represents the batch normalization, and Relu() represents the Relu activation operation.

[0098] Then the second scale recovery and refinement unit which is the same in structure is used to perform again dimension lifting and refinement on the preliminary dimension lifting and refinement result F recovery1 , to obtain the multi-channel features F recovery consistent with the size of the damaged mural image.

[0099] In this step, first, the encoding features F encoder obtained in step (5) are recovered in scale by two times up-sampling, the channels are recovered and locally refined by convolution operation, and finally the gradient vanishing is relieved and the network generalization ability is improved by BatchNorm normalization and Relu activation operation.

[0100] (7) Finally, the multi-channel features are channel filtered through channel recovery to obtain the restored image. Specifically, the multi-channel features F recovery can be restored to the restored image I

[0101] In specific applications, the method described in the application can be realized by a technical machine system containing the above model.

[0102] In order to further show the superiority of the method of the present application, in the specific embodiments of the present application, the image blind repair method disclosed in the present application is compared with three other leading blind repair methods in the repair of a certain mural data set.

[0103] The objective experimental results are shown in Table 1, in which PSNR represents the peak signal-to-noise ratio, SSIM represents the structural similarity, and the higher the values of the above two indicators, the better the quality of the reconstructed image. L1 represents the difference between the repaired image and the original image, and the smaller the value, the better. By analyzing the data in the table, it can be seen that the present application has achieved the best value in the three indicators, and has better repair effect.

[0104] Table 1 Objective results of comparison experiments of the present application

[0105]

[0106] The subjective experimental results are shown in Figure 5 , in which Figure 5 (a) is a damaged mural image, Figure 5 (b), (c), (d) are respectively the repaired images of TransCNN-HAE, BOWN and DT three methods. Figure 5 (e) is the repaired image of the method of the present application. By comparison, it can be seen that the method disclosed in the present application has fewer artifacts, and the repair result has better visual consistency.

[0107] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0108] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A blind image inpainting method based on self-information and predictive mask enhancement, characterized in that, Includes the following steps: Convert damaged images into feature tokens; The feature token is encoded using a first encoding module composed of Transformer Blocks to obtain the encoded feature; A mask prediction is performed on a damaged image, and information enhancement features are generated using the predicted mask and the damaged image, retaining only the pixels in the undamaged areas. Specifically, the information enhancement features are obtained through the following steps: taking the damaged image as input, the deep learning model first expands the number of input channels through channel convolution, then encodes the input features through four cascaded downsampling operations, then recovers the mask prediction features through four upsampling operations, then reduces the number of channels through channel convolution, and finally obtains the predicted mask through threshold segmentation, ultimately obtaining information enhancement features that only contain the pixel features of the undamaged areas of the image itself. The second coding module, composed of information-enhanced Transformer Blocks, encodes the coding features and information-enhanced features to obtain information-enhanced coding features; The information enhancement coding features are recoded using a third coding module composed of Transformer Blocks to obtain recoded features; The recoded features are restored to multi-channel features of the same size as the damaged image through the scale restoration and thinning module; The multi-channel features are then subjected to channel filtering through channel restoration to obtain the repaired image.

2. The image blind inpainting method based on self-information and predictive mask enhancement according to claim 1, characterized in that, Converting damaged images into feature tokens specifically includes: The damaged image I with dimensions h*w*c d Divided into several image blocks of the same size I' d Each image patch has a size of p*p*c, and h represents the damaged image I. d The height of w indicates the damaged image I. d The width; p represents the height or width of the image block; c represents the number of channels; Each image block I' d Flattened to p 2 *c is a one-dimensional feature vector; A linear layer is used to project the flattened one-dimensional feature vectors onto a defined D-dimensional space to obtain each image patch I'. d The corresponding Patch Embedding sequence; All image blocks I' d The Patch Embedding sequences are combined to obtain the Patch Embeddings sequence, which is the damaged image I. d Embedded sequences; For damaged image I d Add a category token to the embedded sequence; Position encoding is performed on the embedded sequence of the added category token to obtain the damaged image I. d The characteristic token.

3. The image blind inpainting method based on self-information and predictive mask enhancement according to claim 1, characterized in that, In the step of encoding the feature token using a first encoding module composed of Transformer Blocks, the first encoding module includes two Transformer Block units with the same structure and superimposed on each other, referred to as the first Transformer Block unit and the second Transformer Block unit. The first Transformer Block unit is used to perform initial encoding of the feature token.

4. The image blind inpainting method based on self-information and predictive mask enhancement according to claim 3, characterized in that, The first Transformer Block unit performs initial encoding on the feature token, specifically including the following encoding process: F T1 =TSBlock(F embedding ) =MLP(Norm(MHA(Norm(F embedding ))+F embedding )) Among them, F T1 This indicates the initial encoding result; TSBlock() represents the Transformer Block unit processing operation; F embedding This represents the feature token obtained by transforming the damaged image. Norm() represents the normalization operation, MHA() represents the multi-head attention operation, and MLP() represents the multilayer perceptron operation.

5. The image blind inpainting method based on self-information and predictive mask enhancement according to claim 1, characterized in that, The damaged image is subjected to damaged region mask prediction. The predicted mask and the damaged image are then used to generate information enhancement features that retain only the undamaged regions. Specifically, this includes: A trained deep learning model is used to predict the mask of the damaged image, and the predicted mask of the damaged area is obtained. The damaged and undamaged areas in the damaged image are obtained based on the predicted mask of the damaged area. By retaining the pixel values ​​of the undamaged areas in the damaged image and replacing the pixel values ​​of the damaged areas with fixed values, we obtain information enhancement features in the damaged image that retain only the pixel values ​​of the undamaged areas.

6. The image blind inpainting method based on self-information and predictive mask enhancement according to claim 1, characterized in that, In the step of encoding the encoded features and the information-enhanced features using the second encoding module composed of information-enhanced Transformer Blocks, the second encoding module includes two information-enhanced Transformer Block units with the same structure and superimposed on each other, referred to as the first information-enhanced Transformer Block unit and the second information-enhanced Transformer Block unit; The first information-enhancing Transformer Block unit is used to perform initial encoding of the encoding features and information-enhancing features, and the second information-enhancing Transformer Block unit is used to re-encode the initial encoding result of the first information-enhancing Transformer Block unit and the information-enhancing features.

7. The image blind inpainting method based on self-information and predictive mask enhancement according to claim 6, characterized in that, The first information-enhancing Transformer Block unit performs initial encoding of the encoded features and information-enhancing features, specifically including the following encoding process: F temp1 =MHA(Norm(F CT1 ))+F CT1 ; F IET1 =Linear(Cat(Norm(MLP(Norm(F temp1 ))+F temp1 ) ,F enhanced )); Among them, F temp1 F represents the intermediate encoded features. IET1 F represents the initial encoding result. CT1 This indicates that the first encoding module obtains the encoded features, F enhanced The symbols represent information enhancement features, Norm() represents normalization operation, MHA() represents multi-head attention operation, MLP() represents multilayer perceptron operation, Cat() represents connection operation, and Linear() represents linear mapping.

8. The image blind inpainting method based on self-information and predictive mask enhancement according to claim 1, characterized in that, In the step of recoding the information enhancement coding features using a third coding module composed of Transformer Blocks, the third coding module includes five Transformer Block units with identical structures that are superimposed on each other.

9. The image blind inpainting method based on self-information and predictive mask enhancement according to claim 1, characterized in that, In the step of restoring the recoded features to multi-channel features of the same size as the damaged image through the scale restoration and thinning module, the scale restoration and thinning module includes two scale restoration and thinning units with the same structure and superimposed on each other, referred to as the first scale restoration and thinning unit and the second scale restoration and thinning unit. The first scale recovery and refinement unit is used to perform preliminary dimensionality enhancement and refinement on the recoded features output by the third encoding module, and obtain preliminary dimensionality enhancement and refinement results.

10. The image blind inpainting method based on self-information and predictive mask enhancement according to claim 9, characterized in that, The first scale recovery and refinement unit performs preliminary dimensionality enhancement and refinement on the recoded features output by the third encoding module, specifically including the following operations: F recovery1 =Relu(BatchNorm(Conv(Upsampling(F encoder )))) Among them, F recovery1 This indicates the initial dimensional enhancement and refinement results, F encoder This represents the recoded features obtained by the third encoding module. Upsampling() represents the upsampling operation, Conv() represents the convolution operation, BatchNorm() represents batch normalization, and Relu() represents the Relu activation operation.

Images