A generative image restoration method

Abstract

The present invention relates to a generative image restoration method, which uses two conditional variational autoencoders CVAE, namely CVAE V h and CVAE V k , respectively learn object segmentation hypothesis and uncertain feature map for image restoration, where the object segmentation hypothesis CVAE V is used h Learning the probability distribution of object categories; based on the object segmentation hypothesis CVAE V h , using CVAE V k Learn the probability distribution of the convolution kernel, obtain the convolution kernel and the uncertain feature map, and use the uncertain feature map to repair the image.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and computer graphics, relates to a generative network, and specifically is a generative image restoration method. Background Art

[0002] Image inpainting involves filling in missing or damaged areas of an image based on existing regions, ensuring that the inpainted image is consistent with the original. Image inpainting has a wide range of applications in areas such as object removal, digital cultural heritage preservation, and image editing. Accurately understanding object categories is crucial for image inpainting. These categories capture the visual correlations between image regions and facilitate inferring the content of damaged areas.

[0003] Traditional image restoration methods use appearance features such as color and gradient to measure the similarity between image regions. These methods assume that objects around the same region have high similarity and exchange information around these regions to restore nearby damaged regions[1][2].

[0004] Relying solely on appearance features is not enough to deal with complex image scenes, so high-level semantic features are applied to image restoration [3][4][5] to make the restoration results more complete in terms of semantics. In recent years, deep learning-based methods have used convolutional neural networks to predict object segmentation maps. Object segmentation maps explicitly provide categories for learning semantic features and better represent the complex relationships between categories. However, the damaged areas provide limited object category information, and the semantic features generated by them are uncertain, which brings challenges to image restoration tasks.

[0005] References:

[0006] [1]Barnes C, Shechtman E, Finkelstein A, et al. PatchMatch: A randomized correspondence algorithm for structural image editing[J]. ACM Trans. Graph., 2009, 28(3):24.

[0007] [2]Bertalmio M, Sapiro G, Caselles V, et al.Image inpainting[C] / / Proceedings of the 27th annual conference on Computer graphics and interactive techniques.2000:417-424.

[0008] [3]Yu J,Lin Z,Yang J,et al.Generative image inpainting with contextual attention[C] / / Proceedings of the IEEE conference on computervision and pattern recognition.2018:5505-5514.

[0009] [4]Chen Q, Koltun V.Photographic image synthesis with cascadedrefinement networks[C] / / Proceedings of the IEEE international conference oncomputer vision.2017:1511-1520.

[0010] [5]Song Y, Yang C, Shen Y, et al. Spg-net: Segmentation prediction and guidance network for image inpainting[J]. arXiv preprint arXiv:1805.03356, 2018. Summary of the Invention

[0011] This paper addresses the problems existing in existing image restoration tasks and proposes a generative image restoration method based on a conditional variational autoencoder (CVAE). The technical solution is as follows:

[0012] A generative image restoration method using two conditional variational autoencoders CVAE, namely CVAE V h and CVAE V k , respectively learn object segmentation hypothesis and uncertain feature map for image restoration, where the object segmentation hypothesis CVAE V is used h Learning the probability distribution of object categories; based on the object segmentation hypothesis CVAE V h , using CVAE V k Learning the probability distribution of the convolution kernel, obtaining the convolution kernel and the uncertain feature map, and using the uncertain feature map to repair the image includes the following steps:

[0013] S1, using CVAE V h Learning the conditional probability distribution of object categories, that is, learning object segmentation hypotheses based on feature maps and obtaining category uncertainty maps, CVAE Vh It consists of an encoder branch, a conditional branch, and a decoder branch; the feature map F and the reference segmentation map S′∈R H×W×N Figure input to CVAE V h , output object segmentation hypothesis set {S l ∈R H×W×N |l=1,…,L} and uncertain feature map U∈R H×W×N , the steps are as follows:

[0014] S11, input image to be repaired I∈R H×W×3 Extract the feature map and obtain the feature map F∈R H×W×C ;

[0015] S12, input the feature map F and the reference segmentation map S′ into the CVAE V h In the encoder branch, the mean μ is obtained s ∈R H ×W×C and standard deviation σ s ∈R H×W×C :

[0016] [μ s ,σ s ]=encoder([S′,F])

[0017] Where C represents the number of channels, [·] represents the accumulation along the channel dimension; μ s ,σ s Constitute Gaussian distribution one;

[0018] S13, input the feature map F into CVAE V h In the conditional branch, we get the mean μ f ∈R H×W×C and standard deviation σ f ∈R H ×W×C :

[0019] [μ f ,σ f ]=condition(F)

[0020] Among them, μ f ,σ f Constitute Gaussian distribution II;

[0021] S14, sample from the Gaussian distribution and generate a latent variable map Z∈R of size (H, W, C) H×W×C , where Z follows a Gaussian distribution;

[0022] S15, the mean μ of the segmentation map obtained in S12 s and standard deviation σ sPerform normalization, superimpose the normalized result with the feature map F, and input it into CVAE V h In the decoder branch, the output object segmentation hypothesis set {S l ∈R H ×W×N |l=1,…,L}:

[0023]

[0024] S l = decoder([F, normalization result])

[0025] Among them, L represents the number of times Z is generated, that is, the number of sampling times, Z∈R H×W×C is a latent variable graph generated by Gaussian distribution. Step S15 is performed for each Z to obtain L object segmentation hypotheses, thereby obtaining the object segmentation hypothesis set {S l ∈R H×W×N |l=1,…,L}, N represents the number of categories;

[0026] S16, the object segmentation hypothesis set {S l ∈R H×W×N The object segmentation hypothesis S in |l=1,…,L} l Find the L2 norm and get the category uncertainty graph U∈R H×W×N :

[0027]

[0028] in, is the normalization parameter, ‖·‖ 2 represents the L2 norm;

[0029] S2, using the cross entropy loss function to calculate the loss value between the object segmentation hypothesis and the reference segmentation map, using KL divergence to calculate the difference between the two Gaussian distributions, training CVAE V h ;

[0030] S3, decompose the feature map F as follows:

[0031] S31, segment the object and assume S l Perform point multiplication with the feature map F to obtain the feature map F′ corresponding to each category n n ∈R H ×W×C , and then perform convolution operation to obtain the convolution kernel K′ corresponding to each category n n ∈R H×W×(S×S×C) , n=1,…,N:

[0032] F′ n,x,y =S x,y,n·F x,y

[0033] K′ n =conv(F′ n )

[0034] Among them, F′ n,x,y ∈R C Represents the feature vector of the nth category at pixel (x, y), S x,y,n ∈R represents the probability that the pixel (x, y) belongs to the nth category, F x,y ∈R C The feature vector representing the pixel point (x, y);

[0035] S32, the category uncertainty map U is combined with the convolution kernel K′ corresponding to each category n Perform point multiplication to obtain the convolution kernel corresponding to each category, and then add the convolution kernels corresponding to these categories to obtain the uncertain convolution kernel K u :

[0036]

[0037] in, Represents a convolution kernel with a resolution of S×S and a number of channels of C at the pixel point (x, y), U x,y,n Represents the uncertainty of the pixel being in the nth category, with a value between 0 and 1;

[0038] S4, using CVAE V k Learn the probability distribution of convolution kernels, obtain convolution kernels and uncertain feature maps, CVAE V k It consists of an encoder branch, a conditional branch, and a decoder branch, which takes the object segmentation hypothesis S, the category uncertainty map U, the feature map, and the uncertain convolution kernel map K u Input to CVAE V k In the convolution kernel K∈R H×W×(S×S×C) , the steps are as follows:

[0039] S41, the uncertain convolution kernel map K u , object segmentation hypothesis S l , the category uncertainty map U and feature map F are input to the generative network CVAE V k In the encoder branch, the mean μ is obtained k ∈R H×W×C and standard deviation σ k ∈R H×W×C :

[0040] [μ k ,σ k ]=encoder([K u ,S l,U,F])

[0041] Among them, μ k ,σ k Constitute Gaussian distribution three;

[0042] S42, segment the object and assume S l , the category uncertainty map U and feature map F are input to CVAE V k In the conditional branch, we get the mean μ c ∈R H×W×C and standard deviation σ c ∈R H×W×C :

[0043] [μ c ,σ c ]=condition([S l ,U,F])

[0044] Among them, μ c ,σ c Constitute the Gaussian distribution four;

[0045] S43, Sampling from Gaussian distribution: Generate a latent variable map Z∈R of size (H,W,C) H×W×C , where Z follows a Gaussian distribution;

[0046] S44, use Gaussian distribution to normalize the mean and standard deviation obtained in S41, and compare the normalized result with the feature map F and the object segmentation hypothesis S l , the category uncertainty map U is superimposed and input into CVAE V together k In the decoder branch of , the output convolution kernel K∈R H×W×(S×S×C) :

[0047]

[0048] K=decoder([S l ,U,F,normalization result])

[0049] S45, convolve the convolution kernel K and the feature map F to obtain the uncertain feature map Q∈R H×W×3 :

[0050] Q = conv(K,F)

[0051] S46, perform convolution operation on the uncertain feature map to obtain the repaired result O∈R H×W×3 ;

[0052] S5, use L2 loss function to calculate convolution kernel K and uncertain convolution kernel map K uThe loss value between them is calculated using KL divergence to calculate the difference between Gaussian distribution three and four, and train CVAE V k .

[0053] The present invention uses two generative networks. The first one learns the probability distribution of object categories, generates multiple object segmentation hypotheses through sampling, and uses the differences between the hypotheses to represent the scores of object category judgments. The differences in the object segmentation hypotheses are used to decompose the convolution kernels according to category. The second generative network learns the distribution of convolution kernels to more accurately capture category features. Experiments have shown that the two generative networks achieve better image restoration results. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of a generative image restoration method;

[0055] Figure 2 This is a comparison between the present invention and other existing best image restoration methods. DETAILED DESCRIPTION

[0056] The technical solution of the present invention is described clearly and completely below with reference to the accompanying drawings. Based on the technical solution of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] High-quality image restoration needs to consider the association of object categories in the missing area. Advanced image restoration methods based on deep learning mainly focus on how they learn semantic features from visual data to represent object categories. Typically, existing methods learn semantic features from objects and use them to predict damaged areas. However, the uncertain categories in the damaged area may mislead the learning of features. The generative image restoration method of the present invention learns the probability distribution of object categories, generates a segmentation hypothesis for capturing the object category, and uses this hypothesis as a condition to further learn the distribution of the convolution kernel. The convolution kernel thus generated captures the object category information of the damaged area. The convolution kernel is used to extract uncertain features to restore the damaged area.

[0058] The generative image restoration method of the present invention specifically comprises the following steps:

[0059] The first step is to train CVAE V separately h and CVAE V k .

[0060] 1. Use CVAE V h Learn the conditional probability distribution of object categories, that is, learn the object segmentation hypothesis based on the feature map and obtain the category uncertainty map.h It consists of an encoder branch, a conditional branch, and a decoder branch, each of which consists of 16 convolutional layers. H×W×N Figure input to CVAE V h , output object segmentation hypothesis set {S l ∈R H×W×N |l=1,…,L} and uncertain feature map U∈R H×W×N The specific steps are as follows:

[0061] (1) Use the convolutional layer to input the image to be repaired I∈R H×W×3 Extract the feature map and obtain the feature map F∈R H×W×C .

[0062] (2) Input the feature map F and the reference segmentation map S′ into the CVAE V h In the encoder branch, the mean μ is obtained s ∈R H ×W×C and standard deviation σ s ∈R H×W×C :

[0063] [μ s ,σ s ]=encoder([S′,F])

[0064] Where C represents the number of channels and [·] represents the accumulation along the channel dimension. s ,σ s Constitute a Gaussian distribution.

[0065] (3) Input the feature map F into CVAE V h In the conditional branch, we get the mean μ f ∈R H×W×C and standard deviation σ f ∈R H ×W×C :

[0066] [μ f ,σ f ]=condition(F)

[0067] Among them, μ f ,σ f Constitute Gaussian distribution 2.

[0068] (4) Sampling from Gaussian distribution: Generate a latent variable map Z∈R of size (H, W, C) H×W×C , where Z follows a Gaussian distribution.

[0069] (5) The mean μ of the segmentation map obtained in (2) sand standard deviation σ s Perform normalization, superimpose the normalized result with the feature map F, and input it into CVAE V h In the decoder branch, the output object segmentation hypothesis set {S l ∈R H×W×N |l=1,…,L}:

[0070]

[0071] S l = decoder([F, normalization result])

[0072] Among them, L represents the number of times Z is generated, that is, the number of sampling times, Z∈R H×W×C is a latent variable graph generated by Gaussian distribution. By performing step (5) on each Z, we can obtain L object segmentation hypotheses, thus obtaining the object segmentation hypothesis set {S l ∈R H×W×N |l=1,…,L}.

[0073] (6) Object segmentation hypothesis set {S l ∈R H×W×N The object segmentation hypothesis S in |l=1,…,L} l ∈R H×W×N , l=1,…,L to find the L2 norm and obtain the category uncertainty graph U∈R H×W×N .

[0074]

[0075] Among them, N represents the number of categories, L represents the number of sampling times, S i represents the i-th hypothesis graph, is the normalization parameter, ‖·‖ 2 represents the L2 norm.

[0076] 2. Use the cross entropy loss function to calculate the loss value between the object segmentation hypothesis and the reference segmentation map, and use the KL divergence to calculate the difference between the two distributions (2) and (3) in 1.

[0077] 3. Perform information decomposition on the feature map F.

[0078] (1) Assume that the object is segmented l Perform point multiplication with the feature map F to obtain the feature map F′ corresponding to each category n ∈R H ×W×C , and then perform convolution operation to obtain the convolution kernel K′ corresponding to each category n ∈R H×W×(S×S×C) , n=1,…,N.

[0079] F′ n,x,y =S x,y,n ·F x,y

[0080] K′ n =conv(F′ n )

[0081] Among them, F′ n,x,y ∈R C Represents the feature vector of the nth category at pixel (x, y), S x,y,n ∈R represents the probability that the pixel (x, y) belongs to the nth category, F x,y ∈R C The feature vector representing the pixel point (x,y).

[0082] (2) Combine the category uncertainty graph U with the convolution kernel K′ corresponding to each category n Perform point multiplication to obtain the convolution kernel corresponding to each category, and then add the convolution kernels corresponding to these categories to obtain the uncertain convolution kernel K u .

[0083]

[0084] in, Represents a convolution kernel with a resolution of S×S and a number of channels of C at the pixel point (x, y), U x,y,n Represents the uncertainty of the pixel belonging to the nth category, with a value between 0 and 1.

[0085] 4. Training CVAE V k Using CVAE V k Learn the probability distribution of the convolution kernel and obtain the convolution kernel and the uncertain feature map. k It consists of an encoder branch, a conditional branch, and a decoder branch, each of which consists of 16 convolutional layers. The object segmentation hypothesis S, the category uncertainty map U, the feature map, and the uncertainty convolution kernel map K are combined. u Input to CVAE V k In the convolution kernel K∈R H×W×(S×S×C) , the specific steps are as follows:

[0086] (1) The uncertain convolution kernel K u , object segmentation hypothesis S l , the category uncertainty map U and feature map F are input to the generative network CVAE V k In the encoder branch, the mean μ is obtained k ∈R H×W×C and standard deviation σ k ∈R H×W×C :

[0087] [μ k ,σ k ]=encoder([K u ,S l ,U,F])

[0088] Among them, μ k ,σ k Constitute the Gaussian distribution three.

[0089] (2) Assume that the object is segmented l , the category uncertainty map U and feature map F are input to CVAE V k In the conditional branch, we get the mean μ c ∈R H×W×C and standard deviation σ c ∈R H×W×C :

[0090] [μ c ,σ c ]=condition([S l ,U,F])

[0091] Among them, μ c ,σ c Constitute the Gaussian distribution four.

[0092] (3) Sampling from Gaussian distribution: Generate a latent variable map Z∈R of size (H, W, C) H×W×C , where Z follows a Gaussian distribution.

[0093] (4) Use Gaussian distribution to normalize the mean and standard deviation obtained in (1), and compare the normalized result with the feature map F and the object segmentation hypothesis S. l , the category uncertainty map U is superimposed and input into CVAE V together k In the decoder branch of , the output convolution kernel K∈R H×W×(S×S×C) .

[0094]

[0095] K=decoder([S l ,U,F,normalization result])

[0096] (5) Convolve the convolution kernel K and the feature map F to obtain the uncertain feature map Q∈R H×W×3 .

[0097] Q = conv(K,F)

[0098] (6) The uncertain feature map is convolved to obtain the repaired result O∈R H×W×3 .

[0099] 5. Use L2 loss function to calculate convolution kernel K and uncertain convolution kernel map K u The loss value between , is calculated using KL divergence, the difference between the two distributions (1) and (2) in 4.

[0100] 6. Use SSIM to calculate the difference between the repaired result O and the reference repair image R, and use ADV loss to measure the unreality of the repaired result.

[0101] The second step is to test the image restoration effect of the entire network.

[0102] 1. Remove CVAE V h and CVAE V k The encoder branch only retains the conditional branch and the decoder branch. First, input the image to be repaired I∈R H×W×3 , extract the feature map F∈R through the convolution layer H×W×C .

[0103] 2. Input the feature map F into CVAE V h In the conditional branch, we get the mean μ f ∈R H×W×C and standard deviation σ f ∈R H ×W×C .

[0104] [μ f ,σ f ]=condition(F)

[0105] 3. Sample from a Gaussian distribution and calculate the mean μ in 2. f and standard deviation σ f Normalize. Superimpose the normalized result with the feature map F and input it into CVAE V h In the decoder branch, the output object segmentation hypothesis S l .

[0106]

[0107] S l = decoder([F, normalization result])

[0108] 4. Object segmentation hypothesis set {S l ∈R H×W×N The object segmentation hypothesis S in |l=1,…,L} l Find the L2 norm and get the category uncertainty graph U∈R H×W×N .

[0109]

[0110] 5. Combine feature map F and object segmentation hypothesis S l , and the category uncertainty map U is superimposed and input into CVAE V respectively. k In the conditional branch, we get the mean μ c ∈R H×W×C and standard deviation σ c ∈R H×W×C .

[0111]

[0112] 6. Sample from a Gaussian distribution and calculate the mean μ in 5. c and standard deviation σ c Normalize. Combine the normalized result with the feature map F and the object segmentation hypothesis S l , superimpose the uncertainty map U and input it into CVAE V k In the decoder branch, we get the convolution kernel set {K l ∈R H×W×(S×S×C) |l=1,…,L}.

[0113]

[0114] K l =decoder([S l ,U,F,normalization result])

[0115] 7. Convolution kernel K l ∈R H×W×(S×S×C) With the feature map F∈R H×W×C Perform convolution operation to obtain the uncertain feature map Q∈R H×W×3 , and then perform convolution operation to obtain the image restoration result R l ∈R H×W×3 .

[0116] 8. Generate multiple repair results {R l ∈R H×W×3 |l=1,…,L}, take the sum and average, and get the final repair result O∈R H×W×3 .

[0117] The datasets used by the present invention to verify the feasibility of the experiment are the Cityscapes dataset and the Outdoor Scenes dataset. Each picture in the dataset has a corresponding reference segmentation map for training the network. Among them, the Cityscapes dataset provides 20 object categories, the number of pictures used for training is 2975, and the number of pictures used for testing is 1525. The OutdoorScenes dataset contains 8 categories, 9900 pictures for training, and 300 pictures for testing. The present invention mainly uses pictures containing irregular missing areas for experiments, and the proportion of missing areas in the pictures is between 20% and 40%. At the same time, regular and random missing areas are used for comparison of methods. The present invention uses three indicators: peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and Frechet inception distance (FID) to measure the results of the experiment. The larger the values ​​of PSNR and SSIM, the smaller the value of FID, which means the better the experimental effect. According to Figure 2 The comparison between the presented method and the best existing image restoration method shows that the method of the present invention has improvements in all indicators, verifying the feasibility of the method of the present invention.

Claims

1. A generative image restoration method using two conditional variational autoencoders CVAE, namely CVAE V h and CVAEV k , respectively learn object segmentation hypothesis and uncertain feature map for image restoration, where, Using object segmentation hypothesis CVAE V h Learning the probability distribution of object categories; based on the object segmentation hypothesis CVAE V h , using CVAE V k Learning the probability distribution of the convolution kernel, obtaining the convolution kernel and the uncertain feature map, and using the uncertain feature map to repair the image includes the following steps: S1, using CVAE V h Learning the conditional probability distribution of object categories, that is, learning object segmentation hypotheses based on feature maps and obtaining category uncertainty maps, CVAE V h It consists of an encoder branch, a conditional branch, and a decoder branch; the feature map F and the reference segmentation map S′∈R H×W×N Figure input to CVAE V h , output object segmentation hypothesis set {S l ∈R H×W×N |l=1,…,L} and uncertain feature map U∈R H×W×N , the steps are as follows: S11, input image to be repaired I∈R H×W×3 Extract the feature map and obtain the feature map F∈R H×W×C ; S12, input the feature map F and the reference segmentation map S′ into CVAE V h In the encoder branch, the mean μ is obtained s ∈R H×W×C and standard deviation σ s ∈R H×W×C : [m s ,s s ]=encoder([S′,F]) Where C represents the number of channels, [·] represents the accumulation along the channel dimension; μ s ,σ s Constitute Gaussian distribution one; S13, input the feature map F into CVAE V h In the conditional branch, we get the mean μ f ∈R H×W×C and standard deviation σ f ∈R H×W×C : [m f ,s f ]=condition(F) Among them, μ f ,σ f Constitute Gaussian distribution II; S14, sample from the Gaussian distribution and generate a latent variable map Z∈R of size (H, W, C) H×W×C , where Z follows a Gaussian distribution; S15, the mean μ of the segmentation map obtained in S12 s and standard deviation σ s Perform normalization, superimpose the normalized result with the feature map F, and input it into CVAE V h In the decoder branch, the output object segmentation hypothesis set {S l ∈R H×W×N |l=1,…,L}: S l = decoder([F, normalization result]) Among them, L represents the number of times Z is generated, that is, the number of sampling times, Z∈R H×W×C is a latent variable graph generated by Gaussian distribution. Step S15 is performed for each Z to obtain L object segmentation hypotheses, thereby obtaining the object segmentation hypothesis set {S l ∈R H×W×N |l=1,…,L}, N represents the number of categories; S16, the object segmentation hypothesis set {S l ∈R H×W×N The object segmentation hypothesis S in |l=1,…,L} l Find the L2 norm and get the category uncertainty graph U∈R H×W×N : in, is the normalization parameter, ‖·‖ 2 represents the L2 norm; S2, using the cross entropy loss function to calculate the loss value between the object segmentation hypothesis and the reference segmentation map, using KL divergence to calculate the difference between the two Gaussian distributions, training CVAE V h ; S3, decompose the feature map F as follows: S31, segment the object and assume S l Perform point multiplication with the feature map F to obtain the feature map F′ corresponding to each category n n ∈R H×W×C , and then perform convolution operation to obtain the convolution kernel K′ corresponding to each category n n ∈R H×W×(S×S×C) , n=1,…,N: F′ n,x,y =S x,y,n ·F x,y K′ n =conv(F′ n ) Among them, F′ n,x,y ∈R C Represents the feature vector of the nth category at pixel (x, y), S x,y,n ∈R represents the probability that the pixel (x, y) belongs to the nth category, F x,y ∈R C The feature vector representing the pixel point (x, y); S32, the category uncertainty map U is combined with the convolution kernel K′ corresponding to each category n Perform point multiplication to obtain the convolution kernel corresponding to each category, and then add the convolution kernels corresponding to these categories to obtain the uncertain convolution kernel K u : in, Represents a convolution kernel with a resolution of S×S and a number of channels of C at the pixel point (x, y), U x,y,n Represents the uncertainty of the pixel being in the nth category, with a value between 0 and 1; S4, using CVAE V k Learn the probability distribution of convolution kernels, obtain convolution kernels and uncertain feature maps, CVAE V k It consists of an encoder branch, a conditional branch, and a decoder branch, which takes the object segmentation hypothesis S, the category uncertainty map U, the feature map, and the uncertain convolution kernel map K u Input to CVAE V k In the convolution kernel K∈R H×W×(S×S×C) , the steps are as follows: S41, the uncertain convolution kernel map K u , object segmentation hypothesis S l , the category uncertainty map U and feature map F are input to the generative network CVAE V k In the encoder branch, the mean μ is obtained k ∈R H×W×C and standard deviation σ k ∈R H×W×C : [μ k ,σ k ]=encoder([K u ,S l ,U,F]) Among them, μ k ,σ k Constitute Gaussian distribution three; S42, segment the object and assume S l , the category uncertainty map U and feature map F are input to CVAE V k In the conditional branch, we get the mean μ c ∈R H×W×C and standard deviation σ c ∈R H×W×C : [m c ,s c ]=condition([S l ,U,F]) Among them, μ c ,σ c Constitute the Gaussian distribution four; S43, Sampling from Gaussian distribution: Generate a latent variable map Z∈R of size (H,W,C) H×W×C , where Z follows a Gaussian distribution; S44, use Gaussian distribution to normalize the mean and standard deviation obtained in S41, and compare the normalized result with the feature map F and the object segmentation hypothesis S l , the category uncertainty map U is superimposed and input into CVAE V together k In the decoder branch of , the output convolution kernel K∈R H×W×(S×S×C) : K=decoder([S l ,U,F,normalization result]) S45, convolve the convolution kernel K and the feature map F to obtain the uncertain feature map Q∈R H×W×3 : Q = conv(K,F) S46, perform convolution operation on the uncertain feature map to obtain the repaired result O∈R H×W×3 ; S5, use L2 loss function to calculate convolution kernel K and uncertain convolution kernel map K u The loss value between them is calculated using KL divergence to calculate the difference between Gaussian distribution three and four, and train CVAE V k .

Images