A GAN image restoration method based on stable field skip connection

Abstract

The application discloses a GAN image repairing method based on stable field jump connection, embeds a stable field operator into a jump connection, uses the stable field operator to preliminarily predict damaged pixels of an encoder feature map in a generation unit, then transfers to a decoder and outputs a generated image. Then, the type of loss is refined, the loss function is redefined as an adversarial loss, a pixel reconstruction loss and a pyramid loss to further guide the training of the network to the correct direction. The method improves the effect of damaged image repairing, meets the demand of human visual perception, and is similar to the original image to a certain extent. And experiments prove that the method achieves good repairing effect on face images, natural scene images and building images.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of damaged image inpainting, and particularly relates to a GAN image inpainting method based on stable field jump connection. BACKGROUND

[0002] Image inpainting is a challenging research topic in the field of computer vision, which fills or corrects the missing pixels in the image by using the prior information in the damaged image or training data, and expects to be as close as possible to the original image under the premise of meeting the demand of human visual perception. Figure 1 In recent years, due to the development of deep learning technology, the performance of image inpainting has been significantly improved. Compared with the traditional image inpainting method, the image inpainting method based on generative adversarial network (GAN) can synthesize more visually realistic and semantically reasonable pixels for the missing area.

[0003] Currently, there is a wide range of research on GAN-based image restoration, with researchers improving GANs in various areas. Yu et al. used GANs to achieve coarse-to-fine image restoration, introducing an attention mechanism during the fine restoration stage. They proposed a context-aware layer that searches for the most similar feature blocks in known regions to those in the missing region (Yu J, Lin Z, Yang J, et al. Generative image inpainting with contextual attention [C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE Computer Society Press, Los Alamitos, 2018: 5505-5514). This method proposed a novel contextual attention layer, which resulted in restored images with a reasonable visual structure, but often produced distorted structures and blurred textures. Furthermore, the method could only repair centered rectangular damaged regions, failing to repair irregular damaged regions. Zhang et al. used multiple identical U-Net structures (Ronneberger O, Fischer P, Brox TU-Net: convolutional networks for biomedical image segmentation [C]. Proceedings of the Medical Image Computing and Computer-Assisted Intervention. Springer, Heidelberg, 2015: 234-241) to implement progressive restoration of damaged images (Zhang HR, Hu ZZ, Luo CZ, et al. Semantic Image Inpainting with Progressive Generative Networks [C]. Proceedings of the 26th ACM international conference on Multimedia. Association for Computing Machinery, New York, 2018: 1939-1947). However, this method also cannot repair irregular damaged areas.Liu et al. proposed partial convolution to repair irregular damaged regions in images by updating the damaged region of each layer and normalizing the convolution weight with the value of the region (Liu G L, Reda F A, Shih K J, et al. Image inpainting for irregular holes using partial convolutions [C]. Proceedings of European Conference on Computer Vision. Springer, Heidelberg, 2018: 89-105). This method proposed partial convolution instead of standard convolution, effectively solving the problems of artifacts, color difference, blur and other problems in the image inpainting results when facing irregular damaged regions, but the handcrafted feature normalization in the update of the damaged region of each layer is difficult to train, and the repair effect is not ideal when facing larger central rectangular damaged regions. Xie et al. improved the two problems of handcrafted feature normalization and missing region update considering only forward propagation in the above partial convolution, and proposed to learn feature standardization and damaged region update in an end-to-end manner (Xie C H, Liu S H, Li C, et al. Image Inpainting With Learnable Bidirectional Attention Maps [C]. Proceedings of the IEEE International Conference on Computer Vision. IEEE Computer Society Press, Los Alamitos, 2019: 8857-8866). This method can effectively adapt to the forward propagation of irregular damaged regions, and the repair effect of damaged images is also improved to a certain extent compared with the previous two methods, but the structure distortion of the repair result exists. Although these several typical image inpainting methods have their own advantages, they all have a common shortcoming, that is, they do not consider the similarity matching between the damaged region and the known region and the correlation between the internal feature repair pixels of the damaged region at the same time, resulting in the repair result may produce texture blur and structure distortion. From the model structure level, it is because in the field of image inpainting, the input is the damaged image to be repaired, and for the center of the damaged region, the input of the skip-connection in U-Net is almost zero, which cannot transmit the texture information to the decoder, resulting in the repair result producing texture blur and structure distortion.From a semantic perspective, this is because these methods fail to meet the requirements of global semantic consistency and local feature continuity, that is, they ignore the semantic consistency and feature continuity of the damaged area, resulting in semantic incoherence in the damaged area of ​​the repair result.

[0004] Stable field operator: Ye Xueyi, Qi Zhenzhen, He Zhiwei, et al. Local region reconstruction of image field directional derivatives[J]. Journal of Image and Graphics, 2014, 19(7): 998-1005. Summary of the Invention

[0005] In view of the fact that existing image restoration methods often ignore the semantic consistency and feature continuity of damaged areas, resulting in the image restoration results may produce content with blurred texture and distorted structure, the present invention provides a GAN image restoration method based on stable field jump connection, which is used to maintain the semantic consistency and feature continuity of damaged areas, and improve the situation where image restoration results may produce content with blurred texture and distorted structure.

[0006] A GAN image restoration method based on stable field skip connection, the steps are as follows:

[0007] Step (1). Construct a generative adversarial network based on stable field skip connections.

[0008] The damaged areas in the image are replaced by masks. The generation unit uses the U-Net model as its basic structure, embedding the stabilization field operator into the skip connection of the U-Net model. The image with the damaged area is used as the input of the generation unit. The stabilization field operator is used to make a preliminary pixel prediction of the damaged area on the encoder feature map. The encoder feature map with the completed pixel prediction of the damaged area is then passed to the decoder to obtain the generated image. The discrimination unit consists of 5 convolutional layers, performs binary classification on the input image, and is responsible for distinguishing between the repaired image and the original image.

[0009] Step (2). Construct the loss function of the generative adversarial network;

[0010] The generated image obtained in step (1) is processed to obtain the repair result of the damaged image, and the pixel reconstruction loss is calculated with the original image; the decoder feature map outputs the image of each scale and the original image of the corresponding scale to calculate the pyramid loss; at the same time, it is used as the input of the discriminant unit, the discrimination result is output and the adversarial loss is calculated. The combination of the three loss functions guides the next step of training of the generative adversarial network in the right direction, so that the generation unit can generate more realistic repaired images, and the discriminant unit can further distinguish the generated repaired images from the original images;

[0011] Step (3). Obtain the repair result of the damaged image through the trained generative adversarial network.

[0012] Step (1). The specific method is as follows:

[0013] Step (1.1). The generation unit consists of an encoder, a decoder, and a jump connection embedded with a stable field operator. The damaged area in the image is replaced by a mask (a centered rectangular mask or an irregular mask). The specific formula is as follows:

[0014] I input =I o ⊙(1-M) (1)

[0015] Where, I o is the original image, M is the mask (M and I o The damaged areas in M ​​have the same size, all pixel values ​​are 1, and the rest are 0, and they are centered rectangular masks or irregular masks), and ⊙ represents the element-by-element multiplication of the matrix.

[0016] The image with damaged areas is input into the generative unit encoder to obtain the encoder feature map. The stable field operator is used to preliminarily predict the pixel information of the damaged area in the feature map using the effective pixel information in the encoder feature map. The stable field model formula is expressed as follows:

[0017] LG(r,r i )=f (2)

[0018] Where L is the linear differential operator, r is the damaged pixel, r i is the effective pixel point, G(r,r i ) is defined as the influence function of the effective pixel point on the damaged pixel point, and the known area that provides energy to the damaged area is defined as the field source f.

[0019] The function I(r) describing the image texture in step (1.2) is defined as:

[0020]

[0021] Where, I0(r i ) is r i The point source intensity is the pixel value, and n is the number of valid pixels involved in the calculation. Equation (3) shows that the value of a damaged pixel can be expressed as the sum of the product of the pixel values ​​of all valid pixels that affect it and the influence function. The valid pixels include known pixels and restored pixels. The influence of valid pixels that are far away on the missing pixels can be ignored. In Equation (3), n = 20, that is, the pixel information of the 20 points around the damaged pixel r is used to complete the pixel prediction of r.

[0022] Step (1.3). Define a single point r from the perspective of directional derivative i The impact factor g(r,r i ), as shown in formula (4):

[0023]

[0024] Where, For r i The distance between r, Represents r i The gradient of the pixel, θ is the gradient and vector The purpose of ε is to prevent the denominator from being 0.

[0025] Step (1.4). Consider the influence factors of all valid pixels around r and perform normalized calculation. The formula is as follows:

[0026]

[0027] Where, Formula (3) is solved.

[0028] After filling r using formula (3), fill the pixels in the damaged area adjacent to r. However, at this time, r has changed from a damaged pixel to a valid pixel. Repeat steps (1.2) to (1.4) to gradually complete the filling of all pixels in the damaged area and obtain a feature map that completes the preliminary prediction of the pixels in the damaged area.

[0029] Step (1.5). The feature map of the damaged area pixels preliminarily predicted in step (1.4) is transferred to the corresponding decoder through a skip connection to achieve channel fusion on the decoder feature map, so that the upsampling calculation of the decoder can utilize the original information in the encoder and the preliminary prediction information of the stable field operator;

[0030] Step (1.6). The decoder outputs the generated image;

[0031] Step (1.7). The discriminant unit consists of 5 convolutional layers, which performs binary classification on the input image and is responsible for distinguishing between the repaired image and the original image.

[0032] Step (2). The specific method is as follows:

[0033] Step (2.1). Refine the loss type and redefine the loss function L to guide the model training in the right direction. The formula is as follows:

[0034] L=λ adv L adv +λ pyramid L pyramid +L rec (6)

[0035] Where, L adv To combat the loss, Lpyramid is the pyramid loss, L rec is the pixel reconstruction loss, λ adv >0 and λ pyramid >0 is the weight value of the corresponding loss function. adv =0.1,λ pyramid =0.5.

[0036] Step (2.2). The generated image output by the generation unit has changed contents in both the damaged and known regions compared to the input. However, only the damaged region needs to be repaired. Therefore, the final repair result is obtained by combining the input known region and the output damaged region. The formula is as follows:

[0037] I inpainting =I g ⊙M+I o ⊙(1-M) (7)

[0038] Where, I g The generated image output by the generation unit.

[0039] The obtained image restoration results are input into the discriminator to calculate the adversarial loss to make the results more realistic. The adversarial loss is defined by the hinge function and is expressed as follows:

[0040]

[0041]

[0042] Where, P data (I inpainting ) is the distribution of repair image samples, Expressing expectations inpainting From P data (I inpainting ) is obtained; P data (I o ) is the true sample distribution, Expressing expectations o From P data (I o ) is obtained; D(I o ),D(I inpainting ) represent the discrimination results of the discriminant unit on the original image and the repaired image, ReLU(·) represents the activation function, L D is the adversarial loss of the discriminative unit and is also the total loss of the discriminative unit.

[0043] Step (2.3). The generated image output by the generation unit will also change the input image I input The damaged area and the known area, so the pixel reconstruction loss L recDecomposed into pixel reconstruction loss L in the missing area of ​​the image hole and pixel reconstruction loss L of known regions valid . With L hole To guide the output of the missing area to be semantically closer to the original image, L valid The generated image output by the guidance generation unit will not change much in the known area. The formula is as follows:

[0044] L rec =λ hole L hole +λ valid L valid (10)

[0045] Where λ hole >0 and λ valid >0 is the corresponding weight value. hole and λ valid The l1 distance is used to measure the difference between the generated image and the original image at the pixel level. The formula is as follows:

[0046] L hole =||I g ⊙MI o ⊙M||1=||I inpainting -I o ||1 (11)

[0047] L valid =||I g ⊙(1-M)-I o ⊙(1-M)||1=||I g ⊙(1-M)-I input ||1 (12)

[0048] Step (2.4). By generating the decoder feature maps of each layer of the unit, the convolution outputs the repaired images of different scales and calculates the l1 distance with the real samples, thereby obtaining the pyramid loss and gradually improving the filling of the damaged area at each scale. The formula is expressed as follows:

[0049]

[0050] Where, is the image restoration result of each scale corresponding to each layer decoder, by I o Downsample to Have the same scale size.

[0051] Furthermore, the λ in step (2) hole =6,λ valid =1.

[0052] The beneficial effects of the present invention are as follows:

[0053] Based on the fact that skip connections in U-Net networks cannot convey detailed texture information of damaged images, this paper embeds a stabilizing field operator into the skip connections. This operator is used to make preliminary predictions of damaged pixels in the encoder feature map in the generation unit, which are then passed to the decoder and output as the generated image. The paper then refines the loss types and redefines the loss functions into adversarial loss, pixel reconstruction loss, and pyramid loss to further guide network training in the right direction.

[0054] This method addresses the problem that existing image restoration methods often neglect semantic consistency and feature continuity in damaged areas, resulting in semantic discontinuity in the restored image. This method improves the restoration of damaged images, achieving a certain degree of similarity to the original image while meeting the needs of human visual perception. Experimental verification has shown that this method achieves excellent restoration results on facial images, natural scene images, and architectural images. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is the overall block diagram of the embodiment of the present invention;

[0056] Figure 2 A structural diagram of a generating unit in the present invention;

[0057] Figure 3 1 is a structural diagram of the discrimination unit in the present invention;

[0058] Figure 4 Schematic diagram of pixel filling in damaged areas according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The present invention will be further described below with reference to the accompanying drawings.

[0060] like Figure 1 As shown in the figure, G is the generation unit, D is the discrimination unit, and I input is the input image of the generation unit, I g is the generated image output by the generation unit, I o is the original image, I inpainting is the final repair result of the damaged image, M is the mask (the mask is used to replace the damaged area in the image, M and I g The matrix M has the same size, the pixel values ​​corresponding to the damaged area are all 1, and the rest are 0, and it is a centered rectangular mask or an irregular mask), and ⊙ represents the matrix element-by-element multiplication. The specific steps of the method of the present invention are as follows:

[0061] Step (1). Construct a generative adversarial network based on stable field skip connections.

[0062] The damaged areas in the image are replaced by masks (including centered rectangular masks and irregular masks). The generation unit uses the U-Net model as the basic structure, embeds the stabilization field operator into the jump connection of the U-Net model, and uses the image with the damaged area as the input of the generation unit. The stabilization field operator is used to make a preliminary pixel prediction of the damaged area on the encoder feature map. The encoder feature map with the completed pixel prediction of the damaged area is then passed to the decoder to obtain the generated image. The discrimination unit consists of 5 convolutional layers, performs binary classification on the input image, and is responsible for distinguishing the repaired image from the original image.

[0063] like Figure 2 As shown in , the generation unit uses the U-Net model as the basic structure, embeds the stable field operator into the jump connection of the U-Net model, takes the image with the damaged area as the input of the generation unit, uses the stable field operator to perform preliminary pixel prediction of the damaged area on the encoder feature map, and passes the encoder feature map with completed pixel prediction of the damaged area to the decoder to obtain the generated image; as shown in Figure 3 As shown in the figure, the discriminant unit consists of 5 convolutional layers, which performs binary classification on the input image and is responsible for distinguishing between the restored image and the original image:

[0064] Step (1.1). The generation unit consists of an encoder, a decoder, and a jump connection embedded with a stable field operator. The damaged area is replaced by a mask. The specific formula is as follows:

[0065] I input =I o ⊙(1-M) (1)

[0066] The image with damaged areas is input into the generative unit encoder to obtain the encoder feature map. The stable field operator is used to preliminarily predict the pixel information of the damaged area in the feature map using the effective pixel information in the encoder feature map. The stable field model formula is expressed as follows:

[0067] LG(r,r i )=f (2)

[0068] Where L is the linear differential operator, r is the damaged pixel, r i is the effective pixel point, G(r,r i ) is defined as the influence function of the effective pixel point on the damaged pixel point, and the known area that provides energy to the damaged area is defined as the field source f.

[0069] The function I(r) describing the image texture in step (1.2) is defined as:

[0070]

[0071] Where, I0(r i ) is ri The point source intensity is the pixel value, and n is the number of valid pixels involved in the calculation. Equation (3) shows that the value of a damaged pixel can be expressed as the sum of the product of the pixel values ​​of all valid pixels that affect it and the influence function. The valid pixels include known pixels and restored pixels. The influence of valid pixels far away on the missing pixels can be ignored. In Equation (3), n = 20 is used to complete the pixel prediction of r using the pixel information of the 20 points around the damaged pixel r.

[0072] Step (1.3). Define a single point r from the perspective of directional derivative i The impact factor g(r,r i ), as shown in formula (4):

[0073]

[0074] Where, d rri For r i The distance between r, Represents r i The gradient of the pixel, θ is the gradient and vector The angle of ε is to prevent the denominator from being 0. Step (1.4). Consider the influence factors of all valid pixels around r and perform normalized calculations. The formula is as follows:

[0075]

[0076] Where, Formula (3) is solved.

[0077] After filling r using formula (3), fill the pixels in the damaged area adjacent to r. However, at this time, r has changed from a damaged pixel to a valid pixel. Repeat steps (1.2) to (1.4) to gradually complete the filling of all pixels in the damaged area, and obtain a feature map that completes the preliminary prediction of the pixels in the damaged area.

[0078] Taking steps (1.2) to (1.4) as an example, the specific method is as follows:

[0079] like Figure 4 As shown in the figure, the rectangular area with gray background is the damaged area, and the rest are known areas. Now use the valid pixels in the feature map to fill the pixel r in the damaged area, r1~r 20 There are 20 points that can effectively affect r, but in this case r 14 ~r 20 It is the missing pixel in the missing area, so only r1~r 13r is filled. After filling r by formula (3), steps (1.2) to (1.4) are reused to fill r 14 , r 15 , and the outermost circle of pixel points in the damaged area are filled, and after the outermost circle of pixel points is filled, the filling of the inner circle of pixels is performed until the filling of all pixel points in the damaged area is completed.

[0080] Step (1.5). The feature map of the initial prediction of the pixel in the damaged area in step (1.4) is transmitted to the corresponding decoder through a skip connection to realize channel fusion on the decoder feature map, so that the up-sampling calculation of the decoder can use the original information in the encoder and the initial prediction information of the stable field operator;

[0081] Step (1.6). The decoder outputs a generated image.

[0082] Step (1.7). The discrimination unit is composed of 5 convolutional layers, which performs binary classification on the input image and is responsible for discriminating the repaired image and the original image.

[0083] Step (2). Construct the loss function of the generative adversarial network.

[0084] The generated image obtained in step (1) is processed to obtain the repair result of the damaged image, and the pixel reconstruction loss is calculated with the original image; the decoder feature map outputs images of each scale and the original image of the corresponding scale to calculate the pyramid loss; at the same time, as the input of the discrimination unit, the discrimination result is output and the adversarial loss is calculated, and the three loss functions are combined to guide the next training of the generative adversarial network to the correct direction, so that the generation unit can generate more realistic repaired images, and the discrimination unit can further distinguish the repaired images and the original images.

[0085] Step (2.1). Refine the types of loss, redefine the loss function L to guide the training of the model to the correct direction. The formula is expressed as follows:

[0086] L = λ adv L adv + λ pyramid L pyramid + L rec (6)

[0087] In the formula, L adv is the adversarial loss, L pyramid is the pyramid loss, L rec is the pixel reconstruction loss, λ adv > 0 and λ pyramid > 0 are the weight values of the corresponding loss functions. Preferably, λ adv = 0.1 and λ pyramid = 0.5.

[0088] Step (2.2). The generated image output by the generation unit has changed contents in both the damaged and known regions compared to the input. However, only the damaged region needs to be repaired. Therefore, the final repair result is obtained by combining the input known region and the output damaged region. The formula is as follows:

[0089] I inpainting =I g ⊙M+I o ⊙(1-M) (7)

[0090] The obtained image restoration results are input into the discriminator to calculate the adversarial loss to make the results more realistic. The adversarial loss is defined by the hinge function and is expressed as follows:

[0091]

[0092]

[0093] Where, P data (I inpainting ) is the distribution of repair image samples, Expressing expectations inpainting From P data (I inpainting ) is obtained; P data (I o ) is the true sample distribution, Expressing expectations o From P data (I o ) is obtained; D(I o ),D(I inpainting ) represent the discrimination results of the discriminant unit on the original image and the repaired image, ReLU(·) represents the activation function, L D is the adversarial loss of the discriminative unit and is also the total loss of the discriminative unit.

[0094] Step (2.3). The generated image output by the generation unit will also change the input image I input The damaged area and the known area, so the pixel reconstruction loss L rec Decomposed into pixel reconstruction loss L in the missing area of ​​the image hole and pixel reconstruction loss L of known regions valid . With L hole To guide the output of the missing area to be semantically closer to the original image, L valid The generated image output by the guidance generation unit will not change much in the known area. The formula is as follows:

[0095] L rec =λ hole L hole+λ valid L valid (10)

[0096] Where λ hole >0 and λ valid >0 is the corresponding weight value. hole =6,λ valid =1. hole and λ valid The l1 distance is used to measure the difference between the generated image and the original image at the pixel level. The formula is as follows:

[0097] L hole =||I g ⊙MI o ⊙M||1=||I inpainting -I o ||1 (11)

[0098] L valid =||I g ⊙(1-M)-I o ⊙(1-M)||1=||I g ⊙(1-M)-I input ||1 (12)

[0099] Step (2.4). By generating the decoder feature maps of each layer of the unit, the convolution outputs the repaired images of different scales and calculates the l1 distance with the real samples, thereby obtaining the pyramid loss and gradually improving the filling of the damaged area at each scale. The formula is expressed as follows:

[0100]

[0101] Where, is the image restoration result of each scale corresponding to each layer decoder, by I o Downsample to Have the same scale size.

[0102] Step (3). Obtain the repair result of the damaged image through the trained generative adversarial network.

[0103] During the training process of the generative adversarial network, the loss continues to decrease. The images generated by the generator unit become more realistic, and the discriminator unit is also more able to distinguish between the generated images and the real images. When the generator unit and the discriminator unit reach Nash equilibrium, it means the end of the generative adversarial network training. At this time, I obtained by formula (7) inpainting This is the final repair result of the damaged image.

[0104] It should be noted that the above embodiments can be freely combined as needed. The above description only provides a detailed description of the preferred embodiments and principles of the present invention, but does not limit the present invention. For those skilled in the art, based on the ideas provided by the present invention, there may be changes in the specific implementation methods, and such changes should also be considered as the scope of protection of the present invention.

Claims

1. A GAN image restoration method based on stable field skip connection, characterized in that: Here are the steps: Step (1). Construct a generative adversarial network based on stable field skip connections; The damaged area in the image is replaced by a mask. The generation unit is based on the U-Net model, and the stabilization field operator is embedded in the skip connection of the U-Net model. The image with the damaged area is used as the input of the generation unit. The stabilization field operator is used to perform preliminary pixel prediction of the damaged area on the encoder feature map. The encoder feature map with completed pixel prediction of the damaged area is then passed to the decoder to obtain the generated image. The discriminant unit consists of 5 convolutional layers, which performs binary classification on the input image and is responsible for distinguishing between the restored image and the original image; Step (2). Construct the loss function of the generative adversarial network; Process the generated image obtained in step (1) to obtain the repaired result of the damaged image, and calculate the pixel reconstruction loss compared with the original image; The decoder feature map outputs images of each scale and calculates the pyramid loss against the original image of the corresponding scale. It also serves as the input to the discriminant unit, outputs the discrimination result, and calculates the adversarial loss. Combining these three loss functions simultaneously guides the next step of training in the generative adversarial network in the right direction, enabling the generator to generate more realistic inpainted images and the discriminant unit to further distinguish the generated inpainted images from the original images. Step (3). Obtain the repair result of the damaged image through the trained generative adversarial network; Step (1). The specific method is as follows: Step (1.1). The generation unit consists of an encoder, a decoder, and a jump connection embedded with a stable field operator. The damaged area in the image is replaced by a mask. The specific formula is as follows: I input =I o ⊙(1-M)(1)In the formula, I o is the original image, M is the mask, and ⊙ represents the matrix element-by-element multiplication; The image with damaged areas is input into the generative unit encoder to obtain the encoder feature map. The stable field operator is used to preliminarily predict the pixel information of the damaged area in the feature map using the effective pixel information in the encoder feature map. The stable field model formula is expressed as follows: LG(r,r i )=f(2) where L is the linear differential operator, r is the damaged pixel, r i is the effective pixel point, G(r,r i ) is defined as the influence function of the effective pixel point on the damaged pixel point, and the known area that provides energy to the damaged area is defined as the field source f; The function I(r) describing the image texture in step (1.2) is defined as: Where, I0(r i ) is r i The point source intensity of a point is the pixel value, and n is the number of valid pixels involved in the calculation. Formula (3) shows that the value of a damaged pixel can be expressed as the sum of the product of the pixel values ​​of all valid pixels that have an impact on it and the influence function. The valid pixels include known pixels and restored pixels. The impact of valid pixels that are far away on the missing pixels can be ignored. In formula (3), n = 20, that is, the pixel information of the 20 points around the damaged pixel r is used to complete the pixel prediction of r. Step (1.3). Define a single point r from the perspective of directional derivative i The impact factor g(r,r i ), as shown in formula (4): Where, For r i The distance between r, Represents r i The gradient of the pixel, θ is the gradient and vector The angle of ε is to prevent the denominator from being 0; Step (1.4). Consider the influence factors of all valid pixels around r and perform normalized calculation. The formula is as follows: Where, Formula (3) is solved; After filling r using formula (3), fill the pixels in the damaged area adjacent to r. However, at this time, r has changed from a damaged pixel to a valid pixel. Repeat steps (1.2) to (1.4) to gradually complete the filling of all pixels in the damaged area, and obtain a feature map that completes the preliminary prediction of the pixels in the damaged area. Step (1.5). The feature map of the damaged area pixels preliminarily predicted in step (1.4) is transferred to the corresponding decoder through a skip connection to achieve channel fusion on the decoder feature map, so that the upsampling calculation of the decoder can utilize the original information in the encoder and the preliminary prediction information of the stable field operator; Step (1.6). The decoder outputs the generated image; Step (1.7). The discriminant unit consists of 5 convolutional layers, which performs binary classification on the input image and is responsible for distinguishing between the repaired image and the original image.

2. The GAN image restoration method based on stable field skip connection according to claim 1 is characterized in that: Step (2). The specific method is as follows: Step (2.1). Refine the type of loss and redefine the loss function L to guide the model training in the right direction; the formula is as follows: L=λ adv L adv +λ pyramid L pyramid +L rec (6) Where, L adv To combat the loss, L pyramid is the pyramid loss, L rec is the pixel reconstruction loss, λ adv >0 and λ pyramid >0 is the weight value of the corresponding loss function; l adv =0.1,λ pyramid =0.5; Step (2.2). The generated image output by the generation unit has changed contents in both the damaged and known regions compared to the input. However, only the damaged region needs to be repaired. Therefore, the final repair result is obtained by combining the input known region and the output damaged region. The formula is as follows: I inpainting =I g ⊙M+I o ⊙(1-M) (7) Where, I g The generated image output by the generation unit; The obtained image restoration results are input into the discriminant unit to calculate the adversarial loss to make the results more realistic; the adversarial loss is defined by the hinge function, and the formula is expressed as follows: Where, P data (I inpainting ) is the distribution of repair image samples, Expressing expectations inpainting From P data (I inpainting ) is obtained; P data (I o ) is the true sample distribution, Expressing expectations o From P data (I o ) is obtained; D(I o ),D(I inpainting ) represent the discrimination results of the discriminant unit on the original image and the repaired image, ReLU(·) represents the activation function, L D is the adversarial loss of the discriminative unit and also the total loss of the discriminative unit; Step (2.3). The generated image output by the generation unit will also change the input image I input The damaged area and the known area, so the pixel reconstruction loss L rec Decomposed into pixel reconstruction loss L in the missing area of ​​the image hole and pixel reconstruction loss L of known regions valid ; L hole To guide the output of the missing area to be semantically closer to the original image, L valid The generated image output by the guidance generation unit will not change much in the known area. The formula is as follows: L rec =λ hole L hole +λ valid L valid (10) Where λ hole >0 and λ valid >0 is the corresponding weight value; λ hole and λ valid The l1 distance is used to measure the difference between the generated image and the original image at the pixel level. The formula is as follows: L hole =||I g ⊙M-I o ⊙M||1=||I inpainting -I o ||1 (11) L valid =||I g ⊙(1-M)-I o ⊙(1-M)||1=||I g ⊙(1-M)-I input ||1 (12); Step (2.4). By generating the decoder feature maps of each layer of the unit, the convolution outputs the repaired images of different scales and calculates the l1 distance with the real samples, thereby obtaining the pyramid loss and gradually improving the filling of the damaged area at each scale. The formula is expressed as follows: Where, is the image restoration result of each scale corresponding to each layer decoder, by I o Downsample to Have the same scale size.

3. A GAN image restoration method based on stable field skip connection according to claim 1 or 2, characterized in that: The λ in step (2) hole =6,λ valid =1.

Images