A method for object image restoration based on symmetry guidance

Abstract

A method for restoring an object image based on symmetry guidance relates to the field of image processing technology. First, the symmetry axis of the incomplete object in the image is detected, and the symmetry information of the object is obtained by means of the symmetry axis. The symmetry information of the known area is filled into the unknown area as the initial value of the area to be filled, so as to solve the problem that the information of the original area to be repaired is zero, and then the image is effectively repaired on this basis. The beneficial effect of the present invention is that the symmetry axis of the symmetrical object with incomplete areas in the image is accurately and effectively detected through the object symmetry axis detection technology based on the generative adversarial network, and finally a symmetrical object image with reasonable structure and correct semantics can be generated, even if a large area is incomplete.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and specifically relates to an object image restoration method based on symmetry guidance. Background Art

[0002] Image inpainting is an important problem in the field of image processing. Its purpose is to fill in the pixels of the missing area so that the whole image can achieve consistency in structure and details. Since image inpainting itself is a highly ill-posed problem, it is necessary to establish optimization criteria under certain constraints to achieve image inpainting. Current image inpainting technology is mainly based on deep learning models. Context Encoder [1] is the first to introduce adversarial generative network deep learning model into image inpainting. The generator of this network is a typical Encoder-Decoder model used to realize the encoding and decoding of image information. The discriminator is a multi-layer convolutional network used to judge the effect of the generated image. Based on the Context Encoder model, Iizuka et al [2] proposed a new model consisting of a generative network and two discriminative networks. The two discriminative networks distinguish the generated entire image from the image of the area to be inpainted, so that the inpainted image will have better global and local consistency. Subsequently, Yu et al. [3] divided image inpainting into two steps and implemented it with two network models: the first step is to generate a rough inpainting result; the second step is to refine the rough inpainting result to generate a result with rich details and relatively complete structure. The inpainting area processed by the above work is a rectangular area located in the center of the image. Later, people began to study image restoration techniques for irregularly shaped areas to be restored. Liu et al. [4] proposed a partial convolution model to replace ordinary convolution so that the result of convolution depends only on valid pixels to control the information transfer within the network. Subsequently, Yu et al. [5] proposed a gated convolution model, which can be regarded as a learnable version of partial convolution. An additional standard convolution layer is used to learn the validity of each pixel / feature position. Nazeri et al. [6] proposed an edge prediction and image restoration technology based on predicted edge maps. The edge information of the image is extracted and predicted to generate a complete edge map of the image. The edge information is then combined with the image to be restored, and the image restoration is guided by the image edge. Although the existing algorithms have achieved certain results, when faced with large defective areas in the image, these algorithms cannot perceive information from places far away from the defective area and cannot generate images with reasonable structure and details. Aiming at the problem of image restoration of objects with symmetrical properties, the present invention proposes a new method that uses the symmetrical information of the object to restore the target image with a large defective area.

[0003] [1] D. Pathak, P. Krahenbuhl, J. Donahue, et al. Context encoders:Feature learning by inpainting[C]. Proceedings of IEEE Conference on ComputerVision and Pattern Recognition, 2016: 2536-2544.

[0004] [2] S. Iizuka, E. Simo-Serra, and H. Ishikawa. Globally and locallyconsistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4):107:1-107:14

[0005] [3] J. Yu, Z. Lin, J. Yang, et al. Generative Image Inpainting withContextual Attention[C]. Proceedings of Computer Vision and PatternRecognition, 2018: 5505-5514.

[0006] [4] G. Liu, F. Reda, K. Shih, et al. Image Inpainting for IrregularHoles Using Partial Convolutions[C]. Proceedings of European Conference onComputer Vision, 2018: 85-100

[0007] [5] J. Yu, Z. Lin, J. Yang, et al. Free-form Image Inpainting withGated Convolution[C]. Proceedings of IEEE International Conference onComputer Vision, 2019:4471–4480.

[0008] [6] K. Nazeri, E. Ng, T. Joseph, et al. EdgeConnect: Structure GuidedImage Inpainting using Edge Prediction[C]. Proceedings of IEEE InternationalConference on Computer Vision Workshops, 2019: 3265-3274. Summary of the invention

[0009] The purpose of the present invention is to propose an object image restoration method based on symmetry guidance to address the problems that the current existing algorithms are difficult to effectively utilize useful information far away from the defective area on the one hand, and when the area to be repaired is large, it is difficult to generate a reasonable object structure relying solely on the information of the neighborhood. This method makes the object structure with symmetrical features in the repaired image more reasonable and the semantics more accurate.

[0010] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a method for restoring an object image based on symmetry guidance, a method for restoring an object image based on symmetry guidance, characterized in that it comprises the following steps:

[0011] Step 1: Use the symmetry axis generator based on the generative adversarial network to predict the symmetry axis of the target image in the image to be repaired ;

[0012] Step 2: Use the edge generator to predict the edge of the defective area on the image to be repaired to obtain the gradient repair image ;

[0013] Step 3: Use the image filler to fill the gradient repair image obtained in step 2 to complete the image repair.

[0014] In step 1 of the present invention, a symmetry axis generator based on a generative adversarial network is used to predict the symmetry axis of the target image in the image to be repaired. The specific method is:

[0015] Step 1.1: Grayscale the image to be repaired , gradient map of the image to be repaired Connect on the feature channel and input to the feature point map generator Get the specific feature point map of the image to be repaired ;

[0016] Step 1.2: Feature point map Linear fitting is performed on the feature points in the image to obtain the image symmetry axis ;

[0017] Step 1.3: And the annotated real image specific feature point map Input to the discriminator Optimizing the generator and the discriminator ;

[0018] For the discriminator , and its loss function is defined as:

[0019] ;

[0020] For the generator , and its loss function is defined as:

[0021] ;

[0022] in, It is a feature matching loss function, which is used to make the feature point map generated by the generator consistent with the features of the specific feature point map of the annotated real image. The calculation formula is as follows:

[0023] ;

[0024] in, Representation Discriminator Network The i-th layer output, K represents the number of layers in the network, Indicates the number of elements in the i-th layer.

[0025] The gradient map of the image to be repaired according to the present invention Use Sobel operator to extract.

[0026] The specific method of using the edge generator in step 2 of the present invention to predict the edge of the defective area on the image to be repaired is:

[0027] Step 2.1: Convert the grayscale image to be repaired , gradient map of the image to be repaired Connect on the feature channel and input to the generator feature encoding network Get image features ;

[0028] Step 2.2: Image features , image symmetry axis Input generator feature flipping network Get the flipped image features ;

[0029] Step 2.3: Combine the two image features , Connect on the feature channel and input to the generator decoding network The gradient repair map is obtained from ;

[0030] Step 2.4: Repair the gradient map And the gradient map of the real image Input to the discriminator Optimizing the generator and the discriminator ;

[0031] For the discriminator D2 , and its loss function is defined as:

[0032] ;

[0033] For the generator , and its loss function is defined as:

[0034] ;

[0035] in, It is a feature matching loss function, which is used to make the gradient restoration map generated by the generator consistent with the features of the gradient map of the real image. The calculation formula is as follows:

[0036] ;

[0037] in, Representation Discriminator Network The i-th layer output, K represents the number of layers in the network, Indicates the number of elements in the i-th layer.

[0038] The specific method of using the image filler in step 3 of the present invention to fill the gradient repair image obtained in step 2 is:

[0039] Step 3.1: The image to be repaired , gradient repair map Connect on the feature channel and input to the generator Generate a complete repaired image in ;

[0040] Step 3.2 will completely repair the image and real images Input into a general feature extraction network VGG19 to perceive whether the image content and structure are reasonable and generate a perceptual loss function With the stylized loss function , and for real images With the complete repair image For each pixel in , a loss function is generated ;

[0041] Step 3.3 will completely repair the image and real images Input to the discriminator Optimizing the generator and the discriminator ;

[0042] For the discriminator , and its loss function is defined as:

[0043] ;

[0044] For the generator , and its loss function is defined as:

[0045] .

[0046] The loss function generated in step 3.2 of the present invention is With loss function The method is:

[0047] is the perceptual loss function, which is used to make the complete restoration image generated by the generator consistent with the characteristics of the real image. By comparing VGG19 to the complete restoration image and real images The intermediate layer output is obtained, and its calculation formula is:

[0048] ;

[0049] in Represents the output of the five layers relu1_1, relu2_1, relu3_1, relu4_1, and relu5_1 in VGG19;

[0050] is the stylized loss function, which has the same effect as Same, stylized loss function By comparing VGG19 to the complete restoration image and real images The Gram matrix constructed by the intermediate layer output is obtained, and the calculation formula is as follows:

[0051] ;

[0052] in Represents the Gram matrix constructed by the output of the five layers relu1_2, relu2_2, relu3_2, relu4_2, and relu5_2 in VGG19.

[0053] The pixel loss function generated in step 3.2 of the present invention is The method is:

[0054] Pixel loss function Its function is to make the complete repaired image generated by the generator consistent with the content of the real image. The difference between the real image and the complete restoration is calculated as:

[0055] .

[0056] The beneficial effects of the present invention are: (1) The method of the present invention proposes a technology for detecting the symmetry axis of an object image. In the face of background interference, the object is approximately symmetrical, the defective area is large, etc., a relatively accurate symmetry axis can be obtained, and the detection accuracy is high. The method has good robustness; (2) The present invention makes full use of the symmetry of the object to provide an effective image restoration method. Experimental results show that the method proposed by the present invention is significantly better than the existing algorithm when it is used to restore the image of a symmetrical object with a large area missing. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the overall process of the image restoration method based on symmetry guidance of the present invention. DETAILED DESCRIPTION

[0058] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments, but they are not intended to limit the invention in any way.

[0059] The present invention provides an object image restoration method based on symmetry guidance. The whole scheme mainly includes three parts: (1) symmetry axis detection of incomplete symmetrical objects; (2) edge prediction of the image area to be restored guided by symmetry; and (3) image content restoration based on the edge prediction map.

[0060] Step 1: The image restoration model proposed in the present invention needs to obtain the symmetry information of the image to be restored, so we use a symmetry axis generator to predict the symmetry axis of the target image on the original image, as follows:

[0061] Step 1.1: Convert the grayscale image to be repaired , gradient map of the image to be repaired (Image gradient map is extracted using Sobel operator) Connect on the feature channel and input to the feature point map generator Get the specific feature point map of the image to be repaired ;

[0062] Step 1.2, feature point map Linear fitting is performed on the feature points in the image to obtain the image symmetry axis ;

[0063] Step 1.3, And the annotated real image specific feature point map Input to the discriminator Optimizing the generator and the discriminator ;

[0064] For the discriminator , and its loss function is defined as:

[0065] ; (1)

[0066] For the generator , and its loss function is defined as:

[0067] ; (2)

[0068] in, It is a feature matching loss function, which is used to make the feature point map generated by the generator consistent with the features of the specific feature point map of the annotated real image. The calculation formula is as follows:

[0069] ; (3)

[0070] in, Representation Discriminator Network The i-th layer output, K represents the number of layers in the network, Indicates the number of elements in the i-th layer.

[0071] Step 2: Use the edge generator to predict the original edge of the masked area on the original image, as follows:

[0072] Step 2.1: Convert the grayscale image to be repaired , gradient map of the image to be repaired (Image gradient map is extracted using Sobel operator) Connect on the feature channel and input to the generator feature encoding network Get image features .

[0073] Step 2.2: Image features , image symmetry axis Input generator feature flipping network Get the flipped image features ;

[0074] Step 2.3: Combine the two image features , Connect on the feature channel and input to the generator decoding network The gradient repair map is obtained from ;

[0075] Step 2.4: Repair the gradient map And the gradient map of the real image Input to the discriminator Optimizing the generator and the discriminator ;

[0076] For the discriminator , and its loss function is defined as:

[0077] ; (4)

[0078] in, is the gradient map of the real image.

[0079] For the generator , and its loss function is defined as:

[0080] ; (5)

[0081] in, It is a feature matching loss function, which is used to make the gradient restoration map generated by the generator consistent with the features of the gradient map of the real image. The calculation formula is as follows:

[0082] ; (6)

[0083] in, Representation Discriminator Network The i-th layer output, K represents the number of layers in the network, Indicates the number of elements in the i-th layer.

[0084] Step 3: Use the image filler to fill the gradient repair image obtained in step 2, as follows:

[0085] Step 3.1: The image to be repaired , gradient repair map Connect on the feature channel and input to the generator Generate a complete repaired image in .

[0086] Step 3.2 will completely repair the image and real images Input into a general feature extraction network VGG19 to perceive whether the image content and structure are reasonable and generate a loss function With loss function (Johnson et al, 2016), and for real images With the complete repair image For each pixel in , a loss function is generated ;

[0087] is a perceptual loss function, which is used to make the complete restoration image generated by the generator consistent with the characteristics of the real image. and real images The intermediate layer output is obtained, and the calculation formula is as follows:

[0088] ; (7)

[0089] in Represents the output of the five layers relu1_1, relu2_1, relu3_1, relu4_1, and relu5_1 in VGG19;

[0090] is the stylized loss function, which has the same effect as Same, compared with VGG19 for the complete restoration image and real images The Gram matrix constructed by the intermediate layer output is obtained, and the calculation formula is as follows:

[0091] ; (8)

[0092] in Represents the Gram matrix constructed by the output of the five layers relu1_2, relu2_2, relu3_2, relu4_2, and relu5_2 in VGG19;

[0093] , which is used to make the complete repaired image generated by the generator consistent with the content of the real image. The difference between the real image and the complete restoration is calculated as:

[0094] ; (9)

[0095] Step 3.3 will completely repair the image and real images Input to the discriminator Optimizing the generator and the discriminator ;

[0096] For the discriminator , and its loss function is defined as:

[0097] ; (10)

[0098] For the generator , and its loss function is defined as:

[0099] . (11)

[0100] The present invention proposes a symmetry-guided image restoration method, which accurately and effectively detects the symmetry axis of a symmetrical object with a defective area in the image through an object symmetry axis detection technology based on a generative adversarial network, and can ultimately generate a symmetrical object image with a reasonable structure and correct semantics, even if a large area of ​​defect occurs.

[0101] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Those skilled in the art should understand that the specific implementation modes of the present invention may be modified or replaced by equivalents with reference to the above embodiments. Any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention are within the scope of protection of the pending claims.

Claims

1. A method for restoring an object image based on symmetry guidance, characterized in that: The following steps are involved: Step 1: Use the symmetry axis generator based on the generative adversarial network to predict the symmetry axis of the target image in the image to be repaired ; Step 2: Use the edge generator to predict the edge of the defective area on the image to be repaired to obtain the gradient repair image ; Step 3: Use the image filler to fill the gradient restoration image obtained in step 2 to complete the image restoration; In step 1, a symmetry axis generator based on a generative adversarial network is used to predict the symmetry axis of the target image in the image to be repaired. The specific method is: Step 1.1: Grayscale image of the image to be repaired , gradient map of the image to be repaired Connect on the feature channel and input to the feature point map generator Get the specific feature point map of the image to be repaired ; Step 1.2: Feature point map Linear fitting is performed on the feature points in the image to obtain the image symmetry axis ; Step 1.3: And the annotated real image specific feature point map Input to the discriminator Optimizing the generator and the discriminator ; For the discriminator , and its loss function is defined as: ； For the generator , and its loss function is defined as: ； in, It is a feature matching loss function, which is used to make the feature point map generated by the generator consistent with the features of the specific feature point map of the annotated real image. The calculation formula is as follows: ； in, Representation Discriminator Network The i-th layer output, K represents the number of layers in the network, Indicates the number of elements in the i-th layer; The specific method of using the image filler in step 3 to fill the gradient repair image obtained in step 2 is: Step 3.1: The image to be repaired , gradient repair map Connect on the feature channel and input to the generator Generate a complete repaired image in ; Step 3.2 will completely repair the image and real images Input into a general feature extraction network VGG19 to perceive whether the image content and structure are reasonable and generate a perceptual loss function With the stylized loss function , and for real images With the complete repair image For each pixel in, a loss function is generated ; Step 3.3 will completely repair the image and real images Input to the discriminator Optimizing the generator and the discriminator ; For the discriminator , and its loss function is defined as: ； For the generator , and its loss function is defined as: 。 2. The object image restoration method based on symmetry guidance according to claim 1, characterized in that: The gradient map of the image to be repaired Use Sobel operator to extract.

3. The object image restoration method based on symmetry guidance according to claim 1, characterized in that: The specific method of using the edge generator in step 2 to predict the edge of the defective area on the image to be repaired is: Step 2.1: Convert the grayscale image to be repaired , gradient map of the image to be repaired Connect on the feature channel and input to the generator feature encoding network Get image features ; Step 2.2: Image features , image symmetry axis Input generator feature flipping network Get the flipped image features ; Step 2.3: Combine the two image features , Connect on the feature channel and input to the generator decoding network The gradient repair map is obtained from ; Step 2.4: Repair the gradient map And the gradient map of the real image Input to the discriminator Optimizing the generator and the discriminator ; For the discriminator , and its loss function is defined as: ； For the generator , and its loss function is defined as: ； in, It is a feature matching loss function, which is used to make the gradient restoration map generated by the generator consistent with the features of the gradient map of the real image. The calculation formula is as follows: ； in, Representation Discriminator Network The i-th layer output, K represents the number of layers in the network, Indicates the number of elements in the i-th layer.

4. The object image restoration method based on symmetry guidance according to claim 1, characterized in that: The loss function generated in step 3.2 With loss function The method is: is the perceptual loss function, which is used to make the complete restoration image generated by the generator consistent with the characteristics of the real image. By comparing VGG19 to the complete restoration image and real images The intermediate layer output is obtained, and its calculation formula is: ; in Represents the output of the five layers relu1_1, relu2_1, relu3_1, relu4_1, and relu5_1 in VGG19; is the stylized loss function, which has the same effect as Same, stylized loss function By comparing VGG19 to the complete restoration image and real images The Gram matrix constructed by the intermediate layer output is obtained, and the calculation formula is as follows: ; in Represents the Gram matrix constructed by the output of the five layers relu1_2, relu2_2, relu3_2, relu4_2, and relu5_2 in VGG19.

5. The object image restoration method based on symmetry guidance according to claim 1, characterized in that: The pixel loss function generated in step 3.2 The method is: Loss Function The purpose is to make the complete repaired image generated by the generator consistent with the content of the real image. The difference between the real image and the complete restoration is calculated as: 。

Images