Image occlusion restoration and reconstruction method based on partial convolution and shifted deep feature rearrangement
Through the partial convolution-based shift depth feature rearrangement method, combined with the adversarial network of generator and discriminator, the accuracy problem of image occlusion loss compensation is solved, efficient image occlusion recovery is achieved, and images of computer vision technology are improved Compensation capability, suitable for a wide range of image data applications.
Patent Information
- Application Number
- CN201910540242.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-06-21
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2039-06-21
AI Technical Summary
The prior art is difficult to achieve precise and detailed reconstruction of missing parts of image occlusion, especially in compensation for image information loss, the accuracy of computer vision technology is not enough to match human perception capabilities.
Using partial convolution-based shift depth feature rearrangement method, the occlusion area is accurately positioned and restored and reconstruction is carried out through the adversarial network of the generator and discriminator, combining the mask and rearranged depth feature information, and the synthetic occlusion image in the training stage is used. Model training is carried out with the mask, and occlusion is automatically removed in the online stage to achieve complete image recovery.
It realizes image occlusion compensation reconstruction with more precise and detailed relative to human perception, improves the accuracy and efficiency of occlusion recovery, reduces the time-consuming process in the training stage, enhances the robustness and universality of the method, and is suitable for a wide range of image data. application.
Smart Images

Figure CN112116531B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the fields of image modeling, computer vision and image generation, and relates to an image occlusion recovery and reconstruction method based on partial convolution and rearrangement of shifted deep features, mainly for reconstructing image information lost due to occlusion problems in images. Background Art
[0002] We all know that the development and application of deep learning has made great progress based on the improvement of modern computer capabilities. When facing an image, even if part of the image is incomplete due to occlusion, humans can now use their own abilities to compensate for the missing parts of the image to a certain extent and obtain complete information in the image. Modern computers have powerful computing power and ultra-large-scale data analysis capabilities, and are fully capable of reconstructing the missing parts of the image, achieving more accurate and detailed image missing compensation reconstruction relative to human perception. Giving computers this powerful ability through perfect intelligent algorithms will bring better solutions to related problems such as occlusion compensation and damage recovery in practical applications.
[0003] The explosion of deep learning has allowed us to see that by learning from large amounts of data, we can use one of the cutting-edge research areas of computer vision, automatic image generation. In this direction, an adversarial network algorithm inspired by the idea of zero-sum game in game theory has made a relatively good breakthrough. This method generates new pictures by continuously gaming the generating network G (Generator) and the discriminating network D (Discriminator), so that G can learn the distribution of the data. If used for picture generation, after training, G can generate realistic images from a range of random numbers.
[0004] This adversarial idea also caters to our work of removing occlusion and restoring reconstruction. What we want to do is to remove occlusion, so we cannot randomly generate realistic images. We need to constrain the generated occluded part and control that the occluded and restored part is not randomly filled. By adding precise positioning of the occluded area, we can perfectly control the visible area from being destroyed, and use the updated mask and rearranged depth feature information to restore the original appearance of the occluded area. Summary of the invention
[0005] The purpose of the present invention is to solve the problem of image occlusion and missing compensation, and to endow the computer with the ability to reconstruct the occluded and missing parts of the image, so as to achieve more accurate and detailed image occlusion compensation reconstruction than human perception. To achieve this function, the present invention designs a comprehensive method strategy consisting of two parts: "method training stage" and "online use stage". The specific technical implementation scheme is as follows.
[0006] The image occlusion recovery and reconstruction method based on partial convolution and rearrangement of shifted deep features includes the following basic steps:
[0007] Method training phase:
[0008] (1) Build an occlusion image training sample set composed of real images of 40 different objects and 1000 real background images, perform normalization processing, calibrate the masks of 40 different real images, and use 1000 unoccluded images as real image labels;
[0009] (2) Use the synthesized occlusion image and the mask of the corresponding occlusion object as input. Before that, the unoccluded image of the occlusion image is input each time so that the shift layer can obtain the real data. After a series of convolutional layers, deconvolution layers that introduce connected convolutional features, batch normalization layers, and activation layers, a 256*256 image is output;
[0010] (3) In the generator, the unobstructed part is controlled to maintain the original output, and only the obstructed area in the image is generated, restored and reconstructed;
[0011] (4) For the shift layer, the convolution layer features and the deconvolution layer feature information that introduces the connected convolution features are all obtained from the original image. The information of the unknown area in the deconvolution layer can be found by searching the known area in the convolution layer by the nearest neighbor, and the shift layer is optimized by the real image input each time.
[0012] Uy=X*(y)-y
[0013] Among them, U y represents the displacement vector, X*(y) represents the information in the occluded area decoder, and y represents the real information of the occluded area.
[0014] (5) Input the generated image obtained in step (2) and the unobstructed real image in step (1) into the discriminator of this model;
[0015] (6) The discriminator is optimized based on the difference between the low-dimensional features of the output after the convolution layer, batch normalization layer and activation layer of the discriminator in (5) and the low-dimensional features of the unobstructed expected output. The optimization objective function of the discriminator is:
[0016] L D =E[logP(D=real|X GT )]+E[logP(D=fake|G(X occlusion , Mask))]
[0017] Among them, E represents entropy, D represents discriminator, G represents generator; X GT is a real image, D = real means that the output of the discriminator is expected to be a real image; X occlusionrepresents the input image with occlusion, Mask represents the mask of the occluder, G(X occlusion , Mask) is the occlusion recovery and reconstruction image, and D = fake means that the output of the discriminator is expected to be the occlusion recovery and reconstruction image;
[0018] (7) The generator is optimized with the goal of deceiving the discriminator and obtaining the expected output, that is, the generator is optimized together with the gap term for the generator described in steps (3), (4), and (6). The optimization objective function of the generator is:
[0019] L G_L1 =||G(X occlusion ,Mask)⊙X GT )||1
[0020] L G_hole =||(1-Mask)⊙(G(X occlusion , Mask)-X GT )||1
[0021] L G_valid =||Mask⊙(G(X occlusion , Mask)-X GT )||1
[0022]
[0023] L G =E[logP(D=real|G(X occlusion ,Mask))]+L G_L1 +L G_hole +L G_valid +L G_shift
[0024] Among them, E represents entropy, D represents discriminator, G represents generator; X GT is a real image, D = real means that the output of the discriminator is expected to be a real image; X occlusion represents the input image with occlusion, Mask represents the mask of the occluder, G(X occlusion , Mask) is the occlusion recovery and reconstruction image, L G_L1 For the whole, L G_hole is the occlusion area, L G_valid is the visible area, L G_shfit The shift layer.
[0025] Repeat the above steps (2) to (7) until the training model converges, that is, the loss value reaches a state where it no longer decreases significantly, and save the parameters of this model generator for use in the online use stage;
[0026] Online use stage:
[0027] (8) Load the parameters of the generator of this model;
[0028] (9) The image with the occluded object and the mask corresponding to the occluded object are input into the model generator to obtain the desired image after removing the occlusion and restoring the reconstruction image and save it.
[0029] The present invention is an image occlusion restoration and reconstruction method based on partial convolution and rearrangement of shifted depth features. Compared with the prior art, the main advantages of the present invention are:
[0030] (1) The present invention conducts in-depth research on image occlusion as the main issue, and achieves image loss compensation and reconstruction that is more accurate and detailed than human perception capabilities.
[0031] (2) The present invention is guided by the adversarial idea, accurately locates the occluded area, updates the mask step by step to control the occluded recovery area, accurately controls the visible area and the occluded recovery and reconstruction area, and enhances the performance of the recovery and reconstruction method.
[0032] (3) Due to the adoption of the strategy of re-arranging deep feature information, the present invention actively fills the area to be restored and reconstructed with the information around the occluded area during occlusion restoration and reconstruction, that is, it searches for information to fill the occluded area in the visible area of the image; thus, it reduces the time consumption of the training phase and stably controls the occlusion restoration and reconstruction.
[0033] (4) The partial convolution and rearrangement depth feature coordination strategy adopted in the present invention controls the restoration and reconstruction area through partial convolution mask update, and controls the occlusion restoration through rearrangement depth feature. The perfect combination of the two effectively enhances the image occlusion restoration effect, ensures the image occlusion restoration quality and enhances the efficiency, achieving a perfect balance between the two.
[0034] (5) The present invention has a high degree of automation and basically does not require human intervention in the actual use stage. Generally, the occluded image can be perfectly restored by accurately locating the occluded position, and has good robustness.
[0035] (6) The present invention is not targeted at any given category of image data, and is more universal in terms of overall strategy. As long as it is trained with a large amount of data of the corresponding category, the resulting model can restore the occluded area, and has a wider range of applicability, thus having a broader application prospect. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 : Network structure framework diagram of the method of the present invention. DETAILED DESCRIPTION
[0037] The present invention is based on the discriminator and the generator, and the partial convolution is combined with the rearranged depth features to achieve a perfect image occlusion restoration effect. The implementation method of the present invention will be introduced and described in more detail below:
[0038] Method training phase:
[0039] The image occlusion recovery model of the partial convolution of the present invention using the rearrangement of the shifted deep features needs to be trained in a certain number of training samples. Therefore, the implementation of the method of the present invention first faces the problem of selecting training data. In order to ensure accurate control of the occlusion situation, the present invention uses synthetic occlusion images as training samples. Since the method of generating models is adopted, there are high requirements on the number of samples. In the specific operation process, 40 real objects are used as occlusion objects and 1,000 pictures are used as backgrounds. A total of 40,000 pictures are synthesized as training samples. In order to accurately locate the occluded area, the masks of the 40 objects are also used as input.
[0040] The synthetic image data of the training samples are preprocessed and standardized, and the synthetic image with occlusion and the mask corresponding to the occlusion are used as input. In order to make the shift layer obtain real data, the corresponding image without occlusion is input each time, but no back propagation is performed. The above is an important operation of the image occlusion recovery and reconstruction method based on partial convolution and shifted depth feature rearrangement of the present invention. The visible area is kept unchanged and the occluded area is restored and reconstructed in the generator. Partial convolution is used, that is, the mask is updated to accurately restore the occluded area. The output of the generator of the model is the occluded restored and reconstructed image, which is put together with the real unoccluded image as the input of the model discriminator, and the effect of image restoration and reconstruction is judged in the discriminator.
[0041] In terms of the model optimization strategy and parameter learning strategy, considering that the present invention adopts a deep network structure, which contains many neurons and a large number of learning parameters, the gradient descent method is used to learn and adjust the weights. Generally speaking, the optimization goal of the discriminator is to distinguish the true and false degree of image occlusion restoration, and the optimization goal of the generator is to control the perfect restoration of the image occluded area, and restore the image occluded area to deceive the discriminator based on the rearrangement of the depth information while keeping the visible area unchanged. We continue to perform the above process to train the entire model until the model converges, and calculate the PSNR (PeakSignal-to-Noise Ratio) peak signal-to-noise ratio and SSIM (Structural SIMilarity) structural similarity to obtain high-quality results.
[0042] Finally, the present invention stores the learned weights to facilitate the use of subsequent extended category training or the next online use stage.
[0043] Online use stage:
[0044] After the learning and construction of the method of the present invention is completed through an offline process, the method of the present invention can be used online fully automatically. It only needs to input the occluded image and the mask of the corresponding occluded object. Without any human intervention, the image can be removed and the occlusion can be restored to its full effect.
[0045] For any image that needs to remove occlusions, the method of the present invention will sequentially use the following processing and analysis steps to perform image occlusion recovery:
[0046] First, the mask of the occluder is determined and used as input together with the occluded image. Then a model matching the generator used in the training process is built. Then the parameter file saved in the training phase is loaded into the defined model. Finally, a series of calculation operations are performed based on the input data into the model, and finally the image without occlusion is obtained.
Claims
1. An image occlusion recovery and reconstruction method based on partial convolution and rearrangement of shifted deep features, comprising the following steps: Method training phase: (1) Build an occlusion image training sample set by synthesizing real images of 40 different objects and 1,000 real background images, perform normalization processing, calibrate the masks of 40 different real images, and use 1,000 unoccluded images as real image labels; (2) Using the synthesized occlusion image and the mask of the corresponding occlusion object as input, after a series of convolutional layers, deconvolution layers that introduce convolutional features, batch normalization layers, and activation layers, a 256*256 image is output; (3) In the generator, the unobstructed part is controlled to maintain the original output, and only the obstructed area in the image is restored and reconstructed; (4) For the shift layer, the information of the convolution layer features and the deconvolution layer features that introduce the connected convolution features are all obtained from the original image. The information of the unknown area in the deconvolution layer can be found by searching the known area in the convolution layer by the nearest neighbor, and the shift layer is optimized by the real image input each time. (5) Input the generated image obtained in step (2) and the unobstructed real image in step (1) into the discriminator of this model; (6) Optimize the discriminator based on the difference between the low-dimensional features of the output after the convolution layer, batch normalization layer and activation layer of the discriminator in (5) and the low-dimensional features of its unobstructed expected output; (7) Optimizing the generator with the goal of deceiving the discriminator and obtaining the expected output, by combining the gap between the restored image of the occluded area in step (3) and the occluded area in the original image, the parameter optimization of the displacement layer in step (4), and the gap between the low-dimensional features of the restored image output by the discriminator and the low-dimensional features of its unoccluded expected output in step (6), and optimizing the generator with the above three goals; (8) Repeat the above steps (2) to (7) until the training model converges, that is, the loss value reaches a state where it no longer decreases significantly, and save the parameters of the model generator for use in the online use stage; Online use stage: (9) Load the parameters of the generator of this model; (10) The image with the occluded object and the mask corresponding to the occluded object are input into the model generator to obtain the desired image after removing the occlusion and restoring the reconstruction image and save it.
2. According to the image occlusion recovery and reconstruction method based on partial convolution and shifted deep feature rearrangement as described in claim 1, it is characterized in that in step (1), the size of the training images is 256*256, and after all the images are synthesized, the occluders are harmoniously combined with the background and are clearly distinguishable, and the training sample set contains 40,000 synthetic images; there are 40 occluders, including people, plants, and animals.
3. According to the image occlusion recovery and reconstruction method based on partial convolution and rearrangement of shifted deep features as described in claim 1, it is characterized in that, in step (2), before each input of the synthesized image with occlusion, the image without occlusion is input once, the purpose is to enable the shift layer to obtain real data in order to control the distribution of the rearranged features of the shift layer, but when the input does not have an occlusion map, the reverse transmission is prohibited, and it does not affect the generator; when the synthetic image with occlusion is input, the mask of the occlusion is input at the same time, and the synthetic image with occlusion and the mask simultaneously enter a series of convolutional layers, introduce a deconvolution layer that connects the convolutional features, a batch normalization layer and an activation layer, and use the mask updated in the previous layer in each layer. As the number of network layers increases, after the mask is updated, the number of pixels with 255 in the output mask becomes less and less, the area of the effective area in the output image becomes larger and larger, and the influence of the mask on the overall loss becomes smaller and smaller; thereby the expected 256*256 image is output.
4. According to the image occlusion restoration and reconstruction method based on partial convolution and shifted deep feature rearrangement according to claim 1, it is characterized in that in step (3), the synthetic occlusion image and the mask simultaneously enter the simultaneous control generator for the restoration of the occluded area, so as to keep the overall difference from being too large, and focus on keeping the unoccluded area, that is, the visible area, unchanged, so as to achieve the purpose of removing only the occlusion and restoring the reconstructed image; the objective function of the control is as follows L G_L1 =||G(X occlusion ,Mask)⊙X GT )||1 L G_hole =||(1-Mask)⊙(G(X occlusion , Mask)-X GT )||1 L G_valid =||Mask⊙(G(X occlusion , Mask)-X GT )||1 in, G stands for generator; X GT is the real image, X occlusion represents the input image with occlusion, Mask represents the mask of the occluder, G(X occlusion , Mask) is the occlusion recovery and reconstruction image; L G_L1 For the whole, L G_hole is the occlusion area, L G_valid is the visible area, L G_shfit The shift layer.
5. The image occlusion restoration and reconstruction method based on partial convolution and shifted deep feature rearrangement according to claim 1 is characterized in that, in step (4), an additional shift layer is added to the third layer of the deconvolution layer that introduces the connected convolutional features to perform deep feature rearrangement; U y =X*(y)-y in, U y represents the displacement vector, X*(y) represents the information in the occluded area decoder, and y represents the real information of the occluded area.
6. According to the image occlusion recovery and reconstruction method based on partial convolution and shifted deep feature rearrangement as described in claim 1, it is characterized in that in step (5), the occlusion recovery and reconstruction image 256*256 obtained by the generator and the image 256*256 without occlusion added are added to the discriminator.
7. The image occlusion restoration and reconstruction method based on partial convolution and shifted deep feature rearrangement according to claim 1 is characterized in that in step (6), the optimization objective function of the discriminator is: L D =E[log P(D=real|X GT )]+E[log P(D=fake|G(X occlusion , Mask))] in, E stands for entropy, D stands for discriminator, G stands for generator; X GT is a real image, D = real means that the output of the discriminator is expected to be a real image; X occlusion represents the input image with occlusion, Mask represents the mask of the occluder, G(X occlusion , Mask) is the occlusion recovery and reconstruction image, D = fake means that the output of the discriminator is expected to be the occlusion recovery and reconstruction image; the structure of the discriminator is shown in the following table: 。 8. The image occlusion restoration and reconstruction method based on partial convolution and shifted deep feature rearrangement according to claim 1 is characterized in that in step (7), the optimization objective function of the generator is: L G =E[log P(D=real|G(X occkusion ,Mask))]+L G_L1 +L G_hole +L G_vaild +L G_shift in, E stands for entropy, D stands for discriminator, G stands for generator; X GT is a real image, D = real means that the output of the discriminator is expected to be a real image; X occlusion represents the input image with occlusion, Mask represents the mask of the occluder, G(X occlusion , Mask) is the occlusion recovery and reconstruction image, L G_L1 For the whole, L G_hole is the occlusion area, L G_valid is the visible area, L G_shfit is the shift layer; the structure of the generator is shown in the following table:
9. According to the image occlusion restoration and reconstruction method based on partial convolution and shifted deep feature rearrangement as described in claim 1, it is characterized in that in step (8), when the loss value no longer decreases significantly, the calculation of PSNR (Peak Signal-to-Noise Ratio) peak signal-to-noise ratio and SSIM (Structural SIMilarity) structural similarity both obtains high-quality results, thereby deciding whether to stop the iteration; the parameters of the generator network layer of the model are saved to a file on the computer hard disk.
10. According to the image occlusion restoration and reconstruction method based on partial convolution and shifted deep feature rearrangement as described in claim 1, it is characterized in that in step (9), the network layer structure model of the generator is defined, and the network layer parameters of the generator are loaded from the computer hard disk.
11. According to the image occlusion recovery and reconstruction method based on partial convolution and shifted deep feature rearrangement as described in claim 1, it is characterized in that, in step (10), the image to be occluded and reconstructed and the mask of the corresponding occluded object are input into the generator model loaded in step (9), and the image after occlusion recovery and reconstruction is output.
