A method and system for hole filling in virtual viewpoint synthesis

Abstract

The present invention provides a method and system for filling holes in virtual viewpoint synthesis. The method includes: a foreground extraction step, extracting a foreground mask that causes large holes in the virtual viewpoint from the original viewpoint image; a background completion step, first using an improved spatiotemporal filling method for filling holes, specifically registering several previous and subsequent frames to the current frame. During the registration process, optical flow tracking is performed on the feature points of the matching pairs to eliminate incorrect matching pairs. The homography matrix calculated after the optical flow tracking is then used for image splicing to achieve partial background filling; then, an improved spatial filling method is used for filling holes, specifically assigning a priori values ​​to the initial bias of pixels when using the PatchMatch algorithm to obtain a completed occluded background; and a background re-transformation step, transforming the completed occluded background into the virtual viewpoint. The present invention can improve the quality of virtual viewpoint synthesis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a hole filling method and system in virtual viewpoint synthesis. BACKGROUND

[0002] At present, 3D display technology has penetrated into all aspects of people's life, and the image seen by human eyes when watching an object has a bit difference, the deviation between two images is called binocular disparity, and the image with the disparity is called a stereoscopic image pair. In order to form a stereoscopic image pair, a virtual view needs to be synthesized, and the human brain can perceive the difference of the stereoscopic image pair to produce a stereoscopic effect, so the most important step in 3D is the synthesis of a virtual viewpoint.

[0003] At present, the most commonly used virtual viewpoint synthesis method is depth image based rendering (DIBR). The method synthesizes a virtual view through an image captured by a real camera and a corresponding depth map. In the process of synthesizing a virtual view using DIBR, three-dimensional image warping needs to be performed. The essence of three-dimensional image warping is to project a pixel in a reference image captured by a real camera to a corresponding pixel position in a virtual view. Compared with a traditional three-dimensional model based rendering method, DIBR can more conveniently synthesize a virtual viewpoint image at a receiving end. However, due to the discontinuity of depth values in a depth map, a large number of holes will be generated in a target view obtained through three-dimensional image warping operation, and the generation of the holes seriously affects the quality of a synthesized view. Filling the holes to obtain a real and natural target view has become a challenging topic in the field of 3D.

[0004] The method of hole filling can be divided into two categories according to the time sequence of the virtual viewpoint synthesis: depth map preprocessing and post-processing of the synthesized view. In the hole filling method based on depth map preprocessing, Fehn et al. use a Gaussian filter to preprocess the depth map. After that, many different filters are used to smooth the depth map, such as the asymmetric Gaussian filter used by Zhang et al. to reduce the geometric distortion phenomenon in the synthesized view; Daribor et al. use an edge-dependent depth filter to perform filtering operations only near the edges of the depth map. Experimental results show that this method can produce obvious geometric distortion, and although this method can solve part of the hole problem, the remaining holes still need to be filled by other methods. The method based on post-processing of the synthesized view can be divided into two categories: spatial domain-based and spatio-temporal domain-based hole filling. The difference between the two is that the spatial domain-based method only uses one frame of image information to fill the hole, while the spatio-temporal domain-based hole filling uses multiple frames of image information in the time domain. In the spatial domain-based hole filling method, M. Schmeing et al. directly propagate the texture near the hole to the hole area by interpolation method. Since the interpolation method is independent of each row in the view, it will cause visual artifacts; Moi et al. combine depth map preprocessing and image inpainting algorithm proposed by Telea to fill the hole. However, this method does not consider the relationship between the foreground and background in the target image, and there is a phenomenon of filling the background hole area with foreground texture; Criminisi et al. proposed a sample-based image inpainting, which pointed out a new direction for solving the problem of large-size hole filling in DIBR system; in recent years, convolutional neural network has a good application in image inpainting, Yu et al. proposed a method based on deep generative model, which has a context attention layer for image inpainting, uses the features of known patches as convolution filters to process generated patches. Although the image inpainting method based on convolutional neural network can repair the content of the unknown hole area, there is still a problem of filling the background hole with foreground texture. In the spatio-temporal domain-based hole filling method, Guibo Luo et al. proposed a hole filling algorithm based on background reconstruction. This algorithm first extracts the foreground in the virtual view using the random walk segmentation algorithm, then uses the random sample consensus algorithm, Gaussian mixture model and Criminisi image inpainting algorithm to reconstruct the background information in the image, and finally uses the reconstructed background information and the foreground information in the virtual view to synthesize the target view. However, when using global homography matrix for image registration, there is a problem of inaccurate image stitching alignment, and the time complexity is high when using Criminisi algorithm to reconstruct the background information in the image.

[0005] In summary, the existing virtual viewpoint hole filling method has the following shortcomings:

[0006] 1. The method of depth map preprocessing can only fill part of the holes, and cannot fill the holes caused by foreground occlusion. Meanwhile, this method can also cause obvious geometric distortion;

[0007] 2. The filling method based on spatial domain can cause visual artifacts when filling the foreground into the background;

[0008] 3. The filling method based on spatial-temporal domain can cause visual artifacts when using global homography matrix for image registration, and can still use other methods for filling when the method based on spatial-temporal domain cannot completely fill the holes. SUMMARY

[0009] The present application provides a hole filling method in virtual view synthesis, which combines the filling method based on spatial-temporal domain and the filling method based on spatial domain. In the filling method based on spatial-temporal domain, the method of optical flow tracking is used to eliminate the feature matching pairs on dynamic objects, effectively solving the problem of inaccurate image registration. When the filling method based on spatial-temporal domain no longer reduces the area of the foreground mask, the filling method based on spatial domain is used to fill the background again, so that the filling of the background maximally uses the elements of the real scene. In the process of spatial domain filling, the foreground causing large holes is extracted to avoid filling the foreground content into the background. In the process of filling the background using the PatchMatch algorithm, the feature information of the image is combined to make the search range of the algorithm more accurate, and the filled background image is more reasonable in vision, thereby improving the quality of virtual view synthesis.

[0010] The present application adopts the following technical solutions:

[0011] In one aspect, a hole filling method in virtual view synthesis comprises:

[0012] A foreground extraction step extracts the foreground mask causing large holes in the virtual view from the original view point image;

[0013] A background completion step first uses the improved spatial-temporal domain filling method for filling. Specifically, several frames before and after the current frame are registered to the current frame. In the process of registration, the feature points of the matching pairs are tracked by optical flow to eliminate the wrong matching pairs. After the optical flow tracking, the homography matrix is calculated to perform image stitching, thereby achieving partial background filling. Then, the improved spatial domain filling method is used for filling. Specifically, the initial bias of the pixels is assigned in advance when the PatchMatch algorithm is used, so as to obtain the completed occluded background.

[0014] An occluded background transformation step transforms the completed occluded background to the virtual view point.

[0015] Preferably, the foreground extraction step specifically comprises:

[0016] Based on the depth information and camera parameters, the original viewpoint 3D is transformed into the right virtual viewpoint image and the left virtual viewpoint image containing holes using the DIBR virtual viewpoint synthesis method;

[0017] The small holes in the right virtual viewpoint image and the left virtual viewpoint image are filled using the Telea method;

[0018] The right virtual viewpoint image hole edge and the left virtual viewpoint hole edge are 3D inverse transformed to the original viewpoint to obtain the foreground right side edge of the original viewpoint forming a hole and the foreground left side edge of the original viewpoint forming a hole, respectively, and the two are combined to obtain the foreground edge of the original viewpoint forming a hole, and morphological dilation is performed to obtain the dilated original viewpoint foreground contour;

[0019] The original viewpoint depth map is subjected to joint bilateral filtering, morphological dilation edge detection and corrosion edge detection to obtain a depth map foreground contour, and the intersection of the depth map foreground contour and the original viewpoint foreground contour is obtained to obtain foreground seed points and background seed points;

[0020] In the original viewpoint image, a random walk segmentation algorithm is used to extract the original viewpoint foreground f1 and its mask m1;

[0021] According to the original viewpoint depth map and the original viewpoint foreground contour, a first threshold is set to obtain a foreground mask m2;

[0022] The intersection of m1 and m2 is obtained to obtain the final original viewpoint foreground mask.

[0023] Preferably, the filling is performed using an improved space-time filling method, which specifically includes:

[0024] Feature points are extracted from each frame of the original viewpoint to obtain feature points and descriptors, and feature matching is performed with several frames before and after to obtain initial matching pairs, and the corresponding feature points are denoted as kp_d. The first frame is matched backward, and the last frame is matched forward. The matching cutoff condition is that the foreground mask area of the current frame is no longer decreasing or reaches a set number of matching frames.

[0025] Optical flow tracking is performed on the feature points kp_o of the current frame to obtain corresponding tracking points kp_op on a certain frame;

[0026] When the distance between kp_d and kp_op is greater than a set second threshold, it is determined that the matching pair is incorrect. The final matching pair is obtained by removing the incorrect matching through the optical flow method, and the background of the current frame of the original viewpoint is obtained after calculating the homography matrix and performing image stitching.

[0027] Perspective transformation is performed on the foreground mask of the current frame of the original viewpoint to obtain the hole mask of the current frame of the original point.

[0028] Preferably, the filling is performed using an improved space-time filling method, which specifically includes:

[0029] based on the original viewpoint current frame background, the original viewpoint current frame hole mask, the current frame feature point and the descriptor, the current frame is blocked to obtain pure background block, background block containing hole and pure hole block;

[0030] For pure background block, the initial bias is set to 0; for pure hole block, the initial bias is set to random; for background block containing hole, it is extended in the up, down, left and right four directions, and the feature matching is carried out with the pure background block to obtain the pure background block with the most feature matching pairs as the best matching, and the initial bias of the pixel to be filled in the background block containing hole is set to the difference value between the pixel coordinates in the best matching pure background block and the coordinates thereof.

[0031] The improved PatchMatch algorithm is used to fill the original viewpoint background after filling; wherein the initial bias of each block is used as the initial value in the initialization process of the PatchMatch algorithm.

[0032] Preferably, the occluded background retransforming step specifically comprises:

[0033] Based on the completed background, the original viewpoint current frame background mask and the background depth map, 3D transformation is carried out to obtain the right virtual viewpoint image and the left virtual viewpoint image.

[0034] It is judged whether the right edge of the right virtual viewpoint and the left edge of the left virtual viewpoint still exist holes, if not, the right virtual viewpoint image and the left virtual viewpoint image without holes are obtained; if yes, the filling method based on the space domain is repeated to fill until there is no hole.

[0035] Preferably, the method for obtaining the background depth map comprises:

[0036] Edge detection is carried out on the foreground mask of the current frame to obtain the edge points to be filled with depth, and a square block with a customizable side length is obtained with any point as the center.

[0037] A plane equation in the square block is fitted through the random sample consensus algorithm, and the depth value of the depth unknown point is calculated according to the fitted plane equation, and the depth map and the corresponding mask of the current frame are updated;

[0038] After the edge points are traversed, edge detection is carried out again on the updated mask, and the above steps are repeated until the depth unknown points are completely filled, and the complete background depth map is obtained.

[0039] On the other hand, a hole filling system in virtual viewpoint synthesis comprises:

[0040] A foreground extraction module is used to extract the foreground mask leading to large holes in the virtual viewpoint in the original viewpoint image.

[0041] The background completion module firstly uses the improved space-time field filling method to fill, specifically, a plurality of frames before and after are registered to the current frame, in the registration process, the feature points of the matching pair are tracked by the optical flow to eliminate the error matching pair, the homography matrix is calculated after the optical flow tracking processing, the image splicing is implemented, and the partial background filling is realized, and then the improved space field filling method is used to fill, specifically, the initial bias of the pixel is valued in advance when the PatchMatch algorithm is used, so that the completed occluded background is obtained.

[0042] The occluded background transformation module is used to transform the completed occluded background into a virtual viewpoint.

[0043] The present application has the following beneficial effects:

[0044] (1) In the foreground extraction process of the present application, only the foreground causing the hole is extracted, and unnecessary area filling in the subsequent filling process is reduced.

[0045] (2) In the image registration process according to the homography matrix, the feature points of the matching pair are tracked by the optical flow, which can effectively solve the problem of inaccurate image splicing alignment caused by the influence of dynamic objects.

[0046] (3) In the filling process of the occluded background still existing after the space-time field filling of the present application, the pixel bias of the PatchMatch algorithm is valued in advance by using the image block feature matching, and the image restoration effect is improved.

[0047] The present application will be further described in detail in combination with the drawings and embodiments, but the present application is not limited to the embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The flow chart of the hole filling method in the virtual viewpoint synthesis of the embodiment of the present application;

[0049] Figure 2 The detailed flow chart of the foreground extraction step of the embodiment of the present application;

[0050] Figure 3 The detailed flow chart of the background completion step of the embodiment of the present application;

[0051] Figure 4 The experimental effect comparison chart of the error matching eliminated by the optical flow method of the embodiment of the present application, wherein (a) represents the image without using the optical flow method to eliminate, and (b) represents the image using the optical flow method to eliminate;

[0052] Figure 5 The block diagram of the embodiment of the present application;

[0053] Figure 6 a detailed flow chart of the occluded background re-transforming step of the embodiment of the present application;

[0054] Figure 7 a structural block diagram of the virtual viewpoint synthesis hole filling system of the embodiment of the present application.

[0055] The present application is further described in detail below with reference to the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0056] The present application is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] Referring to Figure 1 the present embodiment of a virtual viewpoint synthesis hole filling method, comprising:

[0058] a foreground extraction step S101 of extracting a foreground mask causing a large hole in the virtual viewpoint from the original viewpoint image;

[0059] a background completion step S102 of first using an improved space-time domain filling method for filling, specifically, registering a plurality of frames before and after the current frame, in the process of registration, tracking the feature points of the matching pairs by optical flow to eliminate the wrong matching pairs, calculating the homography matrix after the optical flow tracking process to perform image stitching, and realizing partial background filling; then using an improved space domain filling method for filling, specifically, assigning an initial bias of pixels to obtain the completed occluded background by using the PatchMatch algorithm.

[0060] an occluded background re-transforming step S103 of transforming the completed occluded background into the virtual viewpoint.

[0061] The present embodiment combines the space-time domain based filling method and the space domain based filling method. In the space-time domain based filling method, to solve the problem of inaccurate image stitching alignment caused by using the global homography matrix, the method of optical flow tracking is adopted to eliminate the feature matching pairs on the dynamic objects. The experimental results show that the problem of inaccurate stitching alignment can be effectively solved. When the space-time domain based filling no longer reduces the area of the foreground mask, the space domain based method is used to fill the background again, so that the background filling maximally uses the elements of the real scene. In the process of space domain filling, according to the mechanism of large hole formation, the foreground causing the large hole is extracted to avoid filling the foreground content into the background. In the process of filling the background by using the PatchMatch algorithm, the feature information of the image is combined to make the search range of the algorithm more accurate, and the filled background image is more reasonable in vision.

[0062] It should be noted that, in this embodiment, if the hole area is greater than 10% of the total pixels of the virtual viewpoint image, it can be recorded as a large hole, otherwise it is a small hole. The specific percentage depends on the parallax between the original viewpoint and the virtual viewpoint and experience, and this embodiment does not impose any specific restrictions.

[0063] In this embodiment, the foreground extraction step S101 is to extract the foreground that causes the large hole in the virtual viewpoint from the original viewpoint. For the specific processing process, see Figure 2 shown.

[0064] The most commonly used technique in virtual viewpoint synthesis is DIBR, which transforms the original viewpoint image into a virtual viewpoint using depth information and camera parameters according to formula (1).

[0065]

[0066] Among them, [x1,y1,1] T Represents the homogeneous coordinates of the original viewpoint image, [x2,y2,1] T represents the homogeneous coordinates of the virtual viewpoint image, K1 represents the original viewpoint camera intrinsics, K2 represents the virtual viewpoint camera intrinsics, R1 represents the original viewpoint rotation matrix, R2 represents the virtual viewpoint rotation matrix, t1 represents the original viewpoint translation matrix, t2 represents the virtual viewpoint translation matrix, Zc1 represents the original viewpoint pixel depth, and Zc2 represents the virtual viewpoint pixel depth.

[0067] In the above formula, let [x left ,y left ,1] T represents the homogeneous coordinates of the left virtual viewpoint image, K left is the left virtual viewpoint camera internal parameter, R left is the left virtual viewpoint rotation matrix, t left is the left virtual viewpoint translation matrix, Z cleft is the pixel depth of the left virtual viewpoint; [x right ,y right ,1] T represents the homogeneous coordinates of the right virtual viewpoint image, K right is the right virtual viewpoint camera internal parameter, R right is the right virtual viewpoint rotation matrix, t righr is the right virtual viewpoint translation matrix, Z cright is the pixel depth of the right virtual viewpoint. Then according to the formula

[0068]

[0069] The left virtual viewpoint image can be obtained.

[0070] According to the formula

[0071] A right virtual viewpoint image can be obtained.

[0072] In the present application, first, the left virtual viewpoint image and the right virtual viewpoint image are respectively transformed according to formula (1), and small holes are filled based on the Telea method. The fundamental reason for the large holes in the virtual viewpoint is that the visibility of the object in the scene has changed when the viewpoint changes, that is, the area (foreground occlusion) that is not visible in the original viewpoint image becomes visible in the virtual viewpoint image. In the random walk segmentation algorithm for foreground extraction, foreground seed points and background seed points need to be provided. The hole edges of the left virtual viewpoint and the right virtual viewpoint are back-transformed to the original viewpoint according to formula (1), so that the foreground edges causing the holes are obtained, and morphological dilation is performed. At the same time, after the original viewpoint depth map is subjected to joint bilateral filtering, morphological dilation edge detection and corrosion edge detection, the foreground edges obtained by back-transformation are intersected to obtain the foreground seed points and the background seed points. Subsequently, in the original viewpoint image, the random walk segmentation algorithm is used to extract the foreground f1 and its mask m1. However, the foreground extracted by the random walk segmentation algorithm will inevitably extract the background, so the first threshold can be set according to the original viewpoint depth map (since the foreground mask is obtained, the first threshold can be set as the absolute value of the difference between the maximum depth value and the minimum depth value of the five pixel points on the left and right of the foreground contour in the depth map, which can be adjusted according to needs), to obtain the foreground mask m2, and the intersection of m1 and m2 obtains a more accurate foreground mask.

[0073] Specifically, the back-transformation formula is as follows:

[0074]

[0075] wherein, [x1, y1, 1] T represents the homogeneous coordinates of the original viewpoint image, [x2, y2, 1] T represents the homogeneous coordinates of the virtual viewpoint image, K1 represents the original viewpoint camera intrinsic parameter, K2 represents the virtual viewpoint camera intrinsic parameter, R1 is the original viewpoint rotation matrix, R2 is the virtual viewpoint rotation matrix, t1 is the original viewpoint translation matrix. t2 is the virtual viewpoint translation matrix, Zc1 is the depth of the original viewpoint pixel point. Zc2 is the depth of the virtual viewpoint pixel point.

[0076] In the present embodiment, the background completion step S102 is to fill the background occluded by the foreground in the original viewpoint. The specific processing process is shown in Figure 3 .

[0077] After obtaining the foreground mask of each frame of the original viewpoint, the background of the current frame can be filled. The embodiment combines a background filling method based on a space-time domain and a background filling method based on a space domain. In the filling method based on the space-time domain, due to the existence of dynamic objects in the scene, the occluded background in the current frame can be exposed in other frames. Based on this assumption, the useful information in other frames needs to be filled into the current frame. The specific scheme is as follows. First, feature points of each frame of the original viewpoint are extracted, and feature matching is performed with several frames before and after. The corresponding feature points are denoted as kp_d. The first frame is matched backward, and the last frame is matched forward. The matching stop condition is that the foreground mask area of the current frame is no longer reduced or the number of matched frames reaches a set value. In the process of registering a certain frame to the current frame, due to the existence of dynamic objects, the homography matrix calculated will have a large error, which is reflected in the visual aspect that the splicing alignment is not accurate. The feature points kp_o of the current frame are tracked by the optical flow method, and the corresponding tracking points kp_op can be obtained on a certain frame. When the distance between kp_d and kp_op is greater than a second threshold value (the second threshold value is related to the accuracy of the final obtained homography matrix. In theory, the smaller the threshold value, the more accurate the homography matrix. However, the threshold value cannot be set too small. If the threshold value is too small, the number of final matching pairs obtained will be less than 4, which will result in the inability to obtain a homography matrix. In the embodiment, the threshold value can be set to 10 pixels), it can be determined that this matching pair is incorrect. The incorrect matching pairs are removed by the optical flow method, and the homography matrix is calculated to perform image splicing. The experimental results are shown in FIG. 8, which shows that the embodiment can effectively solve the problem of inaccurate splicing alignment. Among them, the matching pair records which feature points in other frames match which feature points in the current frame. Figure 4

[0078] Further, although the filling method based on the space-time domain can fill part of the background, when there is no dynamic object in the scene or the motion of the dynamic object is not obvious, there will still be a problem of incomplete background filling. Therefore, after the filling based on the space-time domain, a filling algorithm based on the space domain is added, which is an improved PatchMatch image repair algorithm. The PatchMatch algorithm includes initialization, propagation, and random disturbance search. In the present application, the improvement of the PatchMatch algorithm is mainly reflected in the initialization process. For details, see FIG. 9. Figure 5 As shown in FIG. 9, the gray part is the unfilled background, which needs to be filled. The white part is the existing background, which is the search area. The filled part of the current frame background obtained by the filling method based on the space-time domain, the feature points of the current frame, and the descriptor are divided into several blocks. According to the initialization process of the PatchMatch algorithm, Figure 5 ​The initial bias of the middle gray pixels needs to be assigned, and the white pixels belong to the search region, and the bias is assigned 0. For a block similar to B1 (a pure background block), most of which contains extracted feature points that can be matched with the blocks in the search area, if it does not contain feature points, such as B3 (a background block with a hole), it is expanded in the up, down, left and right four directions for matching, and the search block with the most matching number is the best match. The initial bias of the gray pixels in the B3 block is set to the difference between the pixel coordinates in the block matched with the feature and the coordinates thereof. For B2 (a pure hole block), the initial bias thereof is set to be random. Experimental results show that the image restoration result is better using the prior initial bias.

[0079] After the background image restoration is completed, the 3D conversion of the occluded background to the virtual viewpoint is required, and thus the depth value of the occluded background needs to be obtained. According to the foreground mask and the depth map of the current frame, the background depth map is obtained, the edge detection is performed on the background mask to obtain the edge points to be filled with the depth, the edge points are traversed, a square block with a definable side length is obtained with a certain point as the center, a plane equation in the square block is fitted through the random sampling consensus algorithm, the depth value of the depth-unknown point is calculated according to the fitted plane equation, and the depth map and the corresponding mask of the current frame are updated. In order to avoid the square block extraction on the edge points that have been fitted again, it is required to judge whether the current edge point is modified by the previous fitted plane. After the edge points are traversed, the edge detection is performed again on the updated mask, and the above-mentioned actions are repeated until the depth-unknown points are completely filled. Through the above-mentioned processing, the complete background depth map can be obtained.

[0080] In the embodiment, the specific processing process of the occluded background re-conversion step 103 is described with reference to the background image restoration method shown in Figure 6 .

[0081] Through formula (1), the occluded background in the background completion step is converted to the virtual viewpoint, and a large hole can be filled. It is judged whether there is still a hole at this time, and if there is, the improved spatial hole filling algorithm is used for filling, and the specific process is as follows:

[0082] The 3D conversion is performed based on the completed background, the background mask and the background depth map of the current frame of the original viewpoint to obtain the right virtual viewpoint image and the left virtual viewpoint image;

[0083] It is respectively judged whether there is still a hole on the right side edge of the right virtual viewpoint and the left side edge of the left virtual viewpoint, if there is no hole, the right virtual viewpoint image and the left virtual viewpoint image without the hole are obtained, and if there is a hole, the spatial-based filling method is repeated to fill until there is no hole.

[0084] In summary, in the foreground extraction process, only the foreground causing the hole is extracted, and the unnecessary region filling in the subsequent filling process is reduced; in the image registration process according to the homography matrix, by tracking the feature points of the matching pair, the problem of inaccurate image stitching alignment caused by the influence of dynamic objects can be effectively solved; after the filling based on the space-time domain, in the filling process of the still existing occluded background, the pixel bias of the PatchMatch algorithm is priorly valued by using the image block feature matching, and the image restoration effect is improved.

[0085] Referring to Figure 7 According to another aspect of the present application, the embodiment also discloses a virtual viewpoint synthesis hole filling system, comprising:

[0086] The foreground extraction module 701 is used for extracting the foreground mask causing the large hole in the virtual viewpoint in the original viewpoint image;

[0087] The background completion module 702 first uses the improved space-time domain filling method for filling, specifically, a plurality of frames before and after are registered to the current frame, in the registration process, the feature points of the matching pair are tracked to eliminate the error matching pair, the homography matrix is calculated after the optical flow tracking processing to perform image stitching, and partial background filling is realized; then, the improved space domain filling method is used for filling, specifically, the initial bias of the pixel is priorly valued when the PatchMatch algorithm is used, so as to obtain the completed occluded background;

[0088] The occluded background transformation module 703 is used for transforming the completed occluded background into the virtual viewpoint.

[0089] The specific implementation of the virtual viewpoint synthesis hole filling system is the same as the virtual viewpoint synthesis hole filling method, and the embodiment will not be repeated.

[0090] The above is only the preferred embodiment of the present application, and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for filling holes in virtual viewpoint synthesis, characterized in that: include: A foreground extraction step extracts the foreground mask that causes the large hole in the virtual viewpoint from the original viewpoint image; The background completion step first uses an improved spatiotemporal filling method to fill in the background. Specifically, several frames before and after are registered to the current frame. During the registration process, optical flow tracking is performed on the feature points of the matching pairs to eliminate incorrect matching pairs. The homography matrix calculated after optical flow tracking is used for image stitching to achieve partial background filling. Then, an improved spatial filling method is used to fill in the background. Specifically, the initial bias prior of the pixels is assigned when using the PatchMatch algorithm to obtain the completed occluded background. The occluded background is transformed again, transforming the completed occluded background into the virtual viewpoint; The improved spatiotemporal filling method is used for filling, specifically including: Feature points are extracted from each frame of the original viewpoint to obtain feature points and descriptors, and feature matching is performed with several frames before and after to obtain an initial matching pair. The corresponding feature points are recorded as kp_d. The first frame is matched backward and the last frame is matched forward. The matching cutoff condition is that the foreground mask area of ​​the current frame does not decrease or the set number of matching frames is reached. Perform optical flow tracking on the feature point kp_o of the current frame and obtain the corresponding tracking point kp_op on a certain frame; When the distance between kp_d and kp_op is greater than the set second threshold, the matching pair is determined to be wrong. The optical flow method is used to eliminate the wrong matches to obtain the final matching pair, and the homography matrix is ​​calculated and then image stitching is performed to obtain the background of the current frame of the original viewpoint. Perform perspective transformation on the foreground mask of the current frame of the original viewpoint to obtain the hole mask of the current frame of the original viewpoint; Use an improved void filling method for filling, including: Based on the current frame background of the original viewpoint, the hole mask of the current frame of the original point, the feature points of the current frame and the descriptor, the current frame is divided into blocks to obtain pure background blocks, background blocks with holes and pure hole blocks; For pure background blocks, the initial bias is set to 0; for pure hole blocks, the initial bias is set to random; for background blocks with holes, they are expanded in the upward, downward, left, and right directions, and feature matched with pure background blocks. The pure background block with the largest number of feature matching pairs is the best match, and the initial bias of the pixel to be filled in the background block with holes is set to the difference between the pixel coordinates in the best matching pure background block and its coordinates; The improved PatchMatch algorithm is used to fill the original viewpoint background after filling; wherein, the initial bias of each block mentioned above is used as the initial value in the initialization process of the PatchMatch algorithm; The step of re-transforming the blocked background specifically includes: Perform 3D transformation based on the completed background, the background mask of the current frame of the original viewpoint, and the background depth map to obtain the right virtual viewpoint image and the left virtual viewpoint image; Determine whether there are holes on the right edge of the right virtual viewpoint and the left edge of the left virtual viewpoint respectively. If not, obtain a right virtual viewpoint image and a left virtual viewpoint image without holes; if so, repeat the airspace-based filling method to fill until there are no holes.

2. The method for filling holes in virtual viewpoint synthesis according to claim 1, wherein: The foreground extraction step specifically includes: Based on the depth information and camera parameters, the DIBR virtual viewpoint synthesis method is used to transform the original viewpoint 3D into a right virtual viewpoint image and a left virtual viewpoint image containing holes; Small holes in the right and left virtual viewpoint images are filled using the Telea-based method; The hole edges of the right virtual viewpoint image and the left virtual viewpoint are 3D inversely transformed to the original viewpoint to obtain the right edge of the foreground where the hole is formed by the original viewpoint and the left edge of the foreground where the hole is formed by the original viewpoint, respectively. The two edges are combined to obtain the foreground edge where the hole is formed by the original viewpoint, and morphological dilation is performed to obtain the expanded foreground contour of the original viewpoint; The original viewpoint depth map is subjected to joint bilateral filtering, morphological dilation and erosion edge detection to obtain the foreground contour of the depth map, which is then intersected with the original viewpoint foreground contour to obtain the foreground seed points and background seed points. In the original viewpoint image, the random walk segmentation algorithm is used to extract the original viewpoint foreground f1 and its mask m1; According to the original viewpoint depth map and the original viewpoint foreground contour, a first threshold is set to obtain a foreground mask m2; Intersect m1 and m2 to get the final original viewpoint foreground mask.

3. The method for filling holes in virtual viewpoint synthesis according to claim 1, wherein: Methods for obtaining a background depth map include: Perform edge detection on the foreground mask of the current frame to obtain the edge points to be filled with depth, traverse these edge points, and obtain a square block with customizable side length with any point as the center; Through the random sampling consistency algorithm, a plane equation is fitted within the square block. According to the fitted plane equation, the depth value of the unknown depth point is calculated, and the depth map of the current frame and the corresponding mask are updated at the same time; After traversing the edge points, edge detection is performed again on the updated mask, and the above steps are repeated until the points with unknown depth are completely filled and a complete background depth map is obtained.

4. A hole filling system in virtual view synthesis, characterized in that: include: A foreground extraction module is used to extract the foreground mask that causes the large hole in the virtual viewpoint in the original viewpoint image; The background completion module first uses an improved spatiotemporal filling method to fill in the background. Specifically, several frames before and after are registered to the current frame. During the registration process, optical flow tracking is performed on the feature points of the matching pairs to eliminate incorrect matching pairs. The homography matrix calculated after optical flow tracking is used for image stitching to achieve partial background filling. Then, an improved spatial filling method is used to fill in the background. Specifically, the initial bias prior of the pixels is assigned when using the PatchMatch algorithm to obtain the completed occluded background. The occluded background re-transformation module is used to transform the completed occluded background into the virtual viewpoint; The improved spatiotemporal filling method is used for filling, specifically including: Feature points are extracted from each frame of the original viewpoint to obtain feature points and descriptors, and feature matching is performed with several frames before and after to obtain an initial matching pair. The corresponding feature points are recorded as kp_d. The first frame is matched backward and the last frame is matched forward. The matching cutoff condition is that the foreground mask area of ​​the current frame does not decrease or the set number of matching frames is reached. Perform optical flow tracking on the feature point kp_o of the current frame and obtain the corresponding tracking point kp_op on a certain frame; When the distance between kp_d and kp_op is greater than the set second threshold, the matching pair is determined to be wrong. The optical flow method is used to eliminate the wrong matches to obtain the final matching pair, and the homography matrix is ​​calculated and then image stitching is performed to obtain the background of the current frame of the original viewpoint. Perform perspective transformation on the foreground mask of the current frame of the original viewpoint to obtain the hole mask of the current frame of the original viewpoint; Use an improved void filling method for filling, including: Based on the current frame background of the original viewpoint, the hole mask of the current frame of the original point, the feature points of the current frame and the descriptor, the current frame is divided into blocks to obtain pure background blocks, background blocks with holes and pure hole blocks; For pure background blocks, the initial bias is set to 0; for pure hole blocks, the initial bias is set to random; for background blocks with holes, they are expanded in the upward, downward, left, and right directions, and feature matched with pure background blocks. The pure background block with the largest number of feature matching pairs is the best match, and the initial bias of the pixel to be filled in the background block with holes is set to the difference between the pixel coordinates in the best matching pure background block and its coordinates; The improved PatchMatch algorithm is used to fill the original viewpoint background after filling; wherein, the initial bias of each block mentioned above is used as the initial value in the initialization process of the PatchMatch algorithm; The obscured background re-transformation module specifically includes: Perform 3D transformation based on the completed background, the background mask of the current frame of the original viewpoint, and the background depth map to obtain the right virtual viewpoint image and the left virtual viewpoint image; Determine whether there are holes on the right edge of the right virtual viewpoint and the left edge of the left virtual viewpoint respectively. If not, obtain a right virtual viewpoint image and a left virtual viewpoint image without holes; if so, repeat the airspace-based filling method to fill until there are no holes.

Images