A method for filling holes in 3D video based on depth image edge detection

Abstract

This invention discloses a method for filling holes in 3D videos based on depth image edge detection, primarily addressing the hole problem that arises when mapping virtual viewpoints within a field of view using depth information from multiple reference viewpoints. The method involves edge detection extraction of the depth maps of each fusion image before virtual viewpoint image fusion, identifying edge and non-edge regions. Then, in non-edge regions, distances before and after mapping are used to fill holes. Subsequently, the filled images are fused using a voting fusion mechanism, followed by fusion of complementary mapping results for virtual viewpoints at each angle within the field of view. After virtual viewpoint image fusion, pixel data from adjacent virtual viewpoints or reference viewpoints are used to fill holes in the current virtual viewpoint. This invention effectively improves the efficiency of hole filling in virtual views by combining edge detection processing before image fusion with post-fusion filling.

Description

Technical Field

[0001] This invention relates to a method for filling holes in 3D videos based on depth image edge detection, belonging to the field of 3D reconstruction technology. Background Technology

[0002] In the process of 3D scene reconstruction, virtual viewpoint generation algorithms can significantly reduce the difficulty of information acquisition and computational complexity. By inputting a small number of reference viewpoint images and combining them with depth information as guidance, virtual views from any viewpoint can be generated within a certain range. After these generated virtual viewpoint views are synthesized, the effect of producing corresponding views according to changes in the observer's viewpoint is achieved, thus successfully reconstructing the 3D scene.

[0003] Holes often appear in the generation of views from virtual viewpoints, both before and after compositing. Furthermore, as the field of view increases, information loss and computational errors lead to numerous holes in the generated virtual viewpoint. These holes can be categorized into out-of-view holes and occlusion holes: Out-of-view holes are typically located at the left or right boundaries of the generated virtual view. These holes occur primarily because the imaging range of the virtual viewpoint exceeds the imaging range of the input reference viewpoint, which cannot provide sufficient boundary information to fill the extra boundary areas seen in the virtual viewpoint. Occlusion holes manifest as black holes at the boundary between the foreground and background, and can also appear as large-scale missing foreground structures. These missing structures appear as small cracks in areas with relatively gentle depth changes, and as severe tears in the foreground structure in areas with drastic depth changes. These holes occur primarily because the occlusion relationships of objects change during the generation of the virtual viewpoint. Some objects or structures that are blocked in the input viewpoint are exposed in the generated virtual viewpoint, but the input viewpoint cannot provide sufficient information to fill these exposed areas, thus creating holes.

[0004] To effectively fill the numerous holes generated during virtual viewpoint generation, a 3D video hole-filling method based on depth image edge detection is proposed. This method takes two viewpoint images with a fixed angle as input, calculates the virtual viewpoint corresponding to the same angle, and fuses these two complementary mapped virtual viewpoint views. Image fusion can effectively fill holes outside the field of view, but occlusion holes generated by the virtual viewpoint generation images are most noticeable at the image edges. Therefore, using edge detection combined with mapping distance judgment for hole filling before image fusion will effectively fill occlusion holes, ultimately outputting a video that conforms to the parameters of a naked-eye 3D screen. Summary of the Invention

[0005] The purpose of this invention is to provide a method for filling holes in 3D videos based on depth image edge detection. In the process of generating a naked-eye 3D video source by inputting RGB images of three reference viewpoints (left, center, and right) and their corresponding depth images, this method uses edge detection and mapping distance to determine whether to fill holes before virtual viewpoint fusion, to fill holes after fusion, and to fill holes after fusion. This effectively reduces out-of-field holes and occlusion holes in the video source, thereby improving the display quality of the naked-eye 3D video source.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows: a method for filling holes in a three-dimensional video based on depth image edge detection, comprising the following steps:

[0007] Step 1) Input the orthogonal projection RGB images of three reference viewpoints (left, center, and right) and their corresponding depth images. Map multiple virtual viewpoints based on the depth information of the depth maps of each reference viewpoint to generate virtual views at different viewpoint positions. For holes in the generated virtual views, use image edge detection processing on the depth map to determine the edge and non-edge regions of the reference viewpoint images.

[0008] Step 2) Calculate the distance before and after the image pixel coordinate mapping, and judge each pixel within a certain range. If these pixels are in the edge area, no hole filling is performed. If these pixels are in the non-edge area, all pixels in the area are filled with the pixels with the corresponding pixel coordinates before the original image mapping to avoid creating holes.

[0009] Step 3) The complementary mapping results at each virtual viewpoint after filling the holes are fused according to the voting fusion method. When there are holes in the current pixel in the fused virtual viewpoint image, pixels from other reference virtual viewpoints are used to fill them.

[0010] As an improvement of the present invention, the left, center and right reference viewpoint images input in step 1) are acquired by a surround camera array, with each camera distributed on an arc. The left and right reference viewpoints determine the field of view of the naked-eye 3D video, and the generated multiple virtual viewpoints are within the field of view.

[0011] As an improvement of the present invention, step 1) is performed by mapping multiple virtual viewpoints based on the depth information of each reference viewpoint in a row-by-row manner. Therefore, the determination of the distance between the front and back of pixel coordinates is also performed in a row-by-row manner, and the hole filling is also performed in a row-by-row manner.

[0012] As an improvement of the present invention, in step 1), during the virtual viewpoint mapping process, holes outside the field of view and occlusion holes are generated. The image processing before fusion is mainly to fill the occlusion holes.

[0013] As an improvement of the present invention, step 1) uses an edge detection algorithm to identify the image edges of the depth map of each reference viewpoint (Canny edge detection is recommended), sets the pixel edge flag of the image edge region to one, and sets the pixel edge flag of the non-edge region to zero, thereby distinguishing the edge region and non-edge region of the image.

[0014] As an improvement of the present invention, errors will occur in the calculation process of the reference viewpoint pixel coordinates to the virtual viewpoint image pixel coordinates in step 2), causing originally adjacent pixels to be non-adjacent after mapping, resulting in holes between non-adjacent pixels.

[0015] As an improvement of the present invention, in step 2), the difference between the current pixel mapping coordinate and the previous pixel mapping coordinate is calculated. If the difference is within a preset range, all pixel coordinates within the interpolation are sequentially subjected to edge judgment. Pixels in non-edge areas are filled with the original pixels to fill all generated holes, while pixels in edge areas are not filled with holes.

[0016] As an improvement of the present invention, the difference in distance between the image pixel coordinates before and after the mapping in step 2) should not be too large. If the range is too large, most of the real holes will be filled, which will affect the subsequent image fusion.

[0017] As an improvement of the present invention, in step 3), the virtual view generated by each virtual viewpoint is formed by fusing the mapping results of two reference views from the left and right sides of the virtual viewpoint position.

[0018] As an improvement of the present invention, the voting fusion method in step 3) can effectively fill large-area real holes and out-of-view holes in 3D videos. This method introduces a hole flag (i.e., filling in pixel flag position one, with the hole set to zero) and a mapping angle for voting fusion. The fusion process is expressed by the following formula:

[0019]

[0020] Where α is the angle between the generated virtual viewpoint and the intermediate main reference viewpoint, α max P is the field of view angle enclosed by the left and right reference viewpoints. When α is negative, the virtual viewpoint is located between the left reference viewpoint and the central main reference viewpoint; when α is positive, the virtual viewpoint is located between the central main reference viewpoint and the right reference viewpoint. V P represents the fusion result of generating virtual views. M P L and P R F represents the mapping results of the middle reference view, left reference view, and right reference view at the virtual viewpoint. M F L and F RThese represent the hole flags indicating the mapping results of the middle reference view, left reference view, and right reference view at the virtual viewpoint, respectively.

[0021] As an improvement of the present invention, in step 3), the fused virtual view selects adjacent pixels or other adjacent reference viewpoints to fill the holes of the current virtual viewpoint. It is recommended to use the pixels corresponding to adjacent reference viewpoints to fill the holes first. If the pixels corresponding to adjacent reference viewpoints are all holes, then the adjacent pixels of the fused image are used to fill the holes.

[0022] Compared to existing technologies, the advantages of this solution are as follows: Under normal circumstances, in the virtual view generated by fusion after mapping the reference view, holes are more likely to appear in locations with large depth differences. These locations with large depth differences are generally the edge locations of the image. Therefore, this invention performs edge detection processing on the original image before virtual viewpoint image fusion and focuses on pre-filling holes in the edge areas that are prone to holes, which greatly reduces the occurrence of holes in the virtual view generated after fusion. Finally, the fusion-filled holes are added, which further improves the efficiency of hole filling in the virtual view. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the void filling process of the present invention;

[0024] Figure 2 This is a schematic diagram showing the arrangement of the left, center, and right reference viewpoints and the virtual viewpoint in Example 1;

[0025] Figure 3 This is a schematic diagram of the mapping calculation and the holes caused by the calculation error in Example 1;

[0026] Figure 4 This is a schematic diagram of the subtraction interval of the mapped pixel positions in Example 1;

[0027] Figure 5 This is a schematic diagram illustrating the process of determining and filling holes based on edge regions in Example 1.

[0028] Figure 6 This is a schematic diagram of virtual viewpoint fusion within the field of view in Example 1;

[0029] Figure 7 This is a schematic diagram of filling holes after virtual view fusion in Example 1. Detailed Implementation

[0030] The invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, those skilled in the art will find that various equivalent modifications to the invention fall within the scope defined by the appended claims.

[0031] Example 1: See Figures 1-7 A method for filling holes in 3D videos based on depth image edge detection, the method comprising the following steps:

[0032] Step 1) Input the orthogonal projected RGB images of three reference viewpoints (left, center, and right) and their corresponding depth images. These reference viewpoint images are acquired using a surround camera array, where the left and right reference viewpoints determine the field of view of the naked-eye 3D video. Based on the depth information of each reference viewpoint, multiple virtual viewpoints within the field of view are mapped. The arrangement of the reference viewpoints and virtual viewpoints is as follows: Figure 2 As shown, the virtual view on the left is obtained by mapping the left reference view and the middle reference view, while the virtual view on the right is obtained by mapping the right reference view and the middle reference view. The field of view of the naked-eye 3D video is α. max .

[0033] The virtual view is mapped by traversing the row pixels of the reference viewpoint and calculating its mapped position in the virtual view based on the position and corresponding depth information of each pixel. Since the calculation result may be a decimal, it is rounded to the nearest integer. However, this rounding operation may cause discontinuities in the mapped coordinates, resulting in holes. A detailed mapping diagram and the generation of holes are shown below. Figure 3 As shown.

[0034] Edge detection is performed on the depth map of each reference viewpoint to identify edge and non-edge regions. The calculated mapping position in the virtual view is subtracted from the previously calculated mapping position to obtain a subtraction interval for the mapped pixel positions, such as... Figure 4 As shown.

[0035] If this interval is within the set range (generally between 1 and 4) and is located in the edge region, no hole filling is performed; if this interval is within the set range and is not located in the edge region, the hole flag of this mapped position interval is set to 1, and the pixel data is the pixel data of the reference view before the mapping calculation, thus effectively filling the holes. Figure 5 As shown.

[0036] Step 2) Two virtual views correspond to the same virtual viewpoint. The virtual view of the left virtual viewpoint is obtained by mapping the left reference viewpoint and the middle reference viewpoint, while the virtual view of the right virtual viewpoint is obtained by mapping the right reference viewpoint and the middle reference viewpoint. The two virtual views of the same viewpoint are merged according to a predetermined fusion rule. Hole filling is achieved through a voting fusion method. While ensuring that valid information from each mapping result is extracted, voting fusion is performed based on the hole flag and mapping angle to fill the holes in each virtual viewpoint. Taking the left virtual viewpoint as an example, the specific fusion diagram is as follows: Figure 6 As shown, where VL2M V is a virtual view obtained by mapping the left reference view to the center. M2L V is a virtual view obtained by mapping a central reference view to the left. L This is the result of merging two virtual views.

[0037] Step 3) When rearranging the fused virtual viewpoints according to the parameters of the naked-eye 3D screen, if the virtual viewpoint to which the current pixel belongs is empty, pixels at the same position in other viewpoints can be selected to fill it. The filling process is as follows: Figure 7 As shown, taking the left virtual view Vn row as an example, during the filling process, pixels from adjacent virtual viewpoints, the left reference viewpoint, or the main reference viewpoint are preferentially selected to fill the holes in the current virtual viewpoint. Considering the high consistency of pixels from adjacent viewpoints, pixels from other virtual viewpoints that are as close as possible to the current virtual viewpoint are selected for filling.

[0038] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for filling holes in 3D video based on depth image edge detection, characterized in that, Includes the following steps: Step 1) Input the orthogonal projection RGB images of three reference viewpoints (left, center, and right) and their corresponding depth images. Map multiple virtual viewpoints based on the depth information of the depth images of each reference viewpoint to generate virtual views at different viewpoint positions. For holes in the generated virtual views, use image edge detection processing on the depth images to determine the edge and non-edge regions of the reference viewpoint images. Step 2) Calculate the distance before and after the image pixel coordinate mapping, and judge each pixel within a certain range. If these pixels are in the edge area, no hole filling is performed. If these pixels are in the non-edge area, all pixels in the non-edge area are filled with the pixels with the corresponding pixel coordinates before the original image mapping to avoid creating holes. Step 3) The complementary mapping results at each virtual viewpoint after filling the holes are fused according to the voting fusion method. When there are holes in the current pixel in the fused virtual viewpoint image, pixels from other reference viewpoints are used to fill them. In the virtual viewpoint mapping process of step 1), holes outside the field of view and occlusion holes are generated. The image processing before fusion is mainly to fill the occlusion holes. In step 2), the difference between the current pixel mapping coordinates and the previous pixel mapping coordinates is calculated. If the difference is within a preset range, all pixel coordinates within the difference are sequentially edge-checked. Pixels in non-edge areas are filled with the original pixels to fill all generated holes, while pixels in edge areas are not filled with holes. In step 3), the virtual view generated by each virtual viewpoint is formed by fusing the mapping results of two reference views from the left and right sides of the virtual viewpoint. In step 3), the fused virtual view selects adjacent pixels or pixels corresponding to other adjacent reference viewpoints to fill the holes of the current virtual viewpoint. If the pixels corresponding to adjacent reference viewpoints are all holes, the adjacent pixels of the fused image are used to fill them.

2. The method for filling holes in a 3D video based on depth image edge detection according to claim 1, characterized in that: The left, center, and right reference viewpoint images input in step 1) are acquired from a surround camera array. Each camera is distributed on an arc. The left and right reference viewpoints determine the field of view of the naked-eye 3D video, and the resulting multiple virtual viewpoints are within the field of view.

3. The method for filling holes in a 3D video based on depth image edge detection according to claim 1, characterized in that: Step 1) The process of mapping and calculating multiple virtual viewpoints based on the depth information of each reference viewpoint is performed on a row-by-row basis. Therefore, the determination of the distance between pixels before and after is also performed on a row-by-row basis, and hole filling is also performed on a row-by-row basis.

4. The method for filling holes in a 3D video based on depth image edge detection according to claim 1, characterized in that: In step 1), the edge detection algorithm is used to identify the image edges of the depth image at each reference viewpoint. The pixel edge flags of the image edge region are set to one, and the pixel edge flags of the non-edge region are set to zero, thereby distinguishing the edge region and non-edge region of the image.

5. The method for filling holes in a 3D video based on depth image edge detection according to claim 1, characterized in that: In step 2), errors will occur during the calculation of the reference viewpoint pixel coordinates to the virtual viewpoint image pixel coordinates, causing originally adjacent pixels to not be adjacent after mapping, resulting in holes between non-adjacent pixels.

6. The method for filling holes in a 3D video based on depth image edge detection according to claim 1, characterized in that: In step 2), the difference in distance between the image pixel coordinates before and after mapping should not be too large. If the range is too large, most of the real holes will be filled, which will affect the subsequent image fusion.

7. The method for filling holes in a 3D video based on depth image edge detection according to claim 1, characterized in that: The voting fusion method in step 3) can effectively fill large-area real holes and out-of-view holes in 3D video. It introduces hole markers and mapping angles for voting fusion, and the fusion process is expressed by the following formula: (1) in The angle between the generated virtual viewpoint and the intermediate main reference viewpoint. The field of view is defined by the left and right reference viewpoints. When the value is negative, the virtual viewpoint is located between the left reference viewpoint and the central main reference viewpoint. When the value is positive, the virtual viewpoint is located between the central main reference viewpoint and the right reference viewpoint. This represents the fusion result of generating virtual views. , and These represent the mapping results of the middle reference view, left reference view, and right reference view at the virtual viewpoint, respectively. , and These represent the hole flags indicating the mapping results of the middle reference view, left reference view, and right reference view at the virtual viewpoint, respectively.

Images