Three-dimensional video hole filling method based on depth picture edge detection

Abstract

The invention discloses a three-dimensional video hole filling method based on depth picture edge detection, and mainly solves the problem of holes occurring when virtual viewpoints in a field angle range are mapped by using depth information of multiple reference viewpoints. According to the method, before virtual viewpoint image fusion is carried out, edge detection extraction is carried out on a depth image of each fusion image, an image edge area and a non-edge area are judged, and then a filling hole is judged according to the distance between the front and back mapping in the non-edge area. And then, fusing the filled to-be-fused images by adopting a voting fusion mechanism, and fusing complementary mapping results of virtual viewpoints at each angle in the field angle. And after the virtual viewpoint images are fused, filling a cavity of the current virtual viewpoint by utilizing pixel data of the adjacent virtual viewpoints or pixel data of the reference viewpoint. According to the invention, through edge detection processing before image fusion and filling after fusion, the efficiency of hole filling in the virtual view is effectively improved.

Description

Technical Field

[0001] The invention relates to a three-dimensional video hole filling method based on depth picture edge detection, and belongs to the technical field of three-dimensional reconstruction. Background Art

[0002] In the process of 3D scene reconstruction, the use of virtual viewpoint generation algorithms can significantly reduce the difficulty of information acquisition and computational complexity. By inputting a small number of reference viewpoint images and combining depth information as a guide, virtual views of any viewpoint can be generated within a certain range. After these generated virtual viewpoint views are synthesized, the effect of generating corresponding views according to the changes in the observer's viewpoint is achieved, thereby successfully reconstructing the 3D scene.

[0003] When generating the corresponding view through the virtual viewpoint, holes often appear in the process before or after synthesis, and as the field of view increases, the lack of information and calculation errors will cause a large number of holes in the generated virtual viewpoint. The holes can be divided into holes outside the field of view and occlusion holes: holes outside the field of view are usually distributed at the left or right boundary of the generated virtual view. Such holes appear in the virtual viewpoint mainly because the imaging range of the virtual viewpoint exceeds the imaging range of the input reference viewpoint, and the reference viewpoint cannot provide enough boundary information to fill the additional boundary area seen in the virtual viewpoint; occlusion holes appear as black holes at the junction of the foreground and background, and also as large-scale missing foreground structures. These missing appear as small cracks where the depth changes relatively gently, and as severe tearing on the foreground structure where the depth changes drastically. Such holes appear in the virtual viewpoint mainly because the occlusion relationship of the object changes in the process of generating the virtual viewpoint. Some objects or structures that are blocked in the input viewpoint are exposed in the generated virtual viewpoint, but the input viewpoint cannot provide effective information to fill these exposed areas, thus causing holes.

[0004] In order to effectively fill the large number of holes generated in the process of virtual viewpoint generation, a three-dimensional video hole filling method based on depth image edge detection is proposed. This method takes two viewpoint images with a fixed angle as input, calculates the virtual viewpoint corresponding to the same angle, and fuses the two complementary mapped virtual viewpoint views. Image fusion can effectively fill the holes outside the field of view, but the occluded holes generated by the virtual viewpoint generated image are most obvious at the edge of the image. Therefore, the edge detection plus mapping distance judgment method is used to fill the holes before image fusion, which will effectively fill the occluded holes and finally output a video that meets the parameters of the naked eye 3D screen. Summary of the invention

[0005] The purpose of the present invention is to provide a three-dimensional video hole filling method based on depth picture edge detection. In the process of inputting RGB images of three reference viewpoints of left, middle and right and their corresponding depth images to generate a naked-eye 3D video source, the method uses edge detection plus mapping distance judgment to fill holes before virtual viewpoint fusion, fills holes by fusion, and fills holes after fusion, thereby effectively reducing holes outside the field of view and occluded holes in the video source, thereby improving the display quality of the naked-eye 3D video source.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows: A three-dimensional video hole filling method based on depth image edge detection comprises the following steps:

[0007] Step 1) inputting the orthogonal projection RGB images of the three reference viewpoints of left, center and right and their corresponding depth images, mapping multiple virtual viewpoints according to the depth information of the depth map of each reference viewpoint, generating virtual views of different viewpoint positions, and judging the edge area and non-edge area of ​​the reference viewpoint image by performing image edge detection processing on the depth map for the holes in the generated virtual view;

[0008] Step 2) Calculate the distance before and after the image pixel coordinates are mapped, and judge each pixel within a certain range. If these pixels are in the edge area, no hole filling processing is performed. If these pixels are in the non-edge area, all pixels in the area are filled with pixels corresponding to the pixel coordinates before the original image is mapped to avoid holes.

[0009] Step 3) The complementary mapping results at each virtual viewpoint after the holes are filled are fused according to the voting fusion method. When there is a hole in the current pixel in the fused virtual viewpoint image, the pixels in other reference virtual viewpoints are used to fill it.

[0010] As an improvement of the present invention, the left, center, and right reference viewpoint images input in step 1) are obtained from a surround camera array, and each camera is distributed on an arc. The left and right reference viewpoints determine the field of view of the naked-eye three-dimensional video, and the multiple virtual viewpoints generated are within the field of view angle.

[0011] As an improvement of the present invention, the process of mapping and calculating multiple virtual viewpoints according to the depth information of each reference viewpoint in step 1) is performed in units of rows, so the front and back distance judgment of pixel coordinates is also performed in units of rows, and the hole filling is also performed in units of rows.

[0012] As an improvement of the present invention, in the process of virtual viewpoint mapping in step 1), holes outside the field of view and occluded holes are generated, and the image is processed before fusion mainly to fill the occluded holes.

[0013] As an improvement of the present invention, step 1) uses an edge detection algorithm to identify the image edge of the depth map of each reference viewpoint (canny edge detection is recommended), sets the pixel edge flag of the image edge area to one, and sets the pixel edge flag of the non-edge area to zero, so as to distinguish the edge area and non-edge area of ​​the image.

[0014] As an improvement of the present invention, in the step 2), errors may occur in the calculation process of the reference viewpoint pixel coordinates to the virtual viewpoint image pixel coordinates, causing originally adjacent pixels to be non-adjacent after mapping, resulting in holes between non-adjacent pixels.

[0015] As an improvement of the present invention, the step 2) calculates the difference between the current pixel mapping coordinates and the previous pixel mapping coordinates. If the difference is within a preset range, all pixel coordinates in the interpolation are subjected to edge judgment in turn, and all holes generated are filled with original pixels in the pixels in the non-edge area, while the pixels in the edge area are not subjected to hole filling processing.

[0016] As an improvement of the present invention, in the step 2), the certain range of the distance difference before and after the image pixel coordinate mapping cannot be too large. If the range is too large, most of the real holes will be filled, which will affect the subsequent image fusion.

[0017] As an improvement of the present invention, the virtual view generated by each virtual viewpoint in step 3) is formed by merging mapping results of two reference views from the left and right sides of the virtual viewpoint position.

[0018] As an improvement of the present invention, the voting fusion method in step 3) can effectively fill large-area real holes and holes outside the field of view of the three-dimensional video. The method introduces the hole flag bit (i.e., fill the pixel flag position with one and the hole is set to zero) and the mapping angle for voting fusion. The fusion process is expressed as the following formula:

[0019]

[0020] Where α is the angle between the generated virtual viewpoint and the intermediate main reference viewpoint, max is the field of view angle enclosed by the left reference viewpoint and the right reference viewpoint. When α is a negative number, the virtual viewpoint is located between the left reference viewpoint and the middle main reference viewpoint. When α is a positive number, the virtual viewpoint is located between the middle main reference viewpoint and the right reference viewpoint. V represents the fusion result of generating virtual view, P M , P L and P R They represent the mapping results of the middle reference view, the left reference view, and the right reference view at the virtual viewpoint, respectively. M 、F L and F RThe hole flags respectively represent the mapping results of the middle reference view, the left reference view, and the right reference view at the virtual viewpoint.

[0021] As an improvement of the present invention, in the step 3), the fused virtual view selects adjacent pixels or corresponding pixels of other adjacent reference viewpoints to fill the holes of the current virtual viewpoint. It is recommended to use the corresponding pixels of the adjacent reference viewpoints to fill the holes. If the corresponding pixels of the adjacent reference viewpoints are all holes, the adjacent pixels of the fused image are used to fill the holes.

[0022] Compared with the prior art, the advantage of this solution is that, under normal circumstances, in the virtual view generated by fusion after mapping the reference view, positions with large depth differences are often more likely to have holes, and these positions with large depth differences are generally the edge positions of the image. Therefore, the present invention performs edge detection processing on the original image before fusion of the virtual viewpoint image, and focuses on pre-filling the holes at the positions where holes are likely to occur in the edge area, which greatly reduces the occurrence of holes in the virtual view generated after fusion. Finally, the filling after fusion is added to more effectively improve the efficiency of hole filling in the virtual view. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the void filling process of the present invention;

[0024] Figure 2 Schematic diagram of arrangement of left, center and right reference viewpoints and virtual viewpoints in Example 1;

[0025] Figure 3 A schematic diagram of the void caused by the mapping calculation and calculation error in Example 1;

[0026] Figure 4 Schematic diagram of mapping pixel position subtraction interval in Example 1;

[0027] Figure 5 A schematic diagram of determining and filling holes according to edge areas in Example 1;

[0028] Figure 6 This is a schematic diagram of virtual viewpoint fusion within the field of view in Example 1;

[0029] Figure 7 Schematic diagram of filling holes after virtual view fusion in Example 1. DETAILED DESCRIPTION

[0030] The invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the invention and are not used to limit the scope of the invention. After reading the invention, various equivalent modifications of the invention by those skilled in the art fall within the scope defined by the claims attached to this application.

[0031] Example 1: See Figure 1-Figure 7 , a three-dimensional video hole filling method based on depth image edge detection, the method comprising the following steps:

[0032] Step 1) Input the orthogonal projection RGB images of the left, center, and right reference viewpoints and their corresponding depth images. These reference viewpoint images are acquired through a surround camera array, where the left and right reference viewpoints determine the field of view of the naked eye 3D video. Based on the depth information of each reference viewpoint, multiple virtual viewpoints within the field of view are mapped. The arrangement of the reference viewpoints and the virtual viewpoints is as follows: Figure 2 As shown, the virtual view on the left is obtained by mapping the left reference view and the middle reference view, while the virtual view on the right is obtained by mapping the right reference view and the middle reference view. The field of view of the naked eye 3D video is α max .

[0033] The mapping method of the virtual view is to traverse the row pixels of the reference viewpoint and calculate the mapping position of each pixel in the virtual view according to its position and corresponding depth information. Since the calculation result may be a decimal, the calculation result is rounded off. The calculation rounding operation may make the mapped coordinates discontinuous, resulting in holes. The specific mapping diagram and hole generation are shown in the figure. Figure 3 shown.

[0034] Perform edge detection on the depth map of each reference viewpoint to determine the edge area and non-edge area of ​​the reference view. Subtract the calculated mapping position in the virtual view from the last calculated mapping position to obtain a mapping pixel position subtraction interval, such as Figure 4 shown.

[0035] If this interval is within the set range (usually between 1-4) and is in the edge area, no hole filling is performed; if this interval is within the set range and is not in the edge area, the hole flag bit of this mapping position interval is set to 1, and the pixel data is the pixel data of the reference view before the mapping calculation, thereby effectively filling the hole. Figure 5 shown.

[0036] Step 2) The same virtual viewpoint corresponds to two virtual views, where the virtual view of the left virtual viewpoint is mapped by the left reference viewpoint and the middle reference viewpoint, and the virtual view of the right virtual viewpoint is mapped by the right reference viewpoint and the middle reference viewpoint. The two virtual views of the same viewpoint are fused according to the predetermined fusion rules. The fusion to fill the holes adopts the voting fusion method. Under the premise of ensuring that valid information in each mapping result can be extracted, voting fusion is performed based on the hole flag and the mapping angle to fill the holes in each virtual viewpoint. Taking the left virtual viewpoint as an example, the specific fusion block diagram is as follows: Figure 6 As shown, VL2M V is the virtual view obtained by mapping the left reference view to the center. M2L is the virtual view obtained by mapping the reference view to the left, V L It is the result of fusion of two virtual views.

[0037] Step 3) When rearranging the fused virtual viewpoints according to the parameters of the naked eye 3D screen, if the virtual viewpoint to which the current pixel belongs is empty, pixels at the same position in other viewpoints can be selected for filling. The filling process is as follows: Figure 7 As shown, taking the left virtual view Vn as an example, in the filling process, pixels of adjacent virtual viewpoints or left reference viewpoints or main reference viewpoints are preferentially selected to fill the holes of the current virtual viewpoint. Considering the high consistency of pixels of adjacent viewpoints, pixels of other virtual viewpoints close to the current virtual viewpoint are preferentially selected for filling as much as possible.

[0038] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A three-dimensional video hole filling method based on depth image edge detection, characterized in that: The following steps are involved: Step 1) inputting the orthogonal projection RGB images of the three reference viewpoints of left, center and right and their corresponding depth images, mapping multiple virtual viewpoints according to the depth information of the depth map of each reference viewpoint, generating virtual views of different viewpoint positions, and judging the edge area and non-edge area of ​​the reference viewpoint image by performing image edge detection processing on the depth map for the holes in the generated virtual view; Step 2) Calculate the distance before and after the image pixel coordinates are mapped, and judge each pixel within a certain range. If these pixels are in the edge area, no hole filling processing is performed. If these pixels are in the non-edge area, all pixels in the area are filled with pixels corresponding to the pixel coordinates before the original image is mapped to avoid holes. Step 3) The complementary mapping results at each virtual viewpoint after the holes are filled are fused according to the voting fusion method. When there is a hole in the current pixel in the fused virtual viewpoint image, the pixels in other reference virtual viewpoints are used to fill it.

2. The method for filling a 3D video hole based on depth image edge detection according to claim 1, characterized in that: The left, center, and right reference viewpoint images input in step 1) are obtained from a surround camera array, and each camera is distributed on an arc. The left and right reference viewpoints determine the field of view of the naked eye 3D video, and the multiple virtual viewpoints generated are within the field of view.

3. The method for filling a 3D video hole based on depth image edge detection according to claim 1, characterized in that: The process of mapping and calculating multiple virtual viewpoints according to the depth information of each reference viewpoint in step 1) is performed in line units, so the front and back distance of pixel coordinates is also judged in line units, and the hole filling is also performed in line units.

4. The method for filling a 3D video hole based on depth image edge detection according to claim 1, characterized in that: In the process of virtual viewpoint mapping in step 1), holes outside the field of view and occlusion holes are generated. The image processing before fusion is mainly to fill the occlusion holes.

5. The method for filling a 3D video hole based on depth image edge detection according to claim 1, characterized in that: In the step 1), an edge detection algorithm is used to identify the image edge of the depth map of each reference viewpoint (canny edge detection is recommended), and the pixel edge flag of the image edge area is set to one, and the pixel edge flag of the non-edge area is set to zero, so as to distinguish the edge area and the non-edge area of ​​the image.

6. The method for filling a 3D video hole based on depth image edge detection according to claim 1, characterized in that: In the step 2), errors may occur in the calculation process of the reference viewpoint pixel coordinates to the virtual viewpoint image pixel coordinates, causing originally adjacent pixels to become non-adjacent after mapping, resulting in holes between non-adjacent pixels.

7. The method for filling a 3D video hole based on depth image edge detection according to claim 1, characterized in that: The step 2) calculates the difference between the current pixel mapping coordinates and the previous pixel mapping coordinates. If the difference is within a preset range, all pixel coordinates in the interpolation are subjected to edge judgment in turn. All holes generated are filled with original pixels in the pixels in the non-edge area, and no hole filling processing is performed on the pixels in the edge area.

8. The method for filling a 3D video hole based on depth image edge detection according to claim 1, characterized in that: In the step 2), the certain range of the distance difference before and after the image pixel coordinate mapping cannot be too large. If the range is too large, most of the real holes will be filled, which will affect the subsequent image fusion.

9. The method for filling a 3D video hole based on depth image edge detection according to claim 1, characterized in that: The virtual view generated by each virtual viewpoint in step 3) is formed by fusion of mapping results of two reference views on the left and right sides of the virtual viewpoint position; the fused virtual view in step 3) selects adjacent pixels or corresponding pixels of other adjacent reference viewpoints to fill the holes of the current virtual viewpoint, and the corresponding pixels of the adjacent reference viewpoints are preferentially used for filling. If the corresponding pixels of the adjacent reference viewpoints are all holes, the adjacent pixels of the fused image are used for filling.

10. The method for filling a 3D video hole based on depth image edge detection according to claim 1, characterized in that: The voting fusion method in step 3) can effectively fill large-area real holes and holes outside the field of view of the three-dimensional video. The method introduces a hole flag, that is, fills the pixel flag position with one, sets the hole to zero, and performs voting fusion with the mapping angle. The fusion process is expressed as the following formula: Where α is the angle between the generated virtual viewpoint and the intermediate main reference viewpoint, max is the field of view angle enclosed by the left reference viewpoint and the right reference viewpoint. When α is a negative number, the virtual viewpoint is located between the left reference viewpoint and the middle main reference viewpoint. When α is a positive number, the virtual viewpoint is located between the middle main reference viewpoint and the right reference viewpoint. V represents the fusion result of generating virtual view, P M , P L and P R They represent the mapping results of the middle reference view, the left reference view, and the right reference view at the virtual viewpoint, respectively. M 、F L and F R The hole flags respectively represent the mapping results of the middle reference view, the left reference view, and the right reference view at the virtual viewpoint.

Images