Real-time hole filling method in high-resolution naked-eye 3D video generation

Abstract

The invention discloses a real-time hole filling method in high-resolution naked-eye 3D video generation, and aims to solve the problem of holes occurring when a plurality of virtual viewpoints are mapped in real time by using image information of a plurality of reference viewpoints. According to the method, holes are mainly filled through a voting fusion technology, and before fusion, preprocessing is performed by adopting the following steps of improving mapping address errors caused by calculation, performing neighbor interpolation after mapping result single-line filtering and the like, and filling the holes in single mapping results one by one. And then, fusing the complementary mapping results of the virtual viewpoints by adopting a voting fusion method so as to effectively fill holes of the virtual viewpoints. And after fusion, filling a cavity of the current virtual viewpoint by using pixel data of adjacent virtual viewpoints in the process of synthesizing the video source according to the naked-eye 3D screen parameters. Through the three filling steps, the hole filling efficiency is effectively improved, and the high-quality naked-eye 3D video is output.

Description

Technical Field

[0001] The invention relates to a real-time hole filling method in generating high-resolution naked-eye 3D video, belonging to the technical field of three-dimensional reconstruction. Background Art

[0002] In the process of 3D scene reconstruction, the use of virtual viewpoint generation algorithms can significantly reduce the difficulty of information acquisition and computational complexity. By inputting a small number of reference viewpoint images and combining depth information as a guide, virtual views of any viewpoint can be generated within a certain range. After these generated virtual viewpoint views are synthesized, the effect of generating corresponding views according to the changes in the observer's viewpoint is achieved, thereby successfully reconstructing the 3D scene.

[0003] When generating the corresponding view through the virtual viewpoint, holes often appear in the synthesis process. As the field of view increases, the lack of information and calculation errors will cause a large number of holes in the generated virtual viewpoint. The holes can be divided into holes outside the field of view and occlusion holes: holes outside the field of view are usually distributed at the left or right border of the generated virtual view. Such holes appear in the virtual viewpoint mainly because the imaging range of the virtual viewpoint exceeds the imaging range of the input reference viewpoint, and the reference viewpoint cannot provide enough boundary information to fill the additional boundary area seen in the virtual viewpoint; occlusion holes appear as black holes at the junction of the foreground and background, and also as large-scale missing foreground structures. These missing appear as small cracks where the depth changes relatively slowly, and as severe tearing on the foreground structure where the depth changes drastically. Such holes appear in the virtual viewpoint mainly because the occlusion relationship of the object changes in the process of generating the virtual viewpoint. Some objects or structures that are blocked in the input viewpoint are exposed in the generated virtual viewpoint, but the input viewpoint cannot provide effective information to fill these exposed areas, thus causing holes.

[0004] In order to effectively fill the large number of holes generated in the process of virtual viewpoint generation, a real-time hole filling method in high-resolution naked-eye 3D video generation is proposed. This method takes two viewpoint images with a fixed angle as input, calculates the virtual viewpoint corresponding to the same angle, and fuses the two complementary virtual viewpoint views. Before and after the fusion, simple hole filling is performed, and finally the video that meets the naked-eye 3D screen parameters is output. Summary of the invention

[0005] The purpose of the present invention is to provide a real-time hole filling method in high-resolution naked-eye 3D video generation. In the process of inputting RGB images of multiple reference viewpoints and their corresponding depth images to generate a naked-eye 3D video source, the method uses three steps: filling holes before virtual viewpoint fusion, filling holes by fusion, and filling holes after fusion, to effectively reduce holes outside the field of view and occluded holes in the video source, thereby improving the display quality of the naked-eye 3D video source.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows: a real-time hole filling method in high-resolution naked-eye 3D video generation, characterized in that it comprises the following steps:

[0007] Step 1) Fill holes before fusion: Input orthogonal projection RGB images of multiple reference viewpoints and their corresponding depth images, map multiple virtual viewpoints according to the depth information of the depth map of each reference viewpoint, and generate virtual views of different viewpoint positions. For the holes in the virtual view, a preprocessing strategy is used to fill them, including improving mapping errors, single-line filtering and neighbor interpolation, etc., to pre-fill the holes;

[0008] Step 2) Fusion filling holes: For each virtual viewpoint, which consists of two complementary virtual views, the hole filling results of the two virtual views are fused using the voting fusion method to further fill the hole area in the virtual viewpoint;

[0009] Step 3) Filling holes after fusion: In the process of generating naked-eye 3D video, pixels are selected from different virtual viewpoints for synthesis according to the parameters of the naked-eye 3D screen. When there is a hole in the virtual viewpoint to which the current pixel belongs, pixels from other virtual viewpoints are preferentially used for filling.

[0010] As an improvement of the present invention, the reference viewpoint image input in step 1) is acquired by a surround camera array, each camera is arranged along an arc, the number of reference viewpoints is greater than or equal to 2, the left and right reference views are used to determine the field of view of the naked eye 3D video, and the multiple virtual viewpoints generated are located within the field of view angle covered by the left and right reference views.

[0011] As an improvement of the present invention, in step 1), the process of mapping multiple virtual viewpoints according to the depth information of each reference viewpoint depth map is performed in line units, and the hole filling process is also completed in line units.

[0012] As an improvement of the present invention, the virtual viewpoint mapping process in step 1) will generate holes outside the field of view and occluded holes, and the occluded holes are mainly filled before fusion. The specific filling method includes: calculating the mapping address of the virtual viewpoint according to the depth information and pixel position of the reference viewpoint, and eliminating the holes caused by calculation errors by rounding the mapping address up and down; or removing discrete points in the virtual viewpoint by single-line filtering, and further filling them using neighbor interpolation on this basis.

[0013] As an improvement of the present invention, in step 2), each virtual viewpoint includes two mapping results, which are respectively derived from mapping data of reference viewpoints on the left and right sides of the virtual viewpoint position.

[0014] As an improvement of the present invention, the fusion filling mechanism in step 2) can effectively fill the holes. The method adopts a voting fusion method, and performs voting fusion based on the hole flag and the mapping angle while ensuring the extraction of valid information in each mapping result. Taking the generation of the left virtual viewpoint as an example, the fusion process can be expressed as the following formula:

[0015]

[0016] Where θ is the optical axis angle between the generated virtual view and the intermediate reference view, θ max is the optical axis angle between the left reference view and the middle reference view, and i represents the position of the current pixel in the virtual view row. V i represents the fusion result of the virtual view generated by the i-th pixel, P M i and P L i denote the mapping results of the middle reference view and the left reference view of the i-th pixel at the virtual viewpoint, respectively, and F M i and F L i The hole flags of the mapping results of the middle reference view and the left reference view of the i-th pixel at the virtual viewpoint are respectively represented. When a pixel contains valid information in the mapping results of multiple reference viewpoints, auxiliary methods such as depth arbitration can be introduced based on this method to further optimize the voting fusion process.

[0017] As an improvement of the present invention, the step 3) uses a single-line filtering method to fill the holes in the fused virtual view. Specifically, a filter window with a size of 1×4 is selected to filter the hole flag and pixel offset coordinates of the current mapping row synchronously to ensure effective filling of the hole area and smooth transition of the image.

[0018] As an improvement of the present invention, in step 3), for three-dimensional display devices with different parameters and different types, the encoding and reorganization process can be flexibly adjusted and scheduled according to the viewpoint arrangement rules of each device to adapt to different display requirements and optimize the video output effect.

[0019] As an improvement of the present invention, the step 3) selects an adjacent virtual viewpoint or a reference viewpoint to fill the hole of the current virtual viewpoint. Based on the high consistency of the pixels of the adjacent viewpoints, other virtual viewpoints that are closer to the current virtual viewpoint are preferably selected.

[0020] Compared with the prior art, the advantages of the present invention are as follows:

[0021] 1. This solution proposes a hole filling method based on line processing. This method does not need to store the entire frame of image data, but only needs to store a single line of data to complete real-time calculation, thereby improving the efficiency of hole filling. This method can meet the real-time hole filling requirements of naked-eye 3D videos with 8K / 30p or even higher resolutions and frame rates.

[0022] 2. This solution fills holes in an orderly manner through three steps, and handles different types of holes and their causes separately. Before fusion, some small holes caused by small-area occlusions are quickly filled to improve the integrity of the initial picture. The filling in the fusion stage mainly fills holes generated outside the field of view and larger holes caused by large-area occlusions to ensure image continuity and quality during the fusion process. After fusion, the filling uses effective pixels from other viewpoints to complement the remaining holes that still exist in the current viewpoint, especially for holes that cannot be filled in the fusion mapping results, to further improve the integrity of the image. This staged and orderly processing method effectively solves the problem of filling different holes and ensures the high-quality presentation of the final image.

[0023] 3. The single-line filtering method proposed in this solution only needs to store four consecutive pixel data, which significantly saves hardware storage space compared to traditional filtering methods. Taking 3×3 median filtering as an example, the conventional method needs to store at least two lines of pixel data to output one filtered data, resulting in a significant increase in hardware storage pressure. This solution effectively reduces the occupation of hardware resources by optimizing storage requirements, providing a better solution for achieving efficient real-time processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the algorithm flow of the present invention;

[0025] Figure 2 Schematic diagram of arrangement of left, center and right reference viewpoints and virtual viewpoints in Example 1;

[0026] Figure 3A schematic diagram of eliminating voids caused by calculation errors in Example 1;

[0027] Figure 4 A schematic diagram of filling holes according to mapping addresses in Example 1;

[0028] Figure 5 This is a schematic diagram of single-line filtering in Example 1;

[0029] Figure 6 This is a schematic diagram of virtual viewpoint fusion in Example 1;

[0030] Figure 7 This is a schematic diagram of filling the cavity after fusion in Example 1. DETAILED DESCRIPTION

[0031] The invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the invention and are not used to limit the scope of the invention. After reading the invention, various equivalent modifications of the invention by those skilled in the art fall within the scope defined by the claims attached to this application.

[0032] Example 1: See Figure 1-Figure 7 A real-time hole filling method in high-resolution naked-eye 3D video generation, when the input reference viewpoints are three viewpoints of left, middle and right, the method comprises the following steps:

[0033] Step 1) Input the orthogonal projection RGB images of the left, center, and right reference viewpoints and their corresponding depth images. These reference viewpoint images are acquired through a surround camera array, where the left and right reference viewpoints determine the field of view of the naked eye 3D video. Based on the depth information of each reference viewpoint, multiple virtual viewpoints are mapped. The arrangement of the reference viewpoints and the virtual viewpoints is as follows: Figure 2 As shown, the virtual view on the left is obtained by mapping the left reference view and the middle reference view, while the virtual view on the right is obtained by mapping the right reference view and the middle reference view. The field of view of naked eye 3D video is 2θ max .

[0034] The mapping method of the virtual view is to traverse the row pixels of the reference viewpoint and calculate the mapping position of each pixel in the virtual view according to its position and corresponding depth information. Since the calculation result may be a decimal, the holes caused by calculation errors can be effectively filled by rounding the calculation result up or down. The specific mapping diagram is as follows Figure 3 shown.

[0035] In the process of mapping row pixels of the reference viewpoint to row pixels of the virtual viewpoint, it may happen that pixels of adjacent reference viewpoints are mapped to non-adjacent virtual viewpoint pixels, such as Figure 4If the pixel of the reference viewpoint is not an edge pixel, two non-adjacent virtual viewpoint pixels can be used to fill the hole between them.

[0036] After the virtual view mapping is completed, scattered points that affect the visual effect may appear in the image. Each pixel of the virtual view has a hole flag. In the subsequent fusion process, these hole flags directly determine the validity of the current virtual view pixel. By using the single-line filtering method to process the hole flag, storage and computing resources can be effectively saved. The specific process is as follows Figure 5 As shown. In the processing, a 1*4 filter window is selected, and point ③ is the filter center. Assume that the mapping coordinates of each point in the filter window are l1, l2, l3, l4, and the hole flags are f1, f2, f3, f4. The difference threshold of the mapping coordinates is T (T>0). m1, m2, and m3 are the comparison results of the absolute value of the difference between the mapping coordinates of l3 and the other three pixels and T. If |l3-l1|≤T, m1 takes 1, otherwise it takes 0. The process of m2 and m3 is similar.

[0037] When the window ③ satisfies the following formula:

[0038]

[0039] It is determined that point ③ is a scattered point in the non-hole area. At this time, the coordinates of the scattered points are updated using the neighboring values ​​in the window.

[0040] When the following equation is satisfied at point ③ in the window:

[0041]

[0042] It is determined that point ③ is a scattered point in the hole area. Although its hole flag is 1, it is actually an invalid value. Its hole flag needs to be corrected, and its coordinate value does not need to be updated.

[0043] Step 2) The same virtual viewpoint corresponds to two virtual views, where the virtual view of the left virtual viewpoint is mapped from the left reference viewpoint and the middle reference viewpoint, and the virtual view of the right virtual viewpoint is mapped from the right reference viewpoint and the middle reference viewpoint. The two virtual views of the same viewpoint are fused according to a predetermined fusion rule to fill the holes in each virtual viewpoint. Taking the left virtual viewpoint as an example, the specific fusion block diagram is as follows: Figure 6 As shown, V L2M V is the virtual view obtained by mapping the left reference view to the center. M2L is the virtual view obtained by mapping the reference view to the left, V L It is the result of fusion of two virtual views.

[0044] The fusion filling method adopts voting fusion. Under the premise of ensuring that effective information in each mapping result can be extracted, voting fusion is performed based on the hole flag and mapping angle. Taking the generation of the left virtual viewpoint as an example, the fusion process can be represented as shown in the following formula:

[0045]

[0046] Where θ is the optical axis angle between the generated virtual view and the intermediate reference view, θ max is the optical axis angle between the left reference view and the middle reference view, and i represents the position of the current pixel in the virtual view row. V i represents the fusion result of the virtual view generated by the i-th pixel, P M i and P L i denote the mapping results of the middle reference view and the left reference view of the i-th pixel at the virtual viewpoint, respectively, and F M i and F L i The hole flags of the mapping results of the middle reference view and the left reference view of the i-th pixel at the virtual viewpoint are respectively represented. When a pixel contains valid information in the mapping results of multiple reference viewpoints, auxiliary methods such as depth arbitration can be introduced based on this method to further optimize the voting fusion process.

[0047] Step 3) When rearranging the fused virtual viewpoints according to the parameters of the naked eye 3D screen, if the virtual viewpoint to which the current pixel belongs is empty, pixels at the same position in other viewpoints can be selected for filling. The filling process is as follows: Figure 7 As shown in the figure, V1-Vn are rows of pixels of the fused virtual viewpoints, and Pixel is a row of pixels in the rearranged image. In the filling process, pixels of adjacent virtual viewpoints or left, middle, and right reference viewpoints are preferentially selected to fill the holes of the current virtual viewpoint. Considering the high consistency of pixels of adjacent viewpoints, pixels of other virtual viewpoints close to the current virtual viewpoint are preferentially selected for filling as much as possible.

[0048] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A real-time hole filling method in high-resolution naked-eye 3D video generation, characterized in that: The following steps are involved: Step 1) Filling holes before fusion: Input multiple orthogonal projection RGB images of reference viewpoints and their corresponding depth images, map multiple virtual viewpoints according to the depth information of the depth map of each reference viewpoint, obtain mapping coordinates and hole flags, generate virtual views of different viewpoint positions according to the coordinates and flags, and use preprocessing strategies to fill holes in virtual views, including improving the mapping error caused by calculation, and using the nearest neighbor interpolation method after single-line filtering of the mapping results to pre-fill holes; Step 2) Fusion filling holes: For each virtual viewpoint, which consists of two complementary virtual views, the hole filling results of the two virtual views are fused using the voting fusion method to further fill the hole area in the virtual viewpoint; Step 3) Filling holes after fusion: In the process of generating naked-eye 3D video, pixels are selected from different virtual viewpoints for synthesis according to the parameters of the naked-eye 3D screen. When there are holes in the virtual viewpoint to which the current pixel belongs, pixels in other virtual viewpoints are used to fill them, with pixels in adjacent virtual viewpoints being given priority.

2. The real-time hole filling method in high-resolution naked-eye 3D video generation according to claim 1, characterized in that: The reference viewpoint image input in step 1) is obtained by a surround camera array, and each camera is arranged along an arc. The number of reference viewpoints is greater than or equal to 2. The left and right reference views are used to determine the field of view of the naked eye 3D video, and the multiple virtual viewpoints generated are located within the field of view angle covered by the left and right reference views.

3. The real-time hole filling method in high-resolution naked-eye 3D video generation according to claim 1, characterized in that: The process of step 1) mapping multiple virtual viewpoints according to the depth information of each reference viewpoint depth map is performed in real time in line units, and the hole filling process is also performed in real time in line units.

4. The real-time hole filling method in high-resolution naked-eye 3D video generation according to claim 1, characterized in that: In the process of virtual viewpoint mapping in step 1), holes outside the field of view and occluded holes are generated. Before fusion, the occluded holes are mainly filled. The specific filling method includes: first, the mapping address of the virtual viewpoint is calculated according to the depth information and pixel position of the reference viewpoint, and the holes caused by calculation errors are eliminated by rounding the mapping address up and down; secondly, the discrete points in the virtual viewpoint are removed by single-line filtering, and on this basis, the nearest neighbor interpolation is used for further filling.

5. The real-time hole filling method in high-resolution naked-eye 3D video generation according to claim 1, characterized in that: In step 2), each virtual viewpoint includes two mapping results, which are respectively derived from the mapping data of the reference viewpoints on the left and right sides of the virtual viewpoint position. A complementary mapping pair is formed based on the mapping data on the left and right sides, which can effectively eliminate large-area holes.

6. The real-time hole filling method in high-resolution naked-eye 3D video generation according to claim 1, characterized in that: The fusion filling mechanism in step 2) can effectively fill the holes. Voting fusion is adopted to ensure that valid information is extracted from each mapping result. Voting fusion is performed based on the hole flag and mapping angle to generate a virtual viewpoint on the left. The fusion process is expressed as the following formula: Where θ is the optical axis angle between the generated virtual view and the intermediate reference view, θ max is the optical axis angle between the left reference view and the middle reference view, i represents the row position of the pixel in the current virtual view, and P V i represents the fusion result of the virtual view generated by the i-th pixel, P M i and P L i denote the mapping results of the middle reference view and the left reference view of the i-th pixel at the virtual viewpoint, respectively, and F M i and F L i The hole flags respectively represent the mapping results of the middle reference view and the left reference view of the i-th pixel at the virtual viewpoint. When a pixel contains valid information in the mapping results of multiple reference viewpoints, auxiliary methods such as depth arbitration can be introduced based on this method to further optimize the voting fusion process.

7. The real-time hole filling method in high-resolution naked-eye 3D video generation according to claim 1, characterized in that: The step 3) uses a single-row filtering method to fill the holes in the fused virtual view. Specifically, a filtering window with a size of 1×4 is selected to filter the hole flag and pixel offset coordinates of the current mapping row synchronously to ensure effective filling of the hole area and smooth transition of the image.

8. The real-time hole filling method in high-resolution naked-eye 3D video generation according to claim 1, characterized in that: In step 3), for three-dimensional display devices of different parameters and types, the encoding and reorganization process is flexibly adjusted and scheduled according to the viewpoint arrangement rules of each device to adapt to different display requirements and optimize the video output effect.

9. The real-time hole filling method in high-resolution naked-eye 3D video generation according to claim 1, characterized in that: The step 3) selects adjacent virtual viewpoints or reference viewpoints to fill the holes of the current virtual viewpoint. Based on the high consistency of the pixels of the adjacent viewpoints, other virtual viewpoints that are closer to the current virtual viewpoint are preferably selected.

Images