A real-time hole filling method for high-resolution naked-eye 3D video generation

Abstract

This invention discloses a real-time hole-filling method for high-resolution glasses-free 3D video generation, aiming to solve the hole problem that occurs when real-time mapping of multiple virtual viewpoints using image information from multiple reference viewpoints. This method primarily fills holes using a voting fusion technique. Before fusion, preprocessing is performed using the following steps: improving mapping address errors caused by calculations, and nearest-neighbor interpolation after single-row filtering of the mapping results, to fill holes in individual mapping results one by one. Subsequently, a voting fusion method is used to fuse the complementary mapping results of each virtual viewpoint to effectively fill the holes of each virtual viewpoint. After fusion, during video source synthesis based on glasses-free 3D screen parameters, pixel data from adjacent virtual viewpoints is used to fill the holes of the current virtual viewpoint. This invention, through three filling steps, effectively improves the efficiency of hole filling and outputs high-quality glasses-free 3D video.

Description

Technical Field

[0001] This invention relates to a real-time hole-filling method for generating high-resolution naked-eye 3D videos, belonging to the field of 3D reconstruction technology. Background Technology

[0002] In the process of 3D scene reconstruction, virtual viewpoint generation algorithms can significantly reduce the difficulty of information acquisition and computational complexity. By inputting a small number of reference viewpoint images and combining them with depth information as guidance, virtual views from any viewpoint can be generated within a certain range. After these generated virtual viewpoint views are synthesized, the effect of producing corresponding views according to changes in the observer's viewpoint is achieved, thus successfully reconstructing the 3D scene.

[0003] Holes often appear during the compositing process when generating views from virtual viewpoints. Furthermore, as the field of view increases, information loss and computational errors lead to numerous holes in the generated virtual viewpoint. These holes can be categorized into out-of-view holes and occlusion holes: Out-of-view holes are typically located at the left or right boundaries of the generated virtual view. These holes occur primarily because the imaging range of the virtual viewpoint exceeds the imaging range of the input reference viewpoint, which cannot provide sufficient boundary information to fill the extra boundary areas seen in the virtual viewpoint. Occlusion holes manifest as black holes at the boundary between the foreground and background, and can also appear as large-scale missing foreground structures. These missing structures appear as small cracks in areas with relatively gentle depth changes, and as severe tears in the foreground structure in areas with drastic depth changes. These holes occur primarily because the occlusion relationships of objects change during the generation of the virtual viewpoint. Some objects or structures that are blocked in the input viewpoint are exposed in the generated virtual viewpoint, but the input viewpoint cannot provide sufficient information to fill these exposed areas, thus creating holes.

[0004] To effectively fill the numerous holes generated during virtual viewpoint generation, a real-time hole-filling method for high-resolution glasses-free 3D video generation is proposed. This method takes two viewpoint images with a fixed angle as input, calculates the virtual viewpoint corresponding to the same angle, and fuses these two complementary mapped virtual viewpoint views. Simple hole filling is performed before and after fusion, ultimately outputting a video that conforms to the parameters of a glasses-free 3D screen. Summary of the Invention

[0005] The purpose of this invention is to provide a real-time hole-filling method for generating high-resolution naked-eye 3D videos. In the process of generating naked-eye 3D video sources by inputting RGB images of multiple reference viewpoints and their corresponding depth images, this method uses three steps—filling holes before virtual viewpoint fusion, filling holes after fusion, and filling holes after fusion—to effectively reduce out-of-field holes and occlusion holes in the video source, thereby improving the display quality of naked-eye 3D video sources.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows: A real-time hole-filling method for high-resolution naked-eye 3D video generation, characterized by comprising the following steps:

[0007] Step 1) Filling Holes Before Fusion: Input orthogonal projected RGB images of multiple reference viewpoints and their corresponding depth images. Map multiple virtual viewpoints based on the depth information of each reference viewpoint's depth map to generate virtual views at different viewpoint positions. For holes in the virtual views, preprocessing strategies are used to fill them, including methods such as improving mapping errors and nearest-neighbor interpolation after single-row filtering, to pre-fill the holes.

[0008] Step 2) Fusing and filling holes: For each virtual viewpoint, which consists of two complementary virtual views, the hole-filling results of the two virtual views are fused using a voting fusion method to further fill the hole areas in the virtual viewpoint.

[0009] Step 3) Filling Holes After Fusion: During the naked-eye 3D video generation process, pixels are selected from different virtual viewpoints for compositing based on the parameters of the naked-eye 3D screen. When a hole exists in the virtual viewpoint to which the current pixel belongs, pixels from other virtual viewpoints are used first to fill it.

[0010] As an improvement of the present invention, the reference viewpoint image input in step 1) is acquired by a surround camera array, with each camera arranged along an arc. The number of reference viewpoints is greater than or equal to 2. The left and right reference views are used to determine the field of view of the naked-eye 3D video. The generated multiple virtual viewpoints are located within the field of view covered by the left and right reference views.

[0011] As an improvement of the present invention, in step 1), the process of mapping multiple virtual viewpoints based on the depth information of each reference viewpoint depth map is carried out in rows, and the hole filling process is also completed in rows.

[0012] As an improvement of the present invention, during the virtual viewpoint mapping process in step 1), out-of-view holes and occlusion holes are generated. Before fusion, the occlusion holes are mainly filled. The specific filling method includes: calculating the mapping address of the virtual viewpoint based on the depth information and pixel position of the reference viewpoint, and eliminating holes caused by calculation errors by rounding the mapping address; or removing discrete points in the virtual viewpoint by single-line filtering, and further filling by nearest neighbor interpolation.

[0013] As an improvement of the present invention, in step 2), each virtual viewpoint contains two mapping results, which are respectively derived from the mapping data of the reference viewpoints on the left and right sides of the virtual viewpoint position.

[0014] As an improvement of the present invention, the fusion filling mechanism in step 2) can effectively fill holes. This method employs a voting fusion approach, ensuring that valid information is extracted from each mapping result, and performs voting fusion based on the hole flag and mapping angle. Taking the generation of the left virtual viewpoint as an example, the fusion process can be expressed as the following formula:

[0015]

[0016] Where θ is the optical axis angle between the generated virtual view and the intermediate reference view, θ max is the optical axis angle between the left reference view and the middle reference view, and i represents the position of the current pixel in the virtual view row. V i P represents the fusion result of generating the virtual view from the i-th pixel. M i and P L i F represents the mapping results of the middle reference view and the left reference view of the i-th pixel at the virtual viewpoint. M i and F L i These represent the hole flags of the mapping results of the middle and left reference views of the i-th pixel at the virtual viewpoint, respectively. When a pixel contains valid information in the mapping results of multiple reference viewpoints, auxiliary methods such as depth arbitration can be introduced based on this method to further optimize the voting fusion process.

[0017] As an improvement of the present invention, step 3) uses a single-row filtering method to fill the holes in the fused virtual view. Specifically, a filtering window of size 1×4 is selected, and the hole flag and pixel offset coordinates of the current mapping row are filtered synchronously to ensure effective filling of the hole area and smooth transition of the image.

[0018] As an improvement of the present invention, step 3) allows the encoding and reorganization process to be flexibly adjusted and scheduled according to the viewpoint arrangement rules of each device for different parameters and different types of 3D display devices, so as to adapt to different display needs and optimize video output effects.

[0019] As an improvement of the present invention, step 3) involves selecting an adjacent virtual viewpoint or a reference viewpoint to fill the hole in the current virtual viewpoint. Taking into account the consistency of pixel height between adjacent viewpoints, other virtual viewpoints that are closer to the current virtual viewpoint are preferentially selected.

[0020] Compared with the prior art, the advantages of the present invention are as follows:

[0021] 1. This solution proposes a hole-filling method based on line processing. This method eliminates the need to store the entire frame of image data; it only requires storing a single line of data to complete real-time calculations, thereby improving hole-filling efficiency. This method can meet the real-time hole-filling requirements of naked-eye 3D videos at resolutions of 8K / 30p and even higher frame rates.

[0022] 2. This solution employs a three-step, systematic approach to filling holes, addressing different types of holes and their causes separately. Before fusion, small holes caused by minor occlusions are quickly filled to improve the integrity of the initial image. During fusion, holes outside the field of view and larger holes caused by extensive occlusion are filled, ensuring image continuity and quality during the fusion process. After fusion, effective pixels from other viewpoints are used to fill remaining holes in the current viewpoint, particularly those that cannot be filled in the fusion mapping result, further enhancing image integrity. This phased, systematic processing effectively solves the problem of filling different types of holes, guaranteeing a high-quality final image.

[0023] 3. The single-row filtering method proposed in this solution only requires storing four consecutive pixel data, significantly saving hardware storage space compared to traditional filtering methods. Taking 3×3 median filtering as an example, conventional methods require storing at least two rows of pixel data to output filtered data, leading to a significant increase in hardware storage pressure. This solution, by optimizing storage requirements, effectively reduces hardware resource consumption, providing a superior solution for achieving efficient real-time processing. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the algorithm flow of the present invention;

[0025] Figure 2 This is a schematic diagram showing the arrangement of the left, center, and right reference viewpoints and the virtual viewpoint in Example 1;

[0026] Figure 3This is a schematic diagram illustrating the elimination of voids caused by calculation errors in Example 1;

[0027] Figure 4 This is a schematic diagram of filling holes according to the mapped address in Example 1;

[0028] Figure 5 This is a schematic diagram of a single-row filter in Example 1;

[0029] Figure 6 This is a schematic diagram of virtual viewpoint fusion in Example 1;

[0030] Figure 7 This is a schematic diagram of filling the voids after fusion in Example 1. Detailed Implementation

[0031] The invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, those skilled in the art will find that various equivalent modifications to the invention fall within the scope defined by the appended claims.

[0032] Example 1: See Figures 1-7 A real-time hole-filling method for high-resolution naked-eye 3D video generation, when the input reference viewpoints are three viewpoints (left, center, and right), the method includes the following steps:

[0033] Step 1) Input the orthogonal projected RGB images of three reference viewpoints (left, center, and right) and their corresponding depth images. These reference viewpoint images are acquired using a surround camera array, where the left and right reference viewpoints determine the field of view of the naked-eye 3D video. Multiple virtual viewpoints are mapped based on the depth information of each reference viewpoint. The arrangement of the reference viewpoints and virtual viewpoints is as follows: Figure 2 As shown, the virtual view on the left is obtained by mapping the left reference view and the middle reference view, while the virtual view on the right is obtained by mapping the right reference view and the middle reference view. The field of view of the naked-eye 3D video is 2θ. max .

[0034] The virtual view is mapped by traversing the row pixels of the reference viewpoint and calculating its mapped position in the virtual view based on the position and corresponding depth information of each pixel. Since the calculation result may be a decimal, rounding the result up or down effectively fills in gaps caused by calculation errors. A specific mapping diagram is shown below. Figure 3 As shown.

[0035] During the mapping of row pixels from a reference viewpoint to row pixels from a virtual viewpoint, pixels from adjacent reference viewpoints may be mapped to pixels from non-adjacent virtual viewpoints, such as... Figure 4As shown in the diagram. If the reference viewpoint's pixel is not an edge pixel, then two non-adjacent virtual viewpoint pixels can be used to fill the hole between them.

[0036] After the virtual view mapping is completed, scattered points may appear in the image, affecting the visual effect. Each pixel in the virtual view has a hole flag, which directly determines the validity of the current virtual view pixel in the subsequent fusion process. Using a single-row filtering method to process the hole flags can effectively save storage and computational resources. The specific process is as follows... Figure 5 As shown. In the processing, a 1*4 filter window is selected, with point ③ as the filter center. Assume that the mapped coordinates of each point in the filter window are l1, l2, l3, l4, and the hole flags are f1, f2, f3, f4, respectively. The threshold for the difference in the mapped coordinates is T (T>0). m1, m2, and m3 are the comparison results of the absolute value of the difference between the mapped coordinates of l3 and the other three pixels and T, respectively. If |l3-l1|≤T, then m1 is 1; otherwise, it is 0. The process for m2 and m3 is similar.

[0037] When window ③ satisfies the following formula:

[0038]

[0039] If point ③ is determined to be a scatter point within a non-hole area, then the coordinates of the scatter point are updated using the neighbor values ​​within the window.

[0040] When the following equation is satisfied at point ③ within the window:

[0041]

[0042] Point ③ is determined to be a scattered point within the cavity area. Although its cavity flag is 1, it is actually an invalid value. Its cavity flag needs to be corrected, but its coordinate value does not need to be updated.

[0043] Step 2) Two virtual views correspond to the same virtual viewpoint. The virtual view of the left virtual viewpoint is obtained by mapping the left reference viewpoint and the center reference viewpoint, while the virtual view of the right virtual viewpoint is obtained by mapping the right reference viewpoint and the center reference viewpoint. The two virtual views of the same viewpoint are merged according to a predetermined fusion rule to fill the holes in each virtual viewpoint. Taking the left virtual viewpoint as an example, the specific fusion diagram is as follows: Figure 6 As shown, where V L2M V is a virtual view obtained by mapping the left reference view to the center. M2L V is a virtual view obtained by mapping a central reference view to the left. L This is the result of merging two virtual views.

[0044] The hole-filling fusion method adopts a voting fusion approach. While ensuring the extraction of valid information from each mapping result, voting fusion is performed based on the hole flag and mapping angle. Taking the generation of the left virtual viewpoint as an example, the fusion process can be represented by the following formula:

[0045]

[0046] Where θ is the optical axis angle between the generated virtual view and the intermediate reference view, θ max is the optical axis angle between the left reference view and the middle reference view, and i represents the position of the current pixel in the virtual view row. V i P represents the fusion result of generating the virtual view from the i-th pixel. M i and P L i F represents the mapping results of the middle reference view and the left reference view of the i-th pixel at the virtual viewpoint. M i and F L i These represent the hole flags of the mapping results of the middle and left reference views of the i-th pixel at the virtual viewpoint, respectively. When a pixel contains valid information in the mapping results of multiple reference viewpoints, auxiliary methods such as depth arbitration can be introduced based on this method to further optimize the voting fusion process.

[0047] Step 3) When rearranging the fused virtual viewpoints according to the parameters of the naked-eye 3D screen, if the virtual viewpoint to which the current pixel belongs is empty, pixels at the same position in other viewpoints can be selected to fill it. The filling process is as follows: Figure 7 As shown, V1-Vn represent a row of pixels in the merged virtual viewpoint, and Pixel represents a row of pixels in the rearranged image. During the filling process, pixels from adjacent virtual viewpoints or left, center, and right reference viewpoints are preferentially selected to fill the holes in the current virtual viewpoint. Considering the high consistency of pixels from adjacent viewpoints, pixels from other virtual viewpoints that are as close as possible to the current virtual viewpoint are used for filling.

[0048] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A real-time hole-filling method for high-resolution naked-eye 3D video generation, characterized in that, The following steps are involved: Step 1) Filling holes before fusion: Input multiple orthogonal projection RGB images of reference viewpoints and their corresponding depth images. Map multiple virtual viewpoints according to the depth information of the depth maps of each reference viewpoint to obtain the mapped coordinates and hole markers. Generate virtual views at different viewpoint positions according to the coordinates and markers. For holes in the virtual views, use preprocessing strategies to fill them, including improving the mapping error caused by calculation and using a nearest neighbor interpolation method after single-row filtering of the mapping results to pre-fill the holes. Step 2) Fusing and filling holes: For each virtual viewpoint, which consists of two complementary virtual views, the hole-filling results of the two virtual views are fused using a voting fusion method to further fill the hole areas in the virtual viewpoint. Step 3) Filling holes after fusion: During the naked-eye 3D video generation process, pixels are selected from different virtual viewpoints for fusion according to the parameters of the naked-eye 3D screen. If there are holes in the virtual viewpoint to which the current pixel belongs, pixels from other virtual viewpoints are used to fill them, with priority given to pixels from adjacent virtual viewpoints. In step 1), during the virtual viewpoint mapping process, holes outside the field of view and occlusion holes are generated. Before fusion, the occlusion holes are mainly filled. The specific filling method includes: first, calculating the mapping address of the virtual viewpoint based on the depth information and pixel position of the reference viewpoint, and eliminating holes caused by calculation errors by rounding the mapping address; second, removing discrete points in the virtual viewpoint by single-line filtering, and further filling by nearest neighbor interpolation on this basis. In step 2), each virtual viewpoint contains two mapping results, which are derived from the mapping data of the reference viewpoints on the left and right sides of the virtual viewpoint position, respectively. The mapping data on the left and right sides form a complementary mapping pair, which can effectively eliminate large-area holes.

2. The real-time hole-filling method for high-resolution naked-eye 3D video generation according to claim 1, characterized in that: The reference viewpoint image input in step 1) is acquired by a surround camera array, with each camera arranged along an arc. The number of reference viewpoints is greater than or equal to 2. The left and right reference views are used to determine the field of view of the naked-eye 3D video. The generated multiple virtual viewpoints are located within the field of view covered by the left and right reference views.

3. The real-time hole-filling method for high-resolution naked-eye 3D video generation according to claim 1, characterized in that: Step 1) Mapping multiple virtual viewpoints based on the depth information of each reference viewpoint depth map is performed in real time on a line-by-line basis, and the hole-filling process is also performed in real time on a line-by-line basis.

4. The real-time hole-filling method for high-resolution naked-eye 3D video generation according to claim 1, characterized in that: The fusion filling mechanism in step 2) can effectively fill holes. Using a voting fusion method, while ensuring the extraction of valid information from each mapping result, voting fusion is performed based on the hole flag and mapping angle to generate a left-side virtual viewpoint. The fusion process is expressed by the following formula: (1) in The angle between the optical axis of the generated virtual view and the intermediate reference view. The angle between the optical axes of the left reference view and the middle reference view. Indicates the row position of the pixel in the current virtual view. Indicates the first The fusion result of generating a virtual view from individual pixels. and They represent the first The mapping result of the middle reference view and the left reference view at the virtual viewpoint for each pixel. and They represent the first The hole flags of the mapping results of the middle reference view and the left reference view of each pixel at the virtual viewpoint can be used to further optimize the voting fusion process when a pixel contains valid information in the mapping results of multiple reference viewpoints.

5. The real-time hole-filling method for high-resolution naked-eye 3D video generation according to claim 1, characterized in that: Step 3) uses a single-row filtering method to fill the holes in the fused virtual view. Specifically, a 1×4 filtering window is selected, and the hole flag and pixel offset coordinates of the current mapping row are filtered synchronously to ensure effective filling of the hole area and smooth transition of the image.

6. The real-time hole-filling method for high-resolution naked-eye 3D video generation according to claim 1, characterized in that: In step 3), the encoding and reorganization process is flexibly adjusted and scheduled according to the viewpoint arrangement rules of each device for different parameters and different types of 3D display devices.

7. The real-time hole-filling method for high-resolution naked-eye 3D video generation according to claim 1, characterized in that: In step 3), adjacent virtual viewpoints or reference viewpoints are selected to fill the holes of the current virtual viewpoint. Taking into account the consistency of pixel height of adjacent viewpoints, other virtual viewpoints that are closer to the current virtual viewpoint are preferred.

Images