A method for realizing image format of high-resolution ultra-wide field of view three-dimensional display

By combining multiple sub-pictures and using a three-dimensional scene completion algorithm, combining projective geometry and hole filling algorithm, the problem of three-dimensional image hole filling at high resolution ultra-wide field of view angle is solved, achieving high-quality image recovery and expanding the observation range Effect.

CN114067050BActive Publication Date: 2024-11-08SOUTHEAST UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111346583.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-15
Publication Date
2024-11-08
Estimated Expiration
2041-11-15

AI Technical Summary

Technical Problem

The prior art is difficult to effectively fill the voids in three-dimensional images at high resolution ultra-wide field of view angles, especially when viewed at different perspectives, and the filling quality is poor, resulting in recovery errors visible to the naked eye.

Method used

Using a new image format, by obtaining three color sub-maps and one depth sub-maps of the target scene, combining them into an input array, and using a three-dimensional scene completion algorithm, combining projective geometry and hole filling algorithm, to achieve High resolution scene hollow fill in any direction of observation.

Benefits of technology

Effectively complete the image holes caused by viewing angle transformation, generate high-quality trusted image pixels, expand the observation range, and achieve fast and efficient real-time calculation and observation on the presentation device.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114067050B_ABST
    Figure CN114067050B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for realizing an image format of high-resolution ultra-wide field-of-view three-dimensional display, comprising the following steps: 1) acquiring four information images of a target scene (a pair of color images located at the maximum left and right deflection angles are left and right side views, a color image that does not overlap with them within the angle is a main view, and a depth map of the main view of the scene); 2) combining the three acquired color images and a depth map into an image information array and inputting it into an imaging device; 3) through an algorithm, the color information of the left and right deflection angle side views can be relied on to complete the three-dimensional scene holes caused by the conversion of viewing angles in the scene; the three-dimensional hole filling algorithm based on the image format operation can realize high-quality wide-field-of-view three-dimensional scene hole filling on the imaging device in a high-resolution scene.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to a method for realizing a high-resolution, ultra-wide field-of-view three-dimensional display image format, belonging to the field of three-dimensional image reconstruction technology. Background Art

[0002] Visual information is the primary means by which humans perceive the external world, providing over 70% of the information we receive about the real world. However, the visual information we typically use is two-dimensional (2D), while real-world scenes are three-dimensional. Mapping spatial information to a 2D plane inevitably results in information loss. Restoring and reconstructing true 3D information is a task of practical significance. One of the difficulties in 3D scene reconstruction is compensating for occlusion. The occlusion caused by objects at different depths in space creates gaps in the 3D information that are difficult to recover during acquisition and observation. Two common methods are used to capture and reconstruct 3D scene information: one is using an RGB-D camera to capture both color and depth information; the other is using a binocular camera to simultaneously acquire color information from two directions, then using reconstruction algorithms to obtain the desired 3D relationships. Neither of these methods effectively addresses the gaps caused by occlusion. Even with high resolution and a large field of view, the reconstructed 3D scene exhibits visible errors, and the filling of these gaps is unconvincing when viewed from different perspectives. To effectively improve the filling quality of holes generated when viewing 3D images from different perspectives at high resolution and wide field of view, a new image transmission format is proposed. This transmission format consists of three color images from different angles and depth information from the main view angle, which can achieve high-resolution ultra-wide field of view 3D hole filling. Summary of the Invention

[0003] Technical Problem: The purpose of this invention is to provide a method for achieving high-resolution, ultra-wide field-of-view three-dimensional hole filling in an image format. This image format acquires three color sub-images and one depth sub-image of the target scene, and combines these four sub-images into a new image for storage and transmission. The transmitted new format image, by combining projective geometry and hole filling algorithms, can fill holes in high-resolution scenes in any observation direction within a wide field of view, thereby expanding the observation range while ensuring the quality of the filled scene.

[0004] Technical Solution: To achieve the above objectives, the present invention provides a method for realizing a high-resolution, ultra-wide field-of-view three-dimensional display image format, comprising the following steps:

[0005] Step 1. Acquire target scene image information. The acquired target scene image information includes two color images of the scene taken by the camera with the maximum imaging angle as the main optical axis, as left and right side view sub-images, and a depth image captured at any angle between the maximum angles to restore the scene, as a depth sub-image.

[0006] Step 2. Reassemble the acquired sub-images into an input array. Each part of this input array is a complete representation of the original sub-image. Input the reassembled array into the imaging device.

[0007] Step 3. Reconstruct the input array using a 3D scene completion algorithm and complete the image holes caused by the change of viewpoint based on the image format.

[0008] in:

[0009] In step 1, image information of the target scene is obtained. In addition to the left and right side view sub-images and the depth sub-image of the scene, more color sub-images and depth sub-images of the target scene are added to the input array as supplementary information to form an image format that achieves high-resolution ultra-wide field of view three-dimensional display.

[0010] In step 2, several sub-images are recombined into an input array, arranged in the form of... The input can be in the form of a square matrix, or in the form of a row vector such as [ABCD] or [ABCD]. T The sub-images are input to the imaging device in column vector form, where the order of the sub-images is arbitrary.

[0011] In step 2, the several sub-images are recombined into an input array by arranging the input sub-images sequentially according to their pixel positions, so that the sub-images are interleaved and form an overall input array.

[0012] The 3D scene completion algorithm in step 3, which uses the input array from step 2 to complete the holes in the 3D scene, can be broken down into two parts: a. pixel restoration of the view at the viewpoint position; b. completion of holes in the image at any observation viewpoint position by the side view sub-image.

[0013] The 3D scene completion algorithm in step 3 enables arbitrary viewpoint changes within the maximum observation angle. For image restoration and hole filling under arbitrary observation angles, the depth information in the depth sub-map indexes pixels at the same spatial position in the two side views as filling pixels in the hole portion. The calculation formula can be written as:

[0014] P V =(1-K)P V1 +KP V2 (1)

[0015] in This represents image information at any observation angle, hereinafter referred to as viewpoint information; Indicates via a single color sub-image The view at the observation point that provides image information for reconstruction is referred to as view information. Indicates via other colored subgraphs The image information provided is used to fill the view at the observation viewpoint, which is referred to as the filling information. K is a calculation factor with a value of {0, 1}, which is used to represent the source of the viewpoint information at the restored observation viewpoint. When K is 0, the viewpoint information is taken from the view information, and when K is 1, the viewpoint information is taken from the filling information.

[0016] The view information P V1 The depth information is obtained from a single color sub-image and a depth sub-image. The calculation formula can be written as follows:

[0017] P V1 =F1P m -B m (2)

[0018] in This represents the original image information provided by a single color sub-image in the image array, hereinafter referred to as the main view information; This indicates that the main view information is mapped to the view information P. V1 The transmission coefficient, hereinafter referred to as the main view coefficient; This refers to the displacement deviation of the main view information caused by the change in viewing angle, also known as the main view deviation.

[0019] The formula for calculating the main view deviation can be written as:

[0020]

[0021]

[0022] b mx =f vx Z O Z i -1 tanβ (5)

[0023] b my =f vy Z O Z i -1 tanβ (6)

[0024] Where f vx 、f vy These are the focal lengths of the imaging device at the viewpoint, along the x and y axes of the imaging plane after camera geometric calibration; f mx 、fmy These are the focal lengths of the imaging device used to recover the selected single color sub-image, as determined by camera geometry calibration, on the x and y axes of the imaging plane; Z O 、Z i These represent the depth of the baseline in the depth information map and the depth of the corresponding pixel in the reconstructed image, respectively; β is the deflection angle at the observation point.

[0025] The filling information P V2 The depth information is obtained from other color submaps and depth submaps, and the calculation formula can be written as:

[0026] P V2 =F2P s +B s (7)

[0027] in This represents the original image information provided by other color sub-images in the image array, hereinafter referred to as lateral information; This indicates that the lateral information mapping fills the information P. s The transmission coefficient, hereinafter referred to as the side-view coefficient; This refers to the displacement deviation of lateral information caused by the change in viewing angle, hereinafter referred to as lateral view deviation.

[0028] The formula for calculating the side view deviation can be written as:

[0029]

[0030]

[0031] b sx =f vx Z O (tanβ-tanα)Z i -1 (10)

[0032] b sy =f vy Z O (tanβ-tanα)Z i -1 (11)

[0033] Where f sx f sy These are the focal lengths of the optical device at the maximum lateral viewing angle on the x and y axes of the imaging plane, after camera geometric calibration; α is the maximum deflection angle of the color sub-image that provides lateral information.

[0034] Beneficial effects: 1. The image format proposed in this invention can effectively fill holes caused by perspective deviation in three-dimensional scenes, generating high-quality and reliable image pixels. It avoids the difficulty of filling holes caused by lack of information in traditional images and the complex and cumbersome steps brought about by iterative methods.

[0035] 2. The image format constructed using multiple sub-images allows the acquired information to be used for hole filling in images with a larger field of view, so that the observation range of the scene is no longer limited to a very small observation angle. At the same time, the holes recovered in this format can be applied to high-resolution images, enabling faster and more efficient real-time calculation and observation on the display device. Attached Figure Description

[0036] Figure 1 This is a sub-image illustration of obtaining target scene image information in step 1 of the present invention;

[0037] Figure 2 This is a schematic diagram of the rectangular image array format using four sub-images as an example, as described in step 2 of the present invention;

[0038] Figure 3 This is a schematic diagram of the column vector image array format described in step 2 of the present invention, using four sub-images as an example;

[0039] Figure 4 This is a schematic diagram of the column vector image array format described in step 2 of the present invention, using four sub-images as an example;

[0040] Figure 5 This is a schematic diagram of the cross-image array format described in step 2 of the present invention, using several sub-images as an example;

[0041] Figure 6 This is a schematic diagram of the method for reconstructing and filling holes in the graphic in step 3 of the present invention. Detailed Implementation

[0042] The invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, those skilled in the art will find that various equivalent modifications to the invention fall within the scope defined by the appended claims.

[0043] Example:

[0044] like Figure 1 As shown in the figure, an image format for achieving high-resolution ultra-wide field of view 3D display disclosed in this embodiment of the invention mainly includes the following steps:

[0045] Step 1) Obtain image information of the target scene.

[0046] First, we obtain the image information of the target scene. Here, we choose a more convenient method, which is to obtain four sub-images of the target scene. These are two color images of the scene taken by the camera with the maximum imaging angle as the main optical axis, serving as the left and right side view sub-images. The color image taken at the angle bisector of the maximum angle, which is the scene normal, is the main view sub-image, and the depth sub-image of the main view is also included. It should be noted that this approach is not unique, and those skilled in the art may also choose to use, but are not limited to: 1. obtaining two scene color images at the maximum observation angle and a depth map at the scene normal; 2. obtaining two scene color images at the maximum observation angle and the depth maps corresponding to the two color images; 3. obtaining two scene color images at the maximum observation angle, a color image at the scene normal position, and a depth map at any position; 4. increasing or decreasing the number of views while retaining the two scene color images at the maximum observation angle and the depth map located between the maximum observation angles, that is, capturing any number of scene color image information and depth information at any angle as supplementary sub-images as needed while retaining these three sub-images, etc. Other equivalent transformations fall within the scope defined by the appended claims.

[0047] The specific method for acquiring image information used in this embodiment is as follows: taking the center point of the target scene as the origin and extending the scene normal as 0°, two RGB cameras with the same intrinsic parameters are set at each of the ±β angles to acquire two side view sub-images of the scene, denoted as: color image A and color image B; and an RGB-D camera is set along the normal direction to acquire the main view sub-image and depth sub-image of the scene, denoted as color image C and depth image D.

[0048] To simplify the calculation and imaging process, the camera can be set with the origin as the center. If not set in this way, the captured sub-images need to be transformed during the preprocessing stage. The image can be scaled according to the distance ratio to ensure that the projection of each pixel of the image onto the scene is of the same or similar scale.

[0049] Step 2) Combine the sub-images into a complete image format.

[0050] The three color images and one depth image are aligned with the same direction and angle and then recombined into a single complete image for computation and imaging. This image is then transmitted as the input array to the computing and imaging device. This array uses four sub-images as elements. The array format can be a 2×2 matrix array, a 1×4 row vector array, or a 4×1 column vector array. The position of each sub-image within the array is arbitrary; during computation, only the position index of the corresponding image pixels needs to be explicitly identified.

[0051] When generating a complete image from four sub-images, to simplify the calculation and imaging process, attention should be paid to image alignment. That is, if any sub-images constituting the array are in the same row or column in the array, the projected pixels in world coordinates should be in the same row or column of the array.

[0052] Step 3) Use the image format from Step 2) to generate a viewpoint image and fill the holes caused by the deflection of the observation angle.

[0053] Considering that the actual world coordinates corresponding to each pixel are the same in different images, the image at the observation viewpoint can be filled in by the pixels of the main view sub-image based on the principles of projection geometry. This represents the filling of the image based on the pixels of the main view subimage at any viewing angle. The viewing view can be represented as follows: (Indexed by the pixels of the main view subimage in an array of four subimages)

[0054] P V1 =F1P m -B m (1)

[0055] in Specifically, f x1 and f y1 These are the transformation coefficients of the main view subgraph along the x-axis and y-axis, respectively.

[0056] This can be obtained from projective geometric relations. Where f vx f mx These are the camera parameters for the viewpoint position along the x-axis and the camera parameters for the main view position, respectively; f vy f my These are the camera parameters at the viewpoint position along the y-axis and the camera parameters at the main viewpoint, respectively. The camera parameters at the viewpoint can be arbitrary and will be simulated and calculated in the imaging device, allowing for more flexible simulation of multi-parameter cameras for imaging.

[0057] This represents the image information provided by the main view subgraph. This information, constrained by the transformation coefficients F1, makes the image P at the observation viewpoint... V1 The corresponding pixel information in the main view sub-image can be calculated in the imaging process based on the transformation relationship of projective geometry.

[0058] This represents the shift deviation in the mapping process of the main view sub-image caused by the viewing angle deflection, where b mx b my These represent the shift deviations of the main view subimage along the x-axis and y-axis, respectively. The shift deviation of the main view subimage can be expressed as:

[0059] b mx =f vx Z O Z i -1 tanβ (2)

[0060] b my =f vy Z O Z i -1 tanβ (3)

[0061] where Z O 、Z i These represent the baseline depth of the depth information map and the depth of each pixel in the scene, respectively. This information can be obtained from the depth sub-map of the image array. To simplify the calculation process, the maximum depth in the image can be selected as the baseline depth, or the depth baseline can be set according to actual needs. Simply adjust the ratio based on the depth values ​​during the calculation. β represents the observation position offset angle, which should fall within the range of the side view sub-maps.

[0062] Step 4) Using the generated image obtained from the front view sub-image mapping in Step 3), fill the holes in the generated image using the image format from Step 2). Although the holes in the observation position view lack pixel information, their corresponding image coordinates can be used to find the corresponding pixels in the two side view sub-images and fill them using the mapping principle of polyprojective geometry. (Note: The original text contains some inconsistencies and unclear punctuation. A more accurate translation would require the full context.) This represents a hole in the image at any viewing angle that needs to be filled using pixels from two side-view sub-images. Indexing the pixels of two side-view sub-images in an array of four sub-images, the hole portion of the viewing view can be represented as:

[0063] P V2 =F2P s +B s (4)

[0064] in Specifically, f x2 and f y2 These are the transformation coefficients of the side view subgraph along the x-axis and y-axis, respectively.

[0065] This can be obtained from projective geometric relations. Where f vx f mx These are the camera parameters for the viewpoint position along the x-axis and the camera parameters for the position of the maximum deflection angle, respectively; f vy f myThese are the camera parameters for the viewpoint position along the y-axis and the camera parameters for the position of the maximum deflection angle, respectively. The camera parameters for the viewpoint position should be consistent with the camera parameters used in step 3) to ensure that the generated viewpoint image at any angle is a simulation of the view captured by the same camera. This conforms to the way the human eye observes, avoids complex preprocessing, and allows for more efficient and reasonable filling of hole pixels using the image array in step 2).

[0066] This represents the image information provided by the side view sub-image. This information, constrained by the transformation coefficient F2, makes the image P at the observation viewpoint... V2 The corresponding pixel information in the side view sub-image can be calculated in the imaging process based on the transformation relationship of projective geometry.

[0067] This represents the shift deviation in the side view sub-image mapping process caused by the viewing angle deflection, where b sx , b sy These represent the shift deviations of the side view sub-images along the x-axis and y-axis, respectively. The shift deviation of the side view sub-images can be expressed as:

[0068] b sx =f vx Z O (tanβ-tanα)Z i -1 (5)

[0069] b sy =f vy Z O (tanβ-tanα)Z i -1 (6)

[0070] where Z O 、Z i Similar to step 3), the information also comes from the depth submap, where α and β are the maximum lateral deflection angle and the observation position deflection angle, respectively.

[0071] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for realizing an image format for high-resolution ultra-wide field of view three-dimensional display, It is characterized in that The following steps are involved: Step 1. Obtaining target scene image information, the acquired target scene image information includes two left and right scene color images taken by a camera with a maximum imaging angle as the main optical axis as left and right side view sub-images and a depth image captured at any angle between the maximum angles to restore the scene as a depth sub-image; Step 2. Recombining the acquired sub-images into an input array, each part of which is a complete expression of the original sub-image, and inputting the recombined array into the imaging device; Step 3. Reconstruct the input array using a three-dimensional scene completion algorithm and complete the completion of image holes caused by perspective transformation based on the image format; In the step 1, the image information of the target scene is obtained, and in addition to the left and right side view sub-images and the depth sub-image of the scene, more color sub-images and depth sub-images of the target scene are added to the input array as supplementary information to form an image format that realizes high-resolution ultra-wide field of view three-dimensional display; The three-dimensional scene completion algorithm in step 3 realizes any viewing angle change within the maximum observation angle. For the restoration of the image and the filling of the holes under any viewing angle, the depth information in the depth sub-image is used to index the pixels at the same spatial position in the two side views as the filling of the pixels in the hole part. The calculation formula is written as: P V =(1-K)P V1 +KP V2 (1) in Represents the image information at any observation angle, hereinafter referred to as viewpoint information; Represented by a single color sub-image A view at an observation viewpoint reconstructed by providing image information, hereinafter referred to as view information; Indicates that other colored sub-images The view at the observation viewpoint after providing image information for image filling is referred to as filling information hereinafter; K is a calculation factor, and its value is {0, 1}, which is used to indicate the source of the viewpoint information at the restored observation viewing angle. When K is 0, the viewpoint information is taken from the view information, and when K is 1, the viewpoint information is taken from the filling information.

2. The method for realizing an image format of high-resolution ultra-wide field of view three-dimensional display according to claim 1, Features: In step 2, several sub-graphs are reassembled into an input array in the form of Input in the form of a square matrix, or in the form of a row vector of the form [ABCD] or in the form of [ABCD] T The sub-images are input into the imaging device in the form of a column vector, where the order of arrangement of the sub-images is arbitrary.

3. The method for realizing an image format of high-resolution ultra-wide field of view three-dimensional display according to claim 1, Features: In step 2, the plurality of sub-images are reassembled into an input array by sequentially arranging the plurality of input sub-images according to pixel positions so that the plurality of sub-images are staggered to form the overall input array.

4. The method for realizing an image format of high-resolution ultra-wide field of view three-dimensional display according to claim 1, Features: The three-dimensional scene completion algorithm in step 3 that uses the input array in step 2 to complete the holes in the three-dimensional scene is specifically decomposed into two parts: a. pixel restoration of the view at the viewing angle position; b. The side view sub-image completes the image holes at any observation angle.

5. The method for realizing an image format of high-resolution ultra-wide field of view three-dimensional display according to claim 1, Features: The view information P V1 The depth information is obtained through a single color sub-image and a depth sub-image. The calculation formula is written as: P V1 =F 1 P m -B m (2) in Represents the original image information provided by a single color sub-image in the image array, hereinafter referred to as the primary view information; Indicates that the main view information is mapped to the view information P V1 The transfer coefficient, hereinafter referred to as the primary view coefficient; It refers to the displacement deviation of the main view information caused by the deviation of the viewing angle, referred to as the main view deviation.

6. The method for realizing an image format for high-resolution ultra-wide field of view three-dimensional display according to claim 5, Features: The main view deviation calculation formula is written as: b mx =f vx Z O Z i -1 tanβ (5) b my =f vy Z O Z i -1 tanβ (6) where f vx 、f vy are the focal lengths of the imaging device at the viewpoint on the x-axis and y-axis of the imaging plane after camera geometric calibration; f mx 、f my are respectively the focal lengths of the imaging device selected for restoring a single color sub-image on the x-axis and y-axis of the imaging plane after camera geometric calibration; Z O , Z i are the depth of the baseline of the depth information map and the depth of the corresponding pixel of the reconstructed image respectively; β is the deflection angle at the observation point.

7. The method for realizing an image format of high-resolution ultra-wide field of view three-dimensional display according to claim 1, Features: The filling information P V2 It is obtained through the depth information provided by other color sub-images and depth sub-images. The calculation formula is written as: P V2 =F 2 P s +B s (7) in represents the original image information provided by other color sub-images in the image array, hereinafter referred to as lateral information; Indicates the lateral information mapping filling information P s The transfer coefficient, hereinafter referred to as the side view coefficient; It refers to the displacement deviation of lateral information caused by the deviation of viewing angle, hereinafter referred to as lateral viewing deviation.

8. The method for realizing an image format for high-resolution ultra-wide field of view three-dimensional display according to claim 7, Features: The side view deviation calculation formula is written as: b sx =f vx Z O (tanβ-tanα)Z i -1 (10) b sy =f vy Z O (tanβ-tanα)Z i -1 (11) where f sx 、f sy are respectively the focal lengths of the optical device with maximum lateral viewing angle on the x-axis and y-axis of the imaging plane after geometric calibration of the camera; α is the maximum deflection angle of the color sub-image providing lateral information.