Method for improving naked-eye 3D display clarity

Abstract

The present invention provides a method for improving naked-eye 3D display clarity, comprising: acquiring a scene containing a target object, photographing the scene at a plurality of observation viewing angles including a main viewing angle, acquiring a depth map obtained by photographing the scene at the main viewing angle, and acquiring a depth-of-field range; on the basis of a scene image captured at the main viewing angle and the depth map, compressing the depth of the scene image within the depth-of-field range by means of scene transformation, so as to obtain a compressed scene image having undergone depth compression; using the compressed scene image to generate a composite viewpoint image at each observation viewing angle on the basis of an occlusion and viewing angle offset relationship between each observation viewing angle and the main viewing angle; and at different viewing angles, using a naked-eye 3D device to display composite viewpoint images or compressed scene images corresponding to the observation viewing angles, so as to holographically display the scene. By adjusting the depth of the scene image approaching the depth-of-field of a focal plane, the present invention can maximize the satisfaction and utilization of the performance and display requirements on a display device.

Description

Method for improving the sharpness of naked-eye 3D display TECHNICAL FIELD

[0001] The present application relates to the field of holographic image display, and in particular to a method for improving the sharpness of holographic photo frame display. BACKGROUND

[0002] A color picture is usually composed of pixels of three colors, red, green and blue. A depth map is also a kind of image; however, instead of providing a color for each pixel, it indicates the distance from the camera to that part of the image (either an absolute distance, or a relative distance with respect to other pixels in the depth map).

[0003] A color picture and a depth picture taken through a camera lens can be combined by an algorithm to generate a picture with depth information, for example, a tree trunk far away is very blurred, while leaves close by are very clear. In software, a 3D model has volume, and the color and distance to the camera of each point of the 3D model under the camera view can be calculated.

[0004] Naked-eye 3D display is based on the encoding of the corresponding display system, so that different observation angles see the corresponding view angle content. The content generation process of each observation angle includes: selecting a view angle of interest as the observation main view angle, and the content of the remaining observation view angle will be generated based on the content of the view angle. For example, a holographic photo frame provides multiple discrete views of a 3D scene, and presents these views within a certain angle range, for example, a 58° wide view cone. This view arrangement will induce the visual perception system to view the 3D object in two main ways: when the user moves his head in the scene, by changing their aspect in the scene (parallax); by presenting different view angles to each eye (stereoscopic vision).

[0005] Figure 2 is a schematic diagram showing the way a holographic photo frame directs different images. In the top center of Figure 2 is a top view of the holographic photo frame. The 3D model view is a stick with a ring and two spheres hanging from it. In the upper right corner are the 2D images displayed by each view perspective. The corresponding view perspective is indicated by the dark lines attached to the mirror. In the upper left corner is an image that displays all 45 views at the same time (a view point map).

[0006] The view point map is broken down as shown in Figure 3. The pictures with numbers from small to large represent the view angles from left to right. When the upper set of faces is displayed on the holographic photo frame, the results are shown in Figures 4a, 4b and 4c. The increasing numbers present the main view angles in front of the observer's eyes, and when the observer's view angle moves from left to right, the eyes see different view angles. This is the reason why the present application has 3D stereoscopic vision when looking at the holographic photo frame.

[0007] The display principle of the holographic photo frame is that the lens grating is used to enable the corresponding view image to be seen from a specific angle (in fact, due to the existence of the grating, there is a certain limitation, and the two eyes exist at a distance, so multiple view images can be seen at the same time). A mathematical mapping relationship is used to map a certain pixel under a certain view to a certain pixel of the final view point image. The portrait picture with depth information is rendered angle by angle, and the view point image is synthesized to obtain the display content of the holographic photo frame. When directly looking at the holographic photo frame, the background of the person is too blurred. The background of the view point image synthesized according to multiple views is at a position away from the focal plane, and the same point of the same object in different views will appear at different positions after the synthesis of the view point image, resulting in an unclear picture.

[0008] In the holographic photo frame, the depth information in the picture also causes different pixels at different depths to exhibit different situations. The clearest depth appears to be the focal plane. As shown in the middle of the picture 2, the position of the focal plane is indicated by the dashed line in the middle of the display screen. Only the ball in the middle does not change the viewing angle position even if the application rotates, and remains clear and sharp, which means that it is displayed at the same pixel space position in all 45 views. That is, the object residing on the focal plane does not move with the movement of the viewer's head. However, objects closer or farther than the plane will exhibit parallax. For example, when the observer moves to the right, the foreground sphere will move to the left, and the circular ring in the background will move to the right. The distance of the circular ring and the foreground sphere relative to the focal plane will also change.

[0009] Parallax causes the pixels of many views of an object to appear simultaneously in the application view, resulting in blurring. When looking at the holographic photo frame, each eye actually sees many views, not just one. In the screenshot as shown in the figure, you can see that this camera simultaneously displays the parallax of views 37-41 with different effects:

[0010] The advantage of this view layering is that when the viewer's head moves around, the viewer does not discretely jump from one view to another, but rather fades in and out between many views. This makes the visual experience smoother and more natural. The disadvantage is that it introduces blurring. The degree of blurring can be understood by comparing the sharpness of different parts of the model. The ball in the middle looks clear and sharp, because the ball in the middle is always on the focal plane, i.e. within the depth of field range, so it can be clearly displayed in many view images. The foreground sphere and the background circular ring look blurred, because they exhibit parallax beyond the depth of field range, which means they will be displayed at different pixel positions according to perspective. Therefore, they appear blurred in this shot, because what is actually seen is a mixture of views 37-41. The application refers to the relatively clear part near the focal plane as the depth of field (DOF).

[0011] In order to reduce the blur, the prior art re-composes the model, for example, the direct method of parallax-induced blur is to place important content on the focal plane. The text looks clearest here, when very close to the plane, the 3D model begins to look very clear and sharp. The second is to reduce the complexity of the model appearing in the frame. Or use depth of field blur, since blur is the result of many discrete images mixed together, so the blur (especially for objects with a lot of parallax) will show discrete views, which sometimes make 3D dizzy.

[0012] Invention disclosure

[0013] The present application proposes to solve the problem of background blur caused by the parallax of the holographic frame. The 3D model dataset has the characteristics of large and complex model scene, and it is impossible to place all the models in the DOF area by composition to make the imaging effect clear.

[0014] In view of the shortcomings of the prior art, the present application proposes a solution, the clarity of naked-eye 3D display depends on the parallax of the view, therefore the present application controls the depth of field distance, controls the parallax size of the view, and thereby controls the blur degree. That is, the core technical idea of the present application is to compress the background to the edge area of the camera DOF, which can reduce the blur degree of the background. In addition, in order to ensure the parallax effect, the parallax of the background is recalculated, and the person and the background are combined to generate a view point graph.

[0015] Specifically, the present application proposes a method for improving the clarity of naked-eye 3D display, characterized in that it comprises:

[0016] An initial step of obtaining a scene with a target object, taking the scene from a plurality of observation angles including a main angle, obtaining a depth map of the scene taken at the main angle, and obtaining a depth of field range;

[0017] A compression step of compressing the depth of the scene graph taken at the main angle to the depth of field range by scene transformation according to the scene graph and the depth map, to obtain a compressed scene graph after depth compression;

[0018] A synthesis step of generating a synthesis view point graph at each observation angle according to the occlusion and angle offset relationship between each observation angle and the main angle by using the compressed scene graph;

[0019] A display step of displaying the synthesis view point graph or the compressed scene graph corresponding to the observation angle at different angles by using naked-eye 3D equipment to display the scene holographically.

[0020] By the compression step, the present application can transform the scene to be displayed according to the performance of the display device and the display requirement, and change the depth of field range, so as to realize the display ability of the naked eye 3D device, so that the scene which is beyond or less than the display depth of field range can be filled in the display ability (depth of field range) of the display device or the display depth of field range specified by the user through the scene transformation, so as to maximize the use of the display device performance or meet the display requirement of the user, and avoid the dizziness caused by the too large distance between the foreground and the background or the poor stereoscopic effect caused by the too narrow distance.

[0021] The method for improving the naked eye 3D display definition, wherein the initial step comprises: constructing a 3D model of the scene, dividing the 3D model into a foreground including the target object and a background not including the target object; the step 2 comprises: rendering the background to the background picture at the farthest end in the depth of field range according to the depth information of the background in the depth map; obtaining a plurality of observation angles of the viewer, rendering the target object two-dimensional picture of the target object under each observation angle, and combining the target object two-dimensional picture of each observation angle with the background picture to obtain the synthetic view point picture.

[0022] The background of the 3D model is rendered to the farthest end in the depth of field range through the scene transformation, so as to improve the problem of poor observation such as dizziness caused by the too large distance between the foreground and the background.

[0023] The method for improving the naked eye 3D display definition, wherein the scene transformation in the compression step comprises: compressing the depth of the scene under the main observation angle to the depth of field range by using the transformation relationship of the non-linear transformation and the rendering.

[0024] In order to improve the efficiency of the scene transformation, the 3D model under the main observation angle is rendered instead of rendering the 3D model under all observation angles, so as to improve the rendering efficiency and ensure the rendering quality.

[0025] The method for improving the naked eye 3D display definition, wherein the depth of field range is determined according to the camera parameters used in the observation, or the depth of field range is determined according to the display depth of field range of the naked eye 3D device.

[0026] The method for improving the naked eye 3D display definition, wherein the compression step comprises: compressing the part of the foreground which is beyond the depth of field range to the nearest end in the depth of field range.

[0027] The foreground of the 3D model is rendered to the nearest end in the depth of field range through the scene transformation, so as to improve the problem of poor observation such as dizziness caused by the too large distance between the foreground and the background.

[0028] The method for improving the clarity of naked-eye 3D display, characterized in that the naked-eye 3D device is a naked-eye 3D display device based on a cylindrical lens grating or a naked-eye 3D display device based on other principles.

[0029] The method for improving the clarity of naked-eye 3D display, characterized in that the compression of the scene in the compression step comprises: using a nonlinear transformation and a transformation relationship of rendering, selectively compressing different regions of the scene respectively, and compressing the depth of non-target objects while retaining the depth details of target objects.

[0030] From the above scheme, the advantages of the present application are as follows:

[0031] By placing the 3D model to be displayed near the depth of field DOF of the focal plane, and using background compression to place the compressed background near the parallax plane through parallax calculation, the actual display effect can still achieve a stereoscopic effect and the occlusion relationship between the person and the background.

[0032] BRIEF DESCRIPTION OF DRAWINGS

[0033] Fig. 1 is a schematic diagram of a color picture and a depth picture combined picture;

[0034] Fig. 2 is a schematic diagram of a holographic photo frame guiding different image display;

[0035] Fig. 3 is a view point diagram;

[0036] Figs. 4a, 4b and 4c are respectively display effect display diagrams of the holographic photo frame at different viewing angles;

[0037] Fig. 5 is a schematic diagram of a scene in which a camera array takes a person's photo;

[0038] Fig. 6 is a 3D model diagram including a foreground and a background;

[0039] Fig. 7 is a rendered 3D model diagram.

[0040] BEST MODE FOR CARRYING OUT THE INVENTION

[0041] In order to make the above features and effects of the present application more explicit and easy to understand, the following embodiments are described in detail below with reference to the accompanying drawings.

[0042] As shown in the figure, it is a method flowchart of the present application.

[0043] Step 1. Take a person's photo through a camera array. As shown in Fig. 5, the left side of the figure is an array composed of multiple cameras.

[0044] Step 2. Train 3D model. As shown in Fig. 6, the 3D model includes foreground and background parts, and the whole model is equivalent to a cubic space in which the figure (target object) and the background are placed. The background is represented by the sun, clouds and the moon, and the figure is represented by a circle. The triangle represents a virtual camera, indicating the position and direction of the main viewing angle. The square represents the depth range (depth of field) that can be clearly displayed in the device, and the boundary of this range is the dividing line between the background and the foreground. The present application only needs to render the background at the dividing position between the background and the foreground, and since this position is still within the depth of field of the naked-eye 3D device, it can ensure that the background does not appear blurred and unclear. The background part of the main viewing angle is rendered clearly.

[0045] Step 3. As shown in Fig. 7, the cut background part model is rendered from the main viewing angle to obtain a fixed background two-dimensional image, which is placed as a flat background wall at the farthest end of the naked-eye 3D device depth of field. Since this position is still within the depth of field, it is still relatively clear. The figure foreground is placed at the original position of the foreground. In this way, the view from the camera center viewpoint does not change.

[0046] Step 4. Render the viewpoint map. According to different viewing angles, the image corresponding to the viewing angle is rendered.

[0047] Step 5. Display the viewpoint map by the naked-eye 3D device.

[0048] Another way is similar to this way, which also achieves background compression, but compresses the background in a nonlinear manner instead of compressing it to a plane as in the above way. The specific implementation of the remaining steps is consistent with the above, but instead of directly using depth information to divide the model into two parts in steps 2 and 3, the following is used: when rendering normally, rendering will project each vertex from its space coordinates to screen coordinates through a transformation matrix, which includes depth and position information on the screen. When rendering the main viewing angle, the present application uses depth information to perform nonlinear mapping on the depth information of the vertices in the screen space, such as using arctan mapping to roughly maintain the depth of vertices at close distances, while projecting vertices at long distances to close distances. At the same time, the inverse matrix of the transformation matrix is used to reversely modify the spatial coordinates of these vertices equivalently, so that the entire model is compressed, and the scene viewed from the main viewing angle does not change.

[0049] Subsequent step 4 is consistent with the previous way, and the model modified by nonlinear mapping is used to render the viewpoint map of different viewing angles.

[0050] The above way can be regarded as a special case of this way under piecewise function nonlinear mapping:

[0051] where d sepThis represents the furthest point of the depth of field. While maintaining the view from the center viewpoint, this function represents depths less than d. sep The scene remains unchanged, but the depth is greater than d. sep The scene is mapped to d sep On the background wall.

[0052] Next, we will explain this nonlinear mapping in detail. For any vertex p in the scene, in normal 3D rendering, we map it through V (view matrix) and P (projection matrix) and F. pd (Perspective division), and mapping it to screen space s i =F pd (p i VP)

[0053] s i The z-coordinate represents the depth of the object as displayed on the glasses-free display device, while the x and y coordinates represent its position on the screen. Therefore, we want to keep the x and y coordinates constant for s i A nonlinear transformation z' = f(z) is performed on the z-coordinate of the scene to place all objects within the depth of field, thus making the entire scene sharper. This transformation can be achieved by adjusting the z-coordinate of the scene. i =F pd (p i The inverse transformation of VP is derived to all vertices of the scene, thus realizing a non-linear mapping of the scene depth. By selecting an appropriate z' = f(z) mapping, the scene is compressed into the depth range while keeping the view from the main viewpoint unchanged, and the depth information of the foreground of the scene is not greatly affected.

[0054] Industrial application

[0055] This invention proposes a method to improve the clarity of naked-eye 3D displays, comprising: acquiring a scene with a target object; photographing the scene from multiple observation angles including a main viewpoint; acquiring a depth map of the scene photographed from the main viewpoint and obtaining the depth of field range; based on the scene map photographed from the main viewpoint and the depth map, compressing the depth of the scene map to the depth of field range through scene transformation to obtain a depth-compressed scene map; using the compressed scene map, generating a synthetic viewpoint map for each observation angle based on the occlusion and viewpoint offset relationship between each observation angle and the main viewpoint; and using a naked-eye 3D device to display the synthetic viewpoint map or compressed scene map corresponding to the observation angle from different viewpoints to holographically display the scene. By placing the depth of the scene map near the depth of field (DOF) of the focal plane through the compression step, this invention can meet the performance and display requirements of display devices, and the actual display effect still achieves a stereoscopic effect and the occlusion relationship between the character and the background while maintaining a clear background.

Claims

1. A method of improving the clarity of a naked-eye 3D display, characterized by, The method comprises: an initial step of capturing a scene with an object, capturing the scene from a plurality of observation viewpoints including a main viewpoint, capturing a depth map of the scene from the main viewpoint, and capturing a depth of field range; a compression step of compressing the depth of the scene map to the depth of field range by scene transformation according to the scene map from the main viewpoint and the depth map, to obtain a compressed scene map after depth compression; a synthesis step of generating a synthesis viewpoint map from each observation viewpoint according to the occlusion and viewpoint offset relationship between each observation viewpoint and the main viewpoint using the compressed scene map; a display step of displaying the synthesis viewpoint map or the compressed scene map corresponding to the observation viewpoint from different viewpoints using a naked-eye 3D device to display the scene holographically.

2. The method of claim 1, wherein the method is performed by a 3D display device. The initial step comprises constructing a 3D model of the scene, dividing the 3D model into a foreground including the object and a background not including the object; step 2 comprises rendering the background to a background picture located at the farthest end of the depth of field range according to the depth information of the background in the depth map; obtaining a plurality of observation viewpoints of the viewer, rendering to obtain an object two-dimensional picture of the object under each observation viewpoint, and combining the object two-dimensional picture of each observation viewpoint with the background picture to obtain the synthesis viewpoint map.

3. The method of claim 1, wherein the method is performed by a 3D display device. The scene transformation in the compression step comprises: using the transformation relationship of nonlinear transformation and rendering to compress the depth of the scene at the main viewpoint to the depth of field range.

4. The method of claim 1, wherein the method is performed by a 3D display device. The depth of field range is determined according to the camera parameters used during observation, or the display depth of field range of the naked-eye 3D device.

5. The method of claim 2, wherein the method is performed by a 3D display device. The compression step comprises: compressing the part of the foreground that exceeds the depth of field range to the nearest end within the depth of field range.

6. The method of improving the clarity of a naked-eye 3D display of claim 1, wherein, The naked-eye 3D device is a naked-eye 3D display device based on a cylindrical lens grating, or a naked-eye 3D display device based on other principles.

7. The method of improving the clarity of a naked-eye 3D display of claim 1, wherein, The compression of the scene in the compression step comprises: using the transformation relationship of nonlinear transformation and rendering to selectively compress the scene in different regions respectively, and compressing the depth of the non-object part while retaining the depth details of the object.

Images