Method for generating 3D scene and computer system

Abstract

There is provided a method of generating a 3D scene, the method comprising the steps of: establishing a 3D mesh having a plurality of mesh vertices; receiving an original image; estimating depth information of the original image; updating the 3D mesh based on the estimated depth information; projecting each grid vertex of the 3D grid to a coordinate system of the 3D scene based on the user position information; and coloring the 3D mesh projected onto the coordinate system based on texture information associated with the original image to obtain a 3D scene.

Description

Technical Field

[0001] The present disclosure relates to image processing technology, and in particular to 3D (three-dimensional) scene generation. Background Art

[0002] Current methods of generating 3D scenes usually use the principle of binocular parallax, that is, letting the right eye and the left eye see slightly different images so that the brain interprets the parallax caused by the different perspectives of the two eyes as depth perception.

[0003] However, the 3D effect generated based on binocular parallax needs to be displayed on a display device specifically used to display 3D scenes, such as a naked-eye stereoscopic display device or a wearable device (such as 3D / VR glasses or head-mounted devices). Without such a 3D display device, users cannot watch 3D scenes.

[0004] Therefore, it is desirable to provide a solution for 3D scene generation that can realize 3D perception on a general display device for displaying 2D images. Summary of the invention

[0005] One embodiment of the present disclosure provides a method for generating a 3D scene. The method is implemented by a computer device. The method includes the step of establishing a 3D mesh. The 3D mesh has a plurality of mesh vertices. The method further includes the step of receiving an original image. The method further includes the step of estimating the depth information of the original image. The method further includes the step of updating the 3D mesh based on the estimated depth information. The method further includes the step of projecting each mesh vertex of the 3D mesh onto a coordinate system of the 3D scene based on user position information. The method further includes the step of coloring the 3D mesh projected onto the coordinate system based on texture information associated with the original image to obtain a 3D scene.

[0006] In one embodiment, the method further comprises the step of displaying the 3D scene on a display device, wherein the display device is neither a autostereoscopic display device nor a wearable device.

[0007] In one embodiment, the method further comprises the step of identifying edge regions of objects in the original image using an image segmentation algorithm. The method further comprises the step of reducing edge artifacts in the edge regions of the objects in the original image to obtain a fine-tuned image. The method further comprises the step of extracting texture information from the fine-tuned image.

[0008] In one embodiment, the step of reducing edge artifacts in the edge region of the object in the original image to obtain the fine-tuned image includes using a blur algorithm to perform blur processing on the edge region of the object in the original image.

[0009] In one embodiment, the step of reducing edge artifacts in an edge region of an object in an original image to obtain a fine-tuned image includes modifying a portion of pixel values ​​in the edge region based on a background region outside the object and adjacent to the edge region.

[0010] In one embodiment, the step of reducing edge artifacts in edge regions of objects in the original image to obtain a fine-tuned image includes determining multiple layers of the original image according to depth information, and expanding layers corresponding to depth values ​​below a threshold.

[0011] In one embodiment, the method further comprises the step of capturing an image of the user using a camera. The method further comprises the step of calculating the user's position information based on the image of the user using a face tracking algorithm or an eye tracking algorithm.

[0012] In one embodiment, the step of coloring the 3D mesh projected onto the coordinate system to obtain the 3D scene based on texture information associated with the original image includes interpolating the color texture of the original image onto the 3D mesh projected onto the coordinate system of the 3D scene.

[0013] In one embodiment, the step of creating a 3D mesh includes creating a texture of a triangle formed by three adjacent mesh vertices.

[0014] An embodiment of the present disclosure provides a computer system for generating a 3D scene. The computer system includes a storage device and a processing device. The processing device loads a program from the storage device to execute the steps of the above method.

[0015] The method and computer device disclosed herein provide a novel technology that generates a scene with a 3D effect that can be displayed on a general display device for displaying 2D images without the need for a naked-eye stereo display device or a wearable device. Since there is no need to draw scenes for both eyes separately, and there is no need to perform interlacing (or weaving) processing for naked-eye stereo display, computing resources can be greatly saved. In addition, the potential risk of vergence-accommodation conflict (VAC) caused by binocular parallax can be avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present disclosure will be better understood from the following illustrative embodiments with accompanying drawings. In addition, it should be understood that in the flowchart of the present disclosure, the execution order of each block can be changed, and / or some blocks can be changed, deleted or merged.

[0017] Figure 1 The conceptual flow of a conventional 3D scene generation method is shown as an example.

[0018] Figure 2 is a flowchart of a method 200 for generating a 3D scene according to an embodiment of the present disclosure.

[0019] Figure 3 is a schematic block diagram of a computer system 300 for generating a 3D scene according to an embodiment of the present disclosure.

[0020] The reference numerals are described as follows:

[0021] 10: Original image

[0022] 100: Depth Information

[0023] 101: Grid

[0024] 102: Updated grid

[0025] 103: Left eye image

[0026] 111: Grid

[0027] 112: Updated grid

[0028] 113: Right eye image

[0029] 120: Side-by-side 3D images

[0030] 200: Methods

[0031] S201-S206: Steps

[0032] 300: Computer Systems

[0033] 301: Processing unit

[0034] 302: Storage device

[0035] 303: Display device

[0036] 304: Photographic Installation

[0037] 306: Application

[0038] 307: First Procedure DETAILED DESCRIPTION

[0039] The following description lists various embodiments of the present invention, but is not intended to limit the content of the present invention. The actual scope of the invention is defined by the claims.

[0040] In the various embodiments listed below, the same reference numerals will be used to represent the same or similar elements or components.

[0041] It should be understood that the terms "comprises" and "includes" in this specification are used to indicate the existence of specific technical features, values, method steps, program operations, elements and / or components, but do not exclude additional technical features, values, method steps, program operations, elements, components, or any combination of the above.

[0042] In this specification and claims, serial numbers, such as "first", "second", etc., are only for convenience of description and have no sequential relationship with each other.

[0043] Descriptions of method embodiments also apply to computer system embodiments, and vice versa.

[0044] Figure 1 The conceptual flow of a conventional 3D scene generation method is shown as an example. In the example shown, a mesh model is used to define the shape and depth of an object (such as a mountain) in an image.

[0045] like Figure 1 As shown, the established mesh 101 includes a plurality of triangles (which may be replaced by other types of polygons) arranged in sequence on the XY plane, and each triangle has a set of vertices. According to the depth information 100 captured from the original image 10 and stored in the depth buffer (also called "z buffer"), a specific height (i.e., the scale along the Z axis) is given to the vertex of each triangle in the mesh 101, thereby updating the mesh 101 to the updated mesh 102. Then, by coloring the triangles in the updated mesh 102, a left eye image 103 (i.e., the left eye view of the human eye) is generated. Similarly, the established mesh 111 includes a plurality of triangles (which may be replaced by other types of polygons) arranged in sequence on the XY plane, and each triangle has a set of vertices. According to the depth information 100 captured from the original image 10 and stored in the depth buffer, a specific height (i.e., the scale along the Z axis) is given to the vertex of each triangle in the mesh 111, thereby updating the mesh 111 to the updated mesh 112. Then, based on the color texture of the original image 10 (e.g., the RGB value of each pixel), the triangles in the updated grid 112 are colored to generate a right-eye image 113 (i.e., the right-eye view of the human eye). The left-eye image 103 and the right-eye image 113 appear to be the same, but in fact there are slight differences. Finally, the left-eye image 103 and the right-eye image 113 are aligned side by side to form a side-by-side 3D image 120. The left side of the side-by-side 3D image 120 is from the left-eye image 103, and the right side is from the right-eye image 113.

[0046] In order to present a 3D effect to the user, a display device specifically used to display 3D scenes is required, such as a naked-eye stereoscopic display device or a wearable device (such as 3D / VR glasses or a head-mounted device). If the display device is a wearable device such as 3D / VR glasses or a head-mounted device, the left eye image 103 and the right eye image 113 will be displayed on the left display panel and the right display panel of the wearable device, respectively. If the display device is a naked-eye stereoscopic display device, the side-by-side 3D image 120 will be further interlaced (or woven) based on the characteristics of the naked-eye stereoscopic display device, such as parameters related to the parallax barrier or lenticular lenses of the naked-eye stereoscopic display device. After the interlaced (or woven) images pass through the parallax barrier or lenticular lenses of the naked-eye stereoscopic display device, the user will see different views in the left and right eyes respectively, and thus perceive the 3D effect by binocular parallax.

[0047] The embodiments of the present disclosure utilize the principle of motion parallax, rather than binocular parallax used in the prior art. Generally speaking, in the embodiments of the present disclosure, the scene is generated based on the position of the user relative to the display device screen. When the user moves and changes his position relative to the screen, the scene generated and presented on the display device will be different. In this way, a 3D effect is achieved without the need for a naked-eye stereoscopic display device or a wearable device.

[0048] Figure 2 FIG. 2 is a flow chart of a method 200 for generating a 3D scene according to an embodiment of the present disclosure. Figure 2 As shown, method 200 includes steps S201-S206.

[0049] In step S201, a 3D mesh is created. The created 3D mesh has a plurality of mesh vertices. Then, the method 200 proceeds to step S202.

[0050] In step S202, an original image is received. Then, the method 200 proceeds to step S203.

[0051] In step S203, the depth information of the original image is estimated. Then, the method 200 proceeds to step S204.

[0052] In step S204, the 3D mesh is updated based on the estimated depth information. Then, the method 200 proceeds to step S205.

[0053] In step S205, each mesh vertex of the 3D mesh is projected onto the coordinate system of the 3D scene based on the user position information. Then, the method 200 proceeds to step S206.

[0054] In step S206 , the 3D mesh projected onto the coordinate system is colored based on texture information associated with the original image to obtain a 3D scene.

[0055] In one embodiment, step 201 includes the operation of creating a texture of a triangle formed by three adjacent mesh vertices. For example, a mesh may include multiple vertices and multiple triangles. Each triangle is formed by three adjacent mesh vertices and has a corresponding texture. When creating a mesh, the texture of each triangle will also be created (or initialized) as a data structure.

[0056] In one embodiment, the original image may originate from an application, such as a game, a video / multimedia player program, a web browser, a photo viewing program, or any other visual entertainment application. More specifically, the original image may be an image that the application requests the underlying hardware (e.g., GPU) to render.

[0057] In one embodiment, the depth information may be represented in the form of a depth map, a disparity map or a point cloud, but the present disclosure is not limited thereto. In step S203, the depth information may be estimated using a model based on a convolution neural network (CNN), but the present disclosure is not limited thereto.

[0058] In one embodiment, the image of the user (hereinafter referred to as "user image") is captured using a camera, and the user position information used in step S205 may be calculated based on the user image using a face tracking algorithm or an eye tracking algorithm. The user position information may refer to the position of the user (or the user's face, the user's eyes, etc.) relative to the display device screen, which may be represented by coordinates.

[0059] In one embodiment, step S205 may use various mature 3D projection techniques to map the vertices of the mesh to the coordinate system. These techniques include orthographic projection, perspective projection, oblique projection, axonometric projection, etc., but the present disclosure is not limited to this. Further explanation, the user position information is used as a parameter of the function associated with the above-mentioned 3D projection technology. These functions are used to construct a transformation matrix, which is a key component of the projection process. This transformation matrix takes individual mesh vertices as input and generates their corresponding positions in the coordinate system. In an example implementation, two 3D projection options are provided during step S205: orthogonal projection and perspective projection. The former is a simplified version of the latter, only considering the user's perspective and ignoring its distance from the screen. On the other hand, perspective projection takes into account the user's perspective and its distance from the screen. These projection options can be controlled by parameters, such as "perspective degree", "virtual camera distance", "z distance" or similar concepts. For example, when the Perspective Degree value is non-zero, a perspective projection is used, and when the Perspective Degree value is zero, an orthographic projection is used.

[0060] In one embodiment, step S206 may further include an operation of interpolating the color texture of the original image on the 3D grid projected onto the coordinate system of the 3D scene. The color texture may be represented in various color space formats, such as RGB, YUV, YCbCr, CMYK, sRGB, HSV, etc., but the present disclosure is not limited thereto.

[0061] In one embodiment, the generated 3D scene can be displayed on a general display device for displaying 2D images, which is neither a autostereoscopic display device nor a wearable device (such as 3D / VR glasses or a head-mounted device).

[0062] In some embodiments, the texture information used in step S206 is extracted from a fine-tuned image. The fine-tuned image can be obtained by reducing edge artifacts in the edge region of the object in the original image. The edge region of the object in the original image can be identified using an image segmentation algorithm. The image segmentation algorithm may include any algorithm for boundary tracking, such as a square tracing algorithm, a Moore-neighbor tracing algorithm, a radial sweep algorithm, a Theo Pavlidis algorithm, etc., but the present disclosure is not limited to this. In one embodiment, the image segmentation algorithm can be implemented using a mature machine learning model, such as U-Net, DeepLab, Mask R-CNN, HRNet, Enet, etc., but the present disclosure is not limited to this.

[0063] There are many methods to reduce edge artifacts in edge areas. The first method provided herein is to use a blurring algorithm to blur the edge area of ​​an object in an original image. The blurring algorithm may be, for example, a box blurring method, a Gaussian blurring method, a pixelation method based on differential privacy, etc., but the present disclosure is not limited thereto. Using the first method, the edge area of ​​the object becomes less obvious, thereby applying a stronger depth effect. The second method provided herein is to modify a portion of the pixel values ​​of the edge area based on a background area outside the object and adjacent to the edge area. Using the second method, a portion of the edge area may be filled with reference to the content of the background, thereby smoothing the edge area of ​​the object. The third method provided herein is to determine multiple layers of the original image based on depth information, wherein each layer corresponds to a depth value, and then expand the layer corresponding to the depth value below a threshold. The degree of expansion may be determined by the depth value. Using the third method, the layer corresponding to the smaller depth value may be expanded more, thereby reducing the size of the edge artifact.

[0064] In one embodiment, the above three methods (ie, at least one of the first method, the second method and the third method) may be used to reduce edge artifacts in edge areas of objects in the original image.

[0065] The method 200 may be implemented by a computer system, the structure of which will be referred to as Figure 3 Give a narrative.

[0066] Figure 3 FIG. 3 is a schematic block diagram of a computer system 300 for generating a 3D scene according to an embodiment of the present disclosure. Figure 3 As shown, the computer system 300 includes components such as a processing unit 301 , a storage device 302 , a display device 303 , and a photographing device 304 .

[0067] The computer system 300 is an electronic device capable of executing computing tasks, such as a personal computer (including a desktop computer, a notebook computer, a tablet computer, etc.) or a server computer.

[0068] The processing unit 301 may include any device / element for executing instructions, such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a microcontroller, a state machine, an application-specific integrated circuit (ASIC), a single-chip system (SoC), etc., but the present disclosure is not limited to this.

[0069] The storage device 302 is a non-volatile memory device, such as a hard disk, a solid state drive, a flash memory, or a read-only memory, but the present disclosure is not limited thereto. According to an embodiment of the present disclosure, the storage device 302 is used to store an application 306 and a first program 307. The application 306 can be any software program that provides visual images or scenes to the audience, such as a game, an audio / video / multimedia player program, a photo viewing program, or any other visual entertainment application. The first program 307 includes a plurality of instructions that enable the processing device 301 to load the first program 307 from the storage device 302 and execute the steps of the method 200.

[0070] The display device 303 is used to display the generated 3D scene. According to an embodiment of the present disclosure, the display device 303 may be neither a naked-eye stereoscopic display device nor a wearable device (such as 3D / VR glasses or a head-mounted device), but a general display device for displaying 2D images.

[0071] The photographing device 304 may be any device for photographing. The photographing method may include ordinary optical photographing or infrared photographing. The present disclosure does not limit the type or photographing method of the photographing device 304. According to an embodiment of the present disclosure, the photographing device 304 is used to capture the user image required for calculating the user position information.

[0072] The method and computer device disclosed herein provide a novel technology that generates a scene with a 3D effect that can be displayed on a general display device for displaying 2D images without the need for a naked-eye stereo display device or a wearable device. Since there is no need to draw scenes for both eyes separately, and there is no need to perform interlacing (or weaving) processing for naked-eye stereo display, computing resources can be greatly saved. In addition, the potential risk of vergence-accommodation conflict (VAC) caused by binocular parallax can be avoided.

[0073] The above paragraphs are described using multiple aspects. Obviously, the teachings of this article can be implemented in multiple ways, and any specific architecture or function disclosed in the examples is only a representative case. According to the teachings of this article, it should be understood in the art that each aspect disclosed in this article can be implemented independently, or two or more aspects can be combined to implement.

[0074] Although the present disclosure has been described above with reference to the embodiments, it is not intended to limit the present disclosure. Anyone skilled in the art may make slight changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the invention shall be determined by the following claims.

Claims

1. A method for generating a 3D scene, implemented by a computer device, comprising the following steps: Creating a 3D mesh, wherein the 3D mesh has a plurality of mesh vertices; Receiving an original image; estimating depth information of the original image; Updating the 3D mesh based on the estimated depth information; Based on the user position information, project each mesh vertex of the 3D mesh onto a coordinate system of a 3D scene; as well as The 3D mesh projected onto the coordinate system is colored based on texture information associated with the original image to obtain the 3D scene.

2. The method of claim 1, further comprising: The 3D scene is displayed on a display device, wherein the display device is neither a auto-stereoscopic display device nor a wearable device.

3. The method of claim 1, further comprising: Using an image segmentation algorithm to identify an edge region of an object in the original image; reducing edge artifacts in the edge region of the object in the original image to obtain a fine-tuned image; and The texture information is extracted from the fine-tuned image.

4. The method of claim 3, wherein the step of reducing edge artifacts in the edge region of the object in the original image to obtain the fine-tuned image comprises: A blur algorithm is used to perform blur processing on the edge area of ​​the object in the original image.

5. The method of claim 3, wherein the step of reducing edge artifacts in the edge region of the object in the original image to obtain the fine-tuned image comprises: Based on a background area outside the object and adjacent to the edge area, a portion of pixel values ​​in the edge area is modified.

6. The method of claim 3, wherein the step of reducing edge artifacts in the edge region of the object in the original image to obtain the fine-tuned image comprises: Determine multiple layers of the original image according to the depth information, wherein each layer corresponds to a depth value; and The layer corresponding to the depth value below a threshold is enlarged.

7. The method of claim 1, further comprising: Using a camera to capture an image of a user; and The user position information is calculated based on the user image using a face tracking algorithm or an eye tracking algorithm.

8. The method of claim 1, wherein the step of coloring the 3D mesh projected onto the coordinate system based on the texture information associated with the original image to obtain the 3D scene comprises: The color texture of the original image is interpolated onto the 3D grid projected onto the coordinate system of the 3D scene.

9. The method of claim 1, wherein the step of establishing the 3D grid comprises: A texture of a triangle formed by three adjacent vertices of the mesh is created.

10. A computer system for generating a 3D scene, comprising a storage device and a processing device, wherein the processing device loads a program from the storage device to perform the following steps: Creating a 3D mesh, wherein the 3D mesh has a plurality of mesh vertices; Receiving an original image; estimating depth information of the original image; Updating the 3D mesh based on the estimated depth information; Based on the user position information, project each mesh vertex of the 3D mesh onto a coordinate system of a 3D scene; as well as The 3D mesh projected onto the coordinate system is colored based on texture information associated with the original image to obtain the 3D scene.

11. The computer system of claim 10, further comprising: A display device, used to display the 3D scene; The display device is neither a naked-eye stereoscopic display device nor a wearable device.

12. The computer system of claim 10, wherein the processing device further performs the following steps: Using an image segmentation algorithm to identify an edge region of an object in the original image; reducing edge artifacts in the edge region of the object in the original image to obtain a fine-tuned image; and The texture information is extracted from the fine-tuned image.

13. The computer system of claim 12, wherein the processing device further performs the following steps to reduce edge artifacts in the edge region of the object in the original image: A blur algorithm is used to perform blur processing on the edge area of ​​the object in the original image.

14. The computer system of claim 12, wherein the processing device further performs the following steps to reduce edge artifacts in the edge region of the object in the original image: Based on a background area outside the object and adjacent to the edge area, a portion of pixel values ​​in the edge area is modified.

15. The computer system of claim 12, wherein the processing device further performs the following steps to reduce edge artifacts in the edge region of the object in the original image: Determine multiple layers of the original image according to the depth information, wherein each layer corresponds to a depth value; and The layer corresponding to the depth value below a threshold is enlarged.

16. The computer system of claim 10, further comprising: A photographic device for capturing an image of a user; The processing device further uses a face tracking algorithm or an eye tracking algorithm to calculate the user position information based on the user image.

17. The computer system of claim 10, wherein the processing device further performs the following steps to colorize the 3D mesh: The color texture of the original image is interpolated onto the 3D grid projected onto the coordinate system of the 3D scene.

18. The computer system of claim 10, wherein the processing device further performs the following steps to create the 3D mesh: A texture of a triangle formed by three adjacent vertices of the mesh is created.

Images