3D scene restoration based on stereo extraction

Abstract

A system and method for rendering a three-dimensional (3D) scene with improved visual characteristics from a pair of two-dimensional images from different viewpoints. The three-dimensional scene is created by obtaining a first two-dimensional (2D) image of a scene object obtained from a first viewpoint and a second two-dimensional image of the scene object obtained from a second viewpoint different from the first viewpoint, creating a depth map based on the first and second two-dimensional images, creating a three-dimensional scene based on the depth map and the first and second two-dimensional images, detecting an area with incomplete image information in the initial three-dimensional scene, reconstructing the detected area in the three-dimensional scene, determining replacement information and correcting the reconstructed area; and rendering the three-dimensional scene with the modified reconstructed area from multiple viewpoints.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application Serial No. 62 / 737,280, filed on September 27, 2018, entitled “3D Scene Inpainting Using Stereo Extraction,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to rendering three-dimensional scenes, and more particularly, to systems and methods for rendering a two-dimensional scene with improved visual properties from a pair of two-dimensional images having different viewpoints. Background Art

[0004] The computer system uses a rendering program to present a three-dimensional (3D) scene object on a two-dimensional (2D) display. In order to create a three-dimensional representation of the scene object, the computer system obtains geometric information of the scene object from multiple two-dimensional images with different viewpoints. Then, the computer system creates a depth map based on the obtained geometric information for creating and rendering a three-dimensional scene on the two-dimensional display.

[0005] A depth map is an image that contains information about the distance from the imaging viewpoint that captures the two-dimensional image of the scene object to the surface of the scene object. This depth is sometimes called Z-depth, which refers to the convention that the central axis of the imager viewpoint is in the direction of the imager's Z axis, rather than in the absolute Z axis of the scene.

[0006] The computer system presents a three-dimensional scene on a two-dimensional display for viewing and manipulation by a user, and the user is able to manipulate scene objects of the three-dimensional scene by changing the viewpoint of the three-dimensional scene. For the viewpoint of the scene objects, the three-dimensional scene does not include accurate information (e.g., color values, depth values, and / or object values, because some faces of the object / scene do not appear in one or more two-dimensional images obtained from the limited number of viewpoints used to create the three-dimensional scene), and the computer system attempts to use information from neighboring pixels to complete the scene. When the computer system attempts to complete the scene (e.g., using information from neighboring pixels), the resulting scene often includes unrealistic looking shapes and / or colors (e.g., a "stretching" effect of color).

[0007] In order to minimize unrealistic appearance of shapes and / or colors, prior art often obtains far more than two 2D images from two different viewpoints (e.g., a panorama of an image). Obtaining a panorama of an image from multiple viewpoints increases the likelihood that at least one viewpoint of a scene object includes information for generating a three-dimensional scene from a variety of viewpoints, thereby improving display accuracy. However, this technique requires a relatively large amount of processing time / power compared to creating a three-dimensional scene from only two two-dimensional images.

[0008] BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The present invention will be best understood from the following detailed description when read in conjunction with the accompanying drawings, in which like components have like reference numerals. When there are multiple similar components, a single reference numeral may be assigned to multiple similar components, and lower case letters may be used to indicate specific components. When referring to a collection of components or to a non-specific component or components, lower case letters may be omitted. This emphasizes that, according to convention, unless otherwise indicated, the various features of the drawings are not drawn to scale. On the contrary, the dimensions of various features may be enlarged or reduced for clarity. The following drawings are included in the drawings:

[0010] Figure 1A is a perspective view of an example of eyewear including an electronic component and a support structure supporting the electronic component;

[0011] Figure 1B yes Figure 1A a top view of an example of an eyeglass, showing the area defined by the eyeglass for accommodating the head of a user wearing the eyeglass;

[0012] Figure 2 yes Figure 1A Block diagram of examples of electronic components supported by an example of glasses, and communications with a personal computing device and a recipient.

[0013] Figure 3A is a flowchart of example steps for rendering a 3D scene;

[0014] Figure 3B is used to reconstruct the first and second 2D images into Figure 3A A flowchart of example steps for a 3D scene in FIG.

[0015] Figure 3C is used for Figure 3B A flowchart of example steps for detecting regions of a 3D scene with incomplete information; and

[0016] Figure 3D It is used to determine Figure 3B A flowchart of example steps for replacing image information in FIG. DETAILED DESCRIPTION

[0017] In the following detailed description, many specific details are set forth by way of example in order to provide a thorough understanding of the relevant teachings. However, it will be apparent to those skilled in the art that these details are not necessary for practicing the present teachings. In other cases, detailed, relatively high-level descriptions of well-known methods, processes, components, and circuits are omitted to avoid unnecessarily obscuring various aspects of the present teachings.

[0018] The term "coupled" as used herein refers to any logical, optical, physical, electrical connection, link or the like, whereby a signal or light generated or provided by one system element is transferred to another coupled element. Unless otherwise specified, coupled elements or devices do not necessarily have to be in physical contact with each other and may be separated by airspace, intermediate components, elements or communication media that may modify, manipulate or carry light or signals.

[0019] The orientation of the glasses, related components, and any devices shown in any of the figures is exemplary only for purposes of illustration and discussion. In operation, the glasses may be oriented in other directions suitable for the particular application of the glasses, such as upward, downward, sideways, or any other direction. Likewise, any directional terms, such as front, back, inward, outward, toward, left, right, lateral, longitudinal, up, down, upper, lower, top, bottom, and side, are exemplary only and not limiting with respect to directions or orientations.

[0020] Figure 1A A front perspective view of an example pair of glasses 12 for capturing images is depicted. The illustrated pair of glasses 12 includes a support structure 13 having temples 14A and 14B extending from a central frame portion 16. The glasses 12 also include hinged joints 18A and 18B, electronic components 20A and 20B, and cores 22A, 22B, and 24. Although the illustrated pair of glasses is in the form of glasses, the glasses may take other forms, such as a headset, a headgear, a helmet, or other device that a user may wear.

[0021] When worn by a user, the support structure 13 supports one or more optical elements within the user's field of view. For example, the central frame portion 16 supports one or more optical elements. The term "optical element" as described herein refers to a lens, a transparent sheet of glass or plastic, a projector, a screen, a display, and other devices for presenting visual images or for users to perceive visual images. In one example, the corresponding temples 14A and 14B are connected to the central frame portion 16 at corresponding articulated joints 18A and 18B. The temples 14A and 14B shown are elongated members having core wires 22A and 22B extending longitudinally therein.

[0022] exist Figure 1A In the embodiment, the temple 14A is shown in a wearable state, and the temple 14B is shown in a folded state. Figure 1A As shown, hinged joint 18A connects temple 14A to right end 26A of central frame portion 16. Similarly, hinged joint 18B connects temple 14B to left end 26B of central frame portion 16. Right end 26A of central frame portion 16 includes a housing for accommodating electronic assembly 20A therein, and left end 26B includes a housing for accommodating electronic assembly 20B therein.

[0023] The plastic or other material is embedded in a core wire 22A that extends longitudinally from the adjacent hinge joint 18A toward the second longitudinal end of the temple 14A. Similarly, the plastic or other material is embedded in a core wire 22B that extends longitudinally from the adjacent hinge joint 18B toward the second longitudinal end of the temple 14B. The plastic or other material is additionally embedded in a core wire 24 that extends from the right end 26A (terminating at the adjacent electronic component 20A) to the left end 26B (terminating at the adjacent electronic component 20B).

[0024] Electronic assemblies 20A and 20B are carried by support structure 13 (e.g., by one or both of temples 14A, 14B and / or central frame portion 16). Electronic assemblies 20A and 20B include power supplies, circuits related to power supplies and communications, communication devices, display devices, computers, memories, modules, and / or the like (not shown). Electronic assemblies 20A and 20B may each include a corresponding imager 10A and 10B for capturing images and / or videos. In the example shown, imager 10A is adjacent to right temple 14A, and imager 10B is adjacent to left temple 14B. Imagers 10A and 10B are spaced apart from each other so as to obtain images of scene objects from two different viewpoints for generating a 3D scene.

[0025] The support structure 13 defines an area for accommodating a portion 52 (e.g., a major portion) of a user / wearer's head (e.g., the area 52 (e.g., defined by the frame 12 and the temples 14A and 14B)). Figure 1B )). The defined area is one or more areas that include at least a portion of the user's head, which is surrounded by, surrounds, is adjacent to, and / or is near the support structure when the user wears the glasses 12. In the example shown, imagers 14A and 14B are located on the glasses so that when the glasses 12 are worn, imagers 14A and 14B are adjacent to the user's respective eyes, which helps to obtain a separation of viewpoints suitable for creating a three-dimensional scene.

[0026] Figure 21 is a block diagram of example electronic components coupled to a display system 135 (e.g., a display of a processing device or other technology for presenting information). The electronic components shown include a controller 100 (e.g., a hardware processor) for controlling various devices in the glasses 12; a wireless module 102 (e.g., Bluetooth™) for facilitating communication between the glasses 12 and a client device (e.g., a personal computing device 50 such as a smartphone); a power circuit 104 (e.g., a battery, a filter, etc.) for powering the glasses 12; a memory 106 such as flash memory for storing data (e.g., images, videos, image processing software, etc.); a selector 32; and one or more imagers 10 (two in the example) for capturing one or more images (e.g., pictures or videos). Although the glasses 12 and the personal computing device are shown as separate components, the functionality of the personal computing device can be incorporated into the glasses, thereby enabling the personal computing device and / or the glasses 12 to perform the functions described herein.

[0027] The selector 32 can trigger (e.g., by momentarily pressing a button) the controller 100 of the glasses 12 to capture an image / video. In an example using a single selector 32, the selector can be used in both a setup mode (e.g., entered by pressing and holding the selector 32 for a period of time, such as 3 seconds) and an image capture mode (e.g., entered after a period of no contact, such as 5 seconds) to capture an image.

[0028] In one example, the selector 32 can be a physical button on the glasses 12 that, when pressed, sends a user input signal to the controller 100. The controller 100 can interpret pressing a button within a predetermined time period (e.g., three seconds) as a request to switch to a different operating mode (e.g., enter / exit an operating settings mode). In other examples, the selector 32 can be a virtual button on the glasses or another device. In yet another example, the selector can be a voice module that interprets voice commands, or an eye detection module that detects where the focus of the eye is directed. The controller 100 can interpret the signal from the selector 32 as a trigger to cycle through the LED 35 to select the intended recipient of the image.

[0029] The wireless module 102 can be coupled to a client / personal computing device 50, such as a smartphone, tablet, phablet, laptop, desktop computer, networking device, access point device, or any other such device that can be connected to the wireless module 102. Bluetooth, Bluetooth LE, Wi-Fi, Wi-Fi Direct, cellular modem, and near field communication systems, as well as multiple examples of any of these systems, can enable these connections to enable communication therebetween. For example, communication between devices can facilitate the transmission of software updates, images, videos, lighting schemes, and / or sounds between the glasses 12 and the client device.

[0030] In addition, personal computing device 50 can communicate with one or more recipients (e.g., recipient personal computing device 51) via network 53. Network 53 can be a cellular network, Wi-Fi, the Internet, or a similar network that allows personal computing devices to transmit and receive images, such as via text, email, instant messaging, etc. Computing devices 50 / 51 can each include a processor and a display. Suitable processors and displays, which can be configured to perform another function described herein, can be found in this generation of personal computing devices and smartphones, such as the iPhone 8 available from Apple Inc. of Cupertino, California. TM and the Samsung Galaxy Note 9 available from Samsung Group in Seoul, South Korea TM .

[0031] The imager 10 for capturing images / video may include digital camera elements such as a charge coupled device, a lens, or any other light capturing element for capturing image data for conversion into electrical signals.

[0032] The controller 100 controls the electronics. For example, the controller 100 includes circuits that receive signals from the imager 10 and process those signals into a format suitable for storage in a memory 106 (e.g., flash memory). The controller 100 powers up and boots into operation in a normal operating mode or into a sleep mode. In one example, the controller 100 includes a microprocessor integrated circuit (IC) customized to process sensor data from the imager 10, and volatile memory that the microprocessor uses for operation. The memory can store software code executed by the controller 100.

[0033] Each electronic component requires power to work. The power supply circuit 104 may include a battery, a power converter, and a power distribution circuit (not shown). The battery may be a rechargeable battery, such as a lithium ion or the like. The power converter and the power distribution circuit may include electronic components for filtering and / or converting voltages for powering various electronic devices.

[0034] Figure 3A A flowchart 300 is depicted, which illustrates example operations of eyewear (e.g., eyewear 12 of FIG. 1 )-side three-dimensional scene rendering performed by a processing system (e.g., by a processor of the eyewear 12 and / or a processor of a computing device remotely connected to the eyewear). For ease of explanation, the steps of flowchart 300 are described with reference to the eyewear 12 described herein, and those skilled in the art will recognize that other imager configurations not limited to the eyewear are used to render the three-dimensional scene. In addition, it is understood that one or more steps may be omitted, performed by another component, or performed in a different order.

[0035] At step 310, a first two-dimensional image of a scene object is obtained from a first viewpoint, and a second two-dimensional image of the scene object is obtained from a second viewpoint. In one example, a first imager 10A of glasses 12 captures a first two-dimensional image of a scene object from a first viewpoint, and a second imager 10B of glasses 12 captures a second two-dimensional image of a scene object from a second viewpoint. The captured images are transferred from imagers 10A and 10B to a processing system for rendering a three-dimensional scene. In one example, a controller 100 of glasses 12 obtains the two-dimensional image and renders a three-dimensional scene from the obtained two-dimensional image. In another example, the controller 100 receives the two-dimensional images and sends them to a processing system of a remote computing device 50 / 51 to render a three-dimensional scene.

[0036] In step 320, the first and second two-dimensional images are reconstructed into a three-dimensional scene of the scene object. The processing system reconstructs the first and second two-dimensional images into a three-dimensional scene of the scene object. The processing system can use stereoscopic vision processing technology (i.e., creating a depth map) and geometric processing technology to create a three-dimensional scene. In one example, the rendered three-dimensional scene includes geometric features (e.g., vertices with x-axis, y-axis, and z-axis coordinates) and image information (e.g., color information, depth information, and object information). The rendered three-dimensional scene will also include polygonal faces of a connected body, such as typical triangular faces or quadrilateral faces, which connect vertices to form a textured surface of the scene object. Based on the description herein, those skilled in the art will understand appropriate stereoscopic vision processing technology and geometric processing technology.

[0037] Figure 3B A flowchart depicting exemplary steps for reconstructing the first and second two-dimensional images into a three-dimensional scene of scene objects (step 320; Figure 3A ). In step 321, a depth map is created from the first and second two-dimensional images. The processing system may create the depth map by processing the first and second two-dimensional images using stereoscopic vision processing techniques. In step 322, a three-dimensional scene is created from the first and second two-dimensional images. The processing system may create the three-dimensional scene by geometrically processing the first and second two-dimensional images and the depth map created in step 321.

[0038] At step 323, regions of the three-dimensional scene having incomplete image information are detected. The processing system may detect regions of the three-dimensional scene having incomplete image information (e.g., lacking color, depth, and / or object information). For example, the processing system may determine incomplete information by examining the shapes of faces constituting the three-dimensional scene and / or information such as confidence values ​​associated with vertices constituting the faces.

[0039] In one example, the processing system processes the faces of the reconstructed three-dimensional scene (step 323a; Figure 3C) to identify groups of consecutive faces that exhibit incomplete information, such as very narrow faces and / or faces with one or more vertices with low confidence values.

[0040] The processing system may identify adjacent regions having degenerate surfaces (e.g., relatively narrow surfaces) as regions having incomplete information (step 323b; Figure 3C ). Relatively narrow facets can be determined by comparing the angles between adjacent lines of these facets to a threshold value, for example, a facet comprising at least one angle less than a threshold value such as 5 degrees, a length of one side of the facet being less than 5% of the length of the other side, and / or a side of the facet having a dimension less than a threshold value such as 1 mm, can be classified as narrow.

[0041] The processing system may also or alternatively identify faces with low confidence values ​​(e.g., faces having at least one vertex with a confidence value below a threshold) as regions with incomplete information (step 323c; Figure 3C ). The confidence value of a vertex may be the confidence value of the corresponding pixel determined during stereo processing of the first and second two-dimensional images to create a depth map (as described above).

[0042] The confidence value of the vertex corresponding to the pixel depends on the match / correlation between the first and second two-dimensional images when the pixel was created. If there is a high correlation (e.g., 75% or higher), there is a relatively high probability that the vertex contains accurate information useful for reconstructing a three-dimensional scene. On the other hand, if the correlation is low (e.g., less than 75%), there is a relatively high probability that the vertex does not contain information accurate enough to be used to reconstruct the three-dimensional scene.

[0043] Reconstruct the detected area in step 324. In one example, the processing system reconstructs the detected area of ​​the three-dimensional scene, for example using geometric processing such as described in step 322 above. When reconstructing the detected area, the processing system can ignore vertices with low confidence values ​​and / or degenerate faces associated therewith. If any, this results in fewer degenerate faces (e.g., relatively narrow faces). Therefore, after this reconstruction step, the faces in the detected area will have different shapes. The faces in the detected area can be removed before reconstruction. In one example, a data structure such as an Indexed-Face-Set for a three-dimensional mesh can be used, which includes an ordered list of all vertices (and their attributes, such as color, texture, etc.) and a face list, where each face points to a vertex index in the vertex list. In this example, a face can be removed by removing it from the face list.

[0044] In step 325, replacement image information is determined and the reconstructed detected regions are corrected. The processing system may determine replacement image information for the reconstructed detected regions and adjust the three-dimensional scene to include the replacement image information in the detected regions when the replacement image information is reconstructed. For example, the processing system may determine replacement image information for each detected region by blending boundary information from respective boundaries of each detected region.

[0045] In one example, to determine replacement image information, the processing system identifies a boundary around each detected region (step 325a; Figure 3D Then, the processing system identifies background information in the detected area (step 325b; Figure 3D ); for example, based on depth information associated with vertices along the boundary. The processing system also identifies foreground information in the detected region (step 325c; Figure 3D ); and, for example, also based on depth information. To identify background / foreground information, the processing system can compare the depth information of each vertex with a threshold (e.g., an average of the depth information from all vertices), identify information associated with vertices having a depth greater than the threshold as background information, and identify information associated with vertices having a depth less than the threshold as foreground information. The processing system then fuses information from the boundaries into the respective regions (step 325d; Figure 3D ), thereby giving higher weight to the background information than the foreground information, which causes the information to mainly diffuse from the background to the foreground.

[0046] Return to reference Figure 3A , in step 330, the three-dimensional scene is rendered from multiple viewpoints. The processing system can render the three-dimensional scene from multiple viewpoints, for example, by applying image synthesis techniques to the three-dimensional scene to create a two-dimensional image from each viewpoint. Based on the description herein, those skilled in the art will understand suitable image synthesis techniques.

[0047] At step 340, the rendered three-dimensional scene is optimized. The processing system can improve the three-dimensional scene from each of the plurality of viewpoints. In one example, the processing system identifies areas in the two-dimensional image of the rendered three-dimensional scene where there are gaps in the image information (i.e., "holes"), and then the processing system fills the holes with replacement image information surrounding the holes. The processing system can fill the holes, giving priority to background information around the holes.

[0048] The rendered three-dimensional scene is presented at step 350. The processing system may present the rendered three-dimensional scene on a display of the eyewear or remote computing device by selectively presenting a two-dimensional image within the rendered three-dimensional scene associated with a selected viewpoint (e.g., based on user input to the eyewear device or remote computing device).

[0049] By performing the above process described with reference to flowchart 300, a more aesthetically pleasing 3D scene can be obtained from only two 2D images, viewable from more viewpoints (e.g., simplified, i.e., color stretched), without resorting to a panoramic view, and thus, superior results can be achieved without resorting to computationally intensive techniques.

[0050] It should be understood that the steps of the processes described herein may be performed by a hardware processor after loading and executing software code or instructions tangibly stored on a physical computer-readable medium (e.g., a magnetic medium, a computer hard drive, an optical disk, a solid-state memory, a flash memory, or other storage medium known in the art). Thus, any function performed by the processor described herein may be implemented as software code or instructions tangibly stored on a physical computer-readable medium. After the processor loads and executes such software code or instructions, the processor may perform any function described herein, including any steps of the methods described herein.

[0051] The term "software code" or "code" as used herein refers to any instructions or instruction sets that affect the operation of a computer or controller. They may exist in a computer executable form (e.g., machine code), which is a set of instructions and data that is directly executed by the central processing unit or controller of a computer, in a human understandable form (e.g., source code) that is executed by the central processing unit or controller of a computer, or in an intermediate form (e.g., object code) generated by a compiler for execution. As used herein, the term "software code" or "code" also includes any human understandable computer instructions or instruction sets, such as scripts, that can be executed on the fly with the help of an interpreter by the central processing unit or processor of a computer.

[0052] Although an overview of the subject matter of the present invention has been described with reference to specific embodiments, various modifications and changes may be made to these examples without departing from the broader scope of the examples of the present disclosure. For example, although the description focuses on eyewear devices, other electronic devices (such as headphones) are also considered to be within the scope of the present subject matter. For convenience only, the term "invention" may be used herein to refer to such examples of the inventive subject matter, either individually or collectively, and if there are more than one invention, it is not intended to automatically limit the scope of the present application to any single disclosure or inventive concept that is actually disclosed.

[0053] The examples shown herein are described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other examples may be used and derived therefrom, so that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Therefore, the detailed description should not be understood in a limiting sense, and the scope of the various examples is limited only by the appended claims and the full scope of equivalents to which the claims are entitled.

Claims

1. A system for creating a three-dimensional (3D) scene, the system comprising: Glasses comprising a first imager and a second imager spaced apart from the first imager, the first imager being configured to obtain a first two-dimensional (2D) image of a scene object from a first viewpoint, and the second imager being configured to obtain a second 2D image of the scene object from a second viewpoint different from the first viewpoint; A processing system coupled to the glasses, the processing system being configured to; obtaining the first two-dimensional image and the second two-dimensional image; creating a depth map based on the first and second two-dimensional images; creating a three-dimensional scene based on the depth map and the first and second two-dimensional images; Detecting regions with incomplete image information in the initial three-dimensional scene; reconstructing the detected region in the three-dimensional scene; Determine replacement information and correct reconstruction areas; as well as Rendering a 3D scene with a modified reconstruction region from multiple viewpoints, The depth map includes pixel vertices and confidence values ​​corresponding to each vertex, and in order to create the 3D scene, the processing system is configured to connect the vertices to form faces, and in order to detect the boundary of the 3D scene with incomplete information, the processing system is configured to identify continuous faces including at least one of degenerate faces or low-confidence faces.

2. The system according to claim 1, wherein: The processing system is further configured to: identifying holes in a rendered three-dimensional scene from one or more viewpoints; and The rendered three-dimensional scene is optimized to fill the hole.

3. The system according to claim 1, wherein: The eyeglasses include a first temple and a second temple, and wherein the first imager is adjacent to the first temple and the second imager is adjacent to the second temple.

4. The system according to claim 1, wherein: In order to determine replacement information for the detected area having incomplete image information, the processing system is configured to: identifying a boundary around each detected region; identifying contextual information within said boundaries; identifying foreground information within the boundary; and The background and foreground boundary information of each region are fused so that the weight of the background boundary information is higher than the weight of the foreground boundary information.

5. The system according to claim 4, wherein: In order to blend the background and foreground boundary information, the processing system is configured to: The missing information of the background boundary information is diffused into the foreground boundary information through each region.

6. The system according to claim 1, wherein: The depth map includes pixel vertices and corresponding image information from the first and second two-dimensional images. To create the 3D scene, the processing system is configured to connect the vertices to form a first surface. To reconstruct the detected area, the processing system is configured to connect the vertices of the boundary area to form a second surface, which is different from the first surface.

7. The system according to claim 6, wherein: The first and second faces include at least one of a triangular face or a quadrilateral face.

8. The system according to claim 1, wherein: Each of the degenerate surfaces has at least one angle that is smaller than a threshold value.

9. The system according to claim 1, wherein: Each low confidence surface includes at least one vertex generated with incompatible values ​​between the first two-dimensional image and the second two-dimensional image.

10. A method for creating a three-dimensional (3D) scene, the method comprising: obtaining a first two-dimensional (2D) image of a scene object from a first viewpoint; obtaining a second two-dimensional image of the scene object from a second viewpoint different from the first viewpoint; creating a depth map based on the first and second two-dimensional images; creating a three-dimensional scene based on the depth map and the first and second two-dimensional images; Detecting regions with incomplete image information in the initial three-dimensional scene; reconstructing the detected region in the three-dimensional scene; Determine replacement information and correct reconstruction areas; as well as Rendering a 3D scene with a modified reconstruction region from multiple viewpoints, The depth map includes pixel vertices and corresponding image information from the first and second two-dimensional images, and creating the three-dimensional scene includes connecting the vertices to form multi-angle faces, and wherein the detecting step includes: Continuous surfaces including at least one of a degenerate surface or a low confidence surface are identified.

11. The method according to claim 10, further comprising: identifying holes in a rendered three-dimensional scene from one or more viewpoints; and The rendered three-dimensional scene is optimized to fill the hole.

12. The method according to claim 10, wherein: The first two-dimensional image is obtained from a first imager adjacent to a first temple of the eyeglasses, and the second two-dimensional image is obtained from a second imager adjacent to a second temple of the eyeglasses.

13. The method according to claim 10, wherein: The determining step comprises: identifying a boundary around each detected region; identifying contextual information within said boundaries; identifying foreground information within the boundary; and The background and foreground boundary information of each region are fused so that the weight of the background boundary information is higher than the weight of the foreground boundary information.

14. The method according to claim 13, wherein: The fusion includes: The missing information of the background boundary information is diffused into the foreground boundary information through each region.

15. The method according to claim 10, wherein: The step of creating a three-dimensional scene comprises: creating a depth map including pixel vertices and corresponding image information from the first and second two-dimensional images, and connecting the vertices to form a first surface, and wherein the reconstruction step further comprises: The vertices of the boundary region are connected to form a second face, the second face being different from the first face.

16. The method according to claim 15, wherein: The first and second faces include at least one of a triangular face or a quadrilateral face.

17. The method according to claim 10, wherein: Each of the degenerate surfaces has at least one angle that is smaller than a threshold value.

18. The method according to claim 10, wherein: Each low confidence face includes at least one vertex generated by an incompatibility value between the first two-dimensional image and the second two-dimensional image exceeding a threshold.

Images