Edge detection and smoothing for virtual object presentation

Abstract

Image processing techniques for rendering holographic images are described. Image data captured from a camera can be converted into a triangular mesh. The triangular mesh may be processed by determining edges, normal locations, and edge distances within the mesh. Mesh and edge smoothing can be performed on the mesh to reduce noise and jagged edges. Additional mesh cleanup and simplification can be performed to reduce the number of vertices and triangles and improve the quality of the mesh. Data from the resulting mesh can be transmitted to a receiving device, where the mesh data is used to recreate a holographic image of the image data captured by the depth sensor.

Description

[Technical field]

[0001] The present specification relates generally to image processing in virtual or augmented reality systems. [Background technology]

[0002] Virtual or augmented reality (VAR) systems can be used to display virtual content and augment the physical reality view. However, three-dimensional (3-D) holographic images rendered by VAR display systems can suffer from poor quality due to noise and impaired volumetric reconstruction. Summary of the Invention [Means for solving the problem]

[0003] An innovative aspect of the subject matter described herein relates to improved volumetric reconstruction of images using improved edge detection and smoothing techniques.

[0004] Innovative aspects of the subject matter described herein include a computer-implemented method that, in some implementations, comprises a number of operations: acquiring image data from a pair of cameras, determining a triangular mesh from the image data, performing mesh smoothing and edge smoothing to remove noise from the triangular mesh, removing one or more vertices from the mesh that do not satisfy a contribution threshold, storing in a memory mesh data that characterizes the mesh after removal of the one or more vertices from the mesh, and transmitting the mesh data to a rendering device configured to render a holographic image based on the mesh data.

[0005] Each implementation may optionally include one or more of the following features. For example, in some implementations, the computer-implemented method includes an operation of determining edges for a triangle. This can be accomplished by performing a set of operations for each vertex in the mesh. The set of operations includes determining a neighboring vertex that is located one vertex away from the vertex, determining whether the neighboring vertex and the vertex share a common triangle, determining that the neighboring vertex and the vertex are interior nodes of the mesh when the neighboring vertex and the vertex are shared by at least two triangles, and determining that the neighboring vertex and the vertex are exterior nodes of the mesh when the neighboring vertex and the vertex are part of a triangle.

[0006] In some implementations, the computer-implemented method includes determining, for each interior node, a distance to an edge of the mesh, the distance being the number of hops it takes to reach the edge from the interior node.

[0007] In some implementations, the operations for performing mesh smoothing and edge smoothing include performing a set of operations for each vertex, the set of operations including determining an average location of vertices that share a triangle with the vertex, and modifying the location of the vertex to more closely approximate the average location of vertices that share the triangle with the vertex.

[0008] In some implementations, the operations of performing mesh smoothing and edge smoothing include identifying vertices that are located a determined number of hops from a particular edge of the mesh, identifying other vertices that share a triangle and are located a determined number of hops from the particular edge, determining an average location of the other vertices, and modifying the locations of the vertices so that they are closer to the average location.

[0009] In some implementations, the computer-implemented method includes determining that two vertices are separated by a distance that is less than a threshold distance, and merging the two vertices.

[0010] In some implementations, the operation of removing one or more vertices from the mesh that do not satisfy the contribution threshold includes identifying a pair of vertices that belong to the same triangle, determining the impact on the mesh if the pair of vertices is merged, and determining that the impact on the mesh is less than the contribution threshold.

[0011] In some implementations, the operation of determining the effect on the mesh when a pair of vertices is merged includes determining the change in angles between all neighboring triangles before and after the pair of vertices is merged.

[0012] In some implementations, the computer-implemented method includes removing one or more triangles from the mesh prior to removing the one or more vertices by removing one or more triangles that form sub-meshes within the mesh.

[0013] In some implementations, the mesh data includes data indicating color in one or more images and depth from a pair of cameras to pixels in one or more images.

[0014] Aspects of the subject matter described herein also include, in some implementations, a virtual or augmented reality system including a frame structure, a computing device, a memory, and a headset. The frame structure includes a pair of depth sensors configured to acquire image data. The computing device is coupled to the frame structure. The computing device is configured to acquire image data from the pair of depth sensors, determine a triangular mesh from the image data, perform mesh smoothing and edge smoothing, remove noise from the triangular mesh, and remove one or more vertices from the mesh that do not satisfy a contribution threshold. The memory is configured to store mesh data that characterizes the mesh after removing one or more vertices from the mesh. The headset is configured to render a holographic image based on the mesh data.

[0015] Aspects of the subject matter described herein also include, in some implementations, a non-transitory computer-readable storage medium including instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations including acquiring image data from a pair of cameras, determining a triangular mesh from the image data, performing mesh smoothing and edge smoothing to remove noise from the triangular mesh, removing one or more vertices from the mesh that do not satisfy a contribution threshold, storing in a memory the mesh data that characterizes the mesh after removal of the one or more vertices from the mesh, and transmitting the mesh data to a rendering device configured to render a holographic image based on the mesh data.

[0016] Other implementations of the present aspects include corresponding systems, apparatus, and computer programs recorded on computer storage devices, respectively, configured to perform the operations of the present methods.

[0017] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. The present invention provides, for example, the following: (Item 1) 1. A computer-implemented method, comprising: acquiring image data from a pair of cameras; determining a mesh of triangles from the image data; performing mesh smoothing and edge smoothing to remove noise from said triangle mesh; removing one or more vertices from said mesh that do not satisfy a contribution threshold; storing in memory mesh data characterizing the mesh after removal of the one or more vertices from the mesh; transmitting the mesh data to a rendering device configured to render a holographic image based on the mesh data; 4. A computer-implemented method comprising: (Item 2) For each vertex in the mesh, determining a neighboring vertex located one vertex away from the vertex; determining whether the neighboring vertices and the vertex share a common triangle; determining that the neighboring vertices and the vertices are interior nodes of the mesh when the neighboring vertices and the vertices are shared by at least two triangles; determining that the neighboring vertices and the vertices are exterior nodes of the mesh when the neighboring vertices and the vertices are part of a triangle; 2. The computer-implemented method of claim 1, further comprising determining an edge for the triangle by: (Item 3) 3. The computer-implemented method of claim 2, further comprising: for each interior node, determining a distance to an edge of the mesh, the distance being the number of hops from the interior node to reach the edge. (Item 4) Performing mesh smoothing and edge smoothing includes: For each vertex, determining an average location of vertices that share a triangle with said vertex; modifying the location of said vertex to be closer to the average location of vertices that share a triangle with said vertex; 2. The computer-implemented method of claim 1, comprising: (Item 5) Performing mesh smoothing and edge smoothing includes: identifying a vertex that is a determined number of hops from a particular edge of the mesh; identifying other vertices that share the triangle and are located the determined number of hops from the particular edge; determining an average location of said other vertices; modifying the locations of the vertices so that the locations of the vertices are closer to the average location; 2. The computer-implemented method of claim 1, comprising: (Item 6) determining that two vertices are separated by a distance that is less than a threshold distance; Merging the two vertices; 2. The computer-implemented method of claim 1, comprising: (Item 7) Removing one or more vertices from the mesh that do not satisfy the contribution threshold includes: Identifying a pair of vertices that belong to the same triangle; determining an effect on the mesh if the pair of vertices are merged; determining that the contribution to the mesh is less than the contribution threshold; 2. The computer-implemented method of claim 1, comprising: (Item 8) 8. The computer-implemented method of claim 7, wherein determining an effect on the mesh when the pair of vertices is merged includes determining a change in angle between all neighboring triangles before and after the pair of vertices is merged. (Item 9) 2. The computer-implemented method of claim 1, further comprising removing one or more triangles from the mesh prior to removing the one or more vertices by removing one or more triangles that form a sub-mesh within the mesh. (Item 10) Item 1, wherein the mesh data comprises data indicative of color in one or more images and depth from the pair of cameras to pixels in the one or more images. (Item 11) 1. A virtual or augmented reality system, comprising: a frame structure comprising a pair of depth sensors configured to acquire image data; A computing device coupled to the frame structure, the computing device comprising: acquiring image data from the pair of depth sensors; determining a mesh of triangles from the image data; performing mesh smoothing and edge smoothing to remove noise from said triangle mesh; removing one or more vertices from said mesh that do not satisfy a contribution threshold; A computing device configured to: a memory configured to store mesh data characterizing the mesh after removing the one or more vertices from the mesh; and a headset configured to render a holographic image based on the mesh data; A virtual or augmented reality system comprising: (Item 12) Item 12. The virtual or augmented reality system of item 11, wherein the computing device is configured to transmit the mesh data to one or more rendering devices configured to render the holographic image based on the mesh data, the one or more rendering devices comprising the headset. (Item 13) To determine edges for the triangles, the computing device comprises: For each vertex in the mesh, determining a neighboring vertex located one vertex away from the vertex; determining whether the neighboring vertices and the vertex share a common triangle; determining that the neighboring vertices and the vertices are interior nodes of the mesh when the neighboring vertices and the vertices are shared by at least two triangles; determining that the neighboring vertices and the vertices are exterior nodes of the mesh when the neighboring vertices and the vertices are part of a triangle; Item 12. The virtual or augmented reality system of item 11, configured to: (Item 14) To perform the mesh smoothing and edge smoothing, the computing device comprises: For each vertex, determining an average location of vertices that share a triangle with said vertex; modifying the location of said vertex to be closer to the average location of vertices that share a triangle with said vertex; Item 12. The virtual or augmented reality system of item 11, configured to: (Item 15) To perform the mesh smoothing and edge smoothing, the computing device comprises: identifying a vertex that is a determined number of hops from a particular edge of the mesh; identifying other vertices that share the triangle and are located the determined number of hops from the particular edge; determining an average location of said other vertices; modifying the locations of the vertices so that the locations of the vertices are closer to the average location; Item 12. The virtual or augmented reality system of item 11, configured to: (Item 16) A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more computing devices, cause the one or more computing devices to: acquiring image data from a pair of cameras; determining a mesh of triangles from the image data; performing mesh smoothing and edge smoothing to remove noise from said triangle mesh; removing one or more vertices from said mesh that do not satisfy a contribution threshold; storing in memory mesh data characterizing the mesh after removal of the one or more vertices from the mesh; transmitting the mesh data to a rendering device configured to render a holographic image based on the mesh data; A non-transitory computer readable storage medium for performing operations including: (Item 17) The one or more computing devices include: For each vertex in the mesh, determining a neighboring vertex located one vertex away from the vertex; determining whether the neighboring vertices and the vertex share a common triangle; determining that the neighboring vertices and the vertices are interior nodes of the mesh when the neighboring vertices and the vertices are shared by at least two triangles; determining that the neighboring vertices and the vertices are exterior nodes of the mesh when the neighboring vertices and the vertices are part of a triangle; Item 17. The non-transitory computer-readable storage medium of item 16, configured to determine edges for the triangles by: (Item 18) Performing mesh smoothing and edge smoothing includes: For each vertex, determining an average location of vertices that share a triangle with said vertex; modifying the location of said vertex to be closer to the average location of vertices that share a triangle with said vertex; 17. The non-transitory computer-readable storage medium of item 16, comprising: (Item 19) Performing mesh smoothing and edge smoothing includes: identifying a vertex that is a determined number of hops from a particular edge of the mesh; identifying other vertices that share the triangle and are located the determined number of hops from the particular edge; determining an average location of said other vertices; modifying the locations of the vertices so that the locations of the vertices are closer to the average location; 17. The non-transitory computer-readable storage medium of item 16, comprising: (Item 20) Removing one or more vertices from the mesh that do not satisfy the contribution threshold includes: Identifying a pair of vertices that belong to the same triangle; determining an effect on the mesh if the pair of vertices are merged; determining whether the contribution to the mesh is less than the contribution threshold by determining a change in angle between all neighboring triangles before and after the pair of vertices is merged; 17. The non-transitory computer-readable storage medium of item 16, comprising: [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 illustrates an example of a virtual or augmented reality (VAR) system having a dual sensor platform.

[0019] [Diagram 2]FIG. 2 depicts an example method for 3D holographic image reconstruction and its location in the pipeline for edge detection and smoothing in VAR images.

[0020] [Diagram 3] FIG. 3 illustrates a unique case of the MarchingCubes algorithm that can be used to generate a triangular mesh.

[0021] [Figure 4] FIG. 4 depicts a portion of an example triangular mesh.

[0022] [Diagram 5] 5A, 5B, 5C, and 5D depict examples of edge smoothing.

[0023] [Figure 6] 6A and 6B depict an example of edge smoothing.

[0024] [Figure 7] 7A and 7B depict an example of mesh simplification.

[0025] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Detailed Description Holograms are part of an immersive, visually collaborative experience that allows various participants to virtually interact with one another. Instead of talking to a screen, users can manipulate virtual content in a shared digital space and engage in virtual activities, such as playing virtual chess with another user. Holographic technology and VAR systems have applications in several areas, such as electronic gaming, online education, virtual assistance, shared work environments, telepresence, and remote collaboration.

[0027] VAR systems often use holographic projections to display images and content to users, but VAR systems struggle to effectively enable user interaction with the holographic projections. Constraints such as the inability of networks or computing devices to handle large amounts of image data affect the ability to stream and render satisfactory holographic projections in real time.

[0028] Even in some systems where sufficient system resources are available, the quality of holographic projections is still low due to poor holographic rendering. For example, holographic images may appear wavy or jagged due to noise artifacts present at the edges of objects such as people. Such quality issues not only adversely affect a user's viewing experience, but they also make it difficult to interact with holographic projections because object boundaries are not clearly defined or visible. The following disclosure describes systems and methods for addressing this problem and improving the user experience of interacting with holographic images and displays.

[0029] As discussed in more detail herein below, image data may be captured from the depth sensor and a triangular mesh may be generated from the image data. The triangular mesh may be processed by determining edges, normal locations, and edge distances within the mesh. Using this information, mesh and edge smoothing may be performed to reduce noise and jagged edges. Additional operations such as texture projection and mesh cleanup and simplification to reduce the number of vertices and triangles may also be performed to improve the quality of the mesh. Data from the resulting mesh may be transmitted to a receiving device where the mesh data is used to recreate a holographic image of the image data captured by the depth sensor.

[0030] Advantageously, the methods of edge detection and mesh smoothing described in this disclosure can be performed with very low latency, thereby enabling real-time rendering of streaming image data. In addition, the methods and systems described in this disclosure can be easily integrated into new and existing applications to enhance user experience across a variety of platforms.

[0031] 1 depicts an exemplary VAR system 100 having a dual sensor platform. The system 100 includes a cloud platform 150, at least two computing devices 110, 160, at least two headsets 140, 190, and at least two sensor mounts 120A, 120B, 170A, 170B, and can render holographic images for users 130, 180. The two users 130, 180 can communicate with each other through the VAR system 100.

[0032] The user 130 can wear a headset 140 and can utilize a computing device 110 to interact with the VAR system 100. The computing device 110 can be connected to one or more depth sensors 120A and 120B, which are combined and form part of a sensor mount. The computing device 110 can also be connected to the headset 140 in a wired or wireless manner. Similarly, the user 180 can wear a headset 190 and can utilize a computing device 160 to interact with the VAR system 100. The computing device 160 can be connected to two depth sensors 170A and 170B, which are combined and form part of another sensor mount. The computing device 160 can also be connected to the headset 190 in a wired or wireless manner.

[0033] Headsets 140 and 190 can be mounted on a user's head 130 and 180, respectively. Each headset 140, 190 can include a stereoscopic head-mounted display to provide separate images for each eye, stereo sound, and head movement tracking sensors, such as gyroscopes, accelerometers, and / or magnetometers. In some implementations, headsets 140, 190 can also include eye movement tracking sensors.

[0034] Each headset 140, 190 can include a pair of lenses for mapping the display to a wide field of view. The headsets 140, 190 can include one or more cameras to capture real-world images that are displayed in the augmented view provided to the user 130, 180 along with the virtual objects. In some implementations, the headsets 140, 190 can include or be coupled to one or more sensors and processors to perform calibration to properly align the virtual objects and real-world images.

[0035] Headsets 140, 190 can also be used to view holographic images. For example, image data captured by depth sensors 120A and 120B can be transmitted to render a holographic image that is viewable through headset 190 worn by user 180. Similarly, image data captured by depth sensors 170A and 170B can be transmitted to render a holographic image that is viewable through headset 140 worn by user 130.

[0036] Each computing device 110, 160 can be connected to a sensor mount, which can include one or more depth sensors. For example, computing device 110 is connected to depth sensors 120A and 120B, and computing device 160 is connected to depth sensors 170A and 170B. Each depth sensor 120A, 120B, 170A, 170B can capture a stream of red, green, blue (RGB) images and data to generate a depth map. By combining data from one or more depth sensors or cameras in each sensor mount, the data acquired by the sensor mount can be used to generate a three-dimensional (3D) image of user 130 or 180. These virtual representations of user 130 or 180 can be transmitted to its counterpart, matched with virtual objects, and combined with real-world images.

[0037] Computing device 110 may be the same as or similar to computing device 160, however computing device 110 is associated with user 130 and computing device 160 is associated with user 180. In particular, computing device 110 may receive data from and / or transmit data to user 130, headset 140, depth sensors 120A, 120B, and cloud platform 150. Computing device 160 may receive data from and / or transmit data to user 180, headset 190, depth sensors 170A, 170B, and cloud platform 150. In some implementations, computing devices 110, 180 may each be a computer, such as a laptop or desktop, and may be configured to process large amounts of graphical data.

[0038] Each of the computing devices 110 and 160 may include one or more processors, memory, a user interface (e.g., a display, mouse, microphone, speaker), a communication interface, and a transceiver, among other components. For example, each computing device 110 / 160 may include one or more graphical processing units (GPUs) to process large amounts of raw image data obtained from the depth sensors 120A, 120B, 170A, 170B into smooth holograms that can be compressed to bandwidths comparable to streaming high-definition (HD) video.

[0039] In particular, the GPU in computing device 110 or 160 may generate a sensor depth map for images captured by depth sensors 120A, 120B, 170A, 170B, determine a 3D space that matches the sensor depth map, determine a smooth mesh geometry in the matched 3D space, perform simplification of the geometry, reduce the amount of graphical data points, project a color image onto the geometry as a texture, and blend the images at overlapping edges. These operations are described in more detail with respect to FIG.

[0040] Image set 115, including images 115A and 115B, provides an example of the type of image processing performed by the GPU. For example, images of a person can be captured by depth sensors 120A and 120B. A triangular mesh can be generated from the images and edges within the mesh can be determined, as shown in image 115A. The mesh and edges can be smoothed and simplified (A1), and then texture and color information can be added to the mesh, resulting in an image that may resemble image 115B. Data from image 115B can be transmitted (A2) through cloud platform 150 to computing device 160 and headset 190 associated with user 180, such that a holographic image 115C based on data from image 115B can be rendered for viewing by user 180 (A3). As can be seen in image 115C, image 115B of the person is superimposed on an image of the environment surrounding the viewer or user 180. The mesh is shown in image 115C for emphasis, but after rendering is complete, the mesh may not be visible to user 180 through headset 190.

[0041] The GPU within computing device 110 may stream image data at high data rates in the manner described above, enabling seamless viewing of the holographic image by user 180.

[0042] The transceivers in the computing devices 110 and 160 include transmitters and receivers for communicating with other devices, such as the depth sensors 120A, 120B, 170A, 170B and headsets 140, 190, and one or more networks, such as the wireless network 155. The transceivers can include amplifiers, modulators, demodulators, encoders, decoders, antennas, and various other components. The transceivers can direct data received from other external devices, such as servers, databases, and electronic devices, to other components in the computing devices 110 / 160, such as the processor or memory. The transceivers can also direct data received from components in the computing devices 110 / 160 to the external devices.

[0043] Data from computing device 110 can be transmitted to and received from computing device 160 through cloud platform 150 via wireless network 155. Data from computing device 160 can be transmitted to and received from computing device 110 through cloud platform 150 via wireless network 155. In general, data from computing devices 110, 160 can be transmitted across various types of wired or wireless networks. In some implementations, data can be transmitted without using cloud platform 150.

[0044] The cloud platform 150 can act as a communications switchboard. In particular, the cloud platform 150 can facilitate users 140, 180 to participate in holographic sessions and interconnect with others. The cloud platform 150 can host sessions on demand and quickly and efficiently scale to an unlimited number of simultaneous sessions between different users.

[0045] In some implementations, the cloud platform 150 may include a gateway, a web portal, and a session server. The gateway may manage one or more databases that store user and session data. The user and session data may include data such as data indicating the user logged into the VAR system 100, session start time, session duration, session topic, scheduled session time, user involved in the session, and session information such as IP and / or MAC address of the user device involved in the session. Data from a session may be stored with user authorization and may be recalled when a session is restored or when data from a past session is requested by a user.

[0046] The gateway may manage multiple session servers, receive and transmit data, and facilitate streaming and rendering of holographic images. In some cases, the gateway may provide one or more application program interfaces (APIs) to user devices or system components, such as computing devices 110, 160 or headsets 140, 190.

[0047] The web portal is a web interface for end users and administrators. Data for configuring the web portal can be provided by the gateway. Data sent and received through the portal can be transmitted to the gateway. The session server can host real-time communication between users during a session.

[0048] Generally, cloud platform 150 may be implemented using one or more servers, databases, and / or computing devices to execute software and algorithms to implement the operations described above. Cloud platform 150 may provide storage services for data transmitted between various components of system 100, e.g., computing devices 110, 160. Cloud platform 150 may be integrated within one location or distributed across more than one location.

[0049] The wireless network 155 may connect the cloud platform 150 to the computing devices 110, 160 associated with the users 130, 180. In some implementations, the wireless network 155 may be the Internet. In general, the wireless network 155 may include various types of networks, including short-range networks, such as Bluetooth and WiFi networks, and long-range networks, e.g., cellular networks or the Internet. In some implementations, the wireless network 155 may include a cloud system, including the cloud platform 150.

[0050] The wireless network 155 may include access points, storage systems, cloud systems, modules, one or more databases, including one or more media databases, and servers, including one or more network servers. The one or more network servers may include any suitable computing device, including, but not limited to, a personal computer, a server computer, a series of server computers, a minicomputer, and a mainframe computer, or a combination thereof. The one or more network servers may also include an application server or a series of servers. The one or more network servers may be used for and / or provide cloud and / or network computing services.

[0051] 1 depicts an example with two users and computing devices, in some implementations only one computing device and / or only one headset can be used. For example, a single computing device can be used when one person has an avatar instead of a hologram, or when a person is giving a lecture in front of the depth sensor without the need for a headset.

[0052] FIG. 2 illustrates an example flow chart for generating a 3D holographic image, including edge detection and smoothing within the image. The operations described with respect to FIG. 2 can be performed by one or more computing devices, such as computing device 110 or 160. In some implementations, the operations can be performed by a computing device in cloud platform 150 or a computing device other than device 110 or 160 that is coupled to headset 140 or 190. Unless otherwise noted, a user referenced in the discussion of FIG. 2 refers to user 140 or user 190. A depth sensor referenced in the discussion of FIG. 2 refers to a pair of depth sensors 120A and 120B or a pair of depth sensors 170A and 170B. A headset referenced in the discussion of FIG. 2 refers to headset 140 or headset 190.

[0053] The process shown in Figure 2 begins at operation 205, where data can be acquired from one or more depth sensors. The acquired data can include images captured by a camera in the depth sensor. The data can also include color information and a depth value for each pixel (depth from the camera to that pixel). Various suitable methods can be used to determine the depth values, such as stereo triangulation, optical triangulation, structured light, time of flight, or interferometry.

[0054] The data obtained from the depth sensor can be used to construct a 3D image space or voxel space. In operation 210, the voxel space can be reconstructed by dividing the space in front of the camera into a set of small cubes. The computing device can divide the voxel space based on the depth values ​​for each pixel.

[0055] In operation 215, the small cubes can be converted into a triangular mesh. The cubes are formed from eight voxels. Triangles can be identified based on whether the voxels are filled. The MarchingCubes algorithm or other suitable techniques can be applied iteratively to each cube to identify the triangles. FIG. 3 shows 15 unique cases in the MarchingCubes algorithm that can be used to generate the triangular mesh. After processing all the cubes, for example, the union of the identified triangles as identified by the MarchingCubes algorithm can be used to generate the triangular mesh.

[0056] Next, in operation 220, the characteristics of the triangles can be determined by calculating the normal of each triangle. The vertices point to the corners of one of the identified triangles. Using the vertices of each triangle, the vector formed by the vertices of the triangle can be used to generate a normal for the triangle. In this triangular mesh, the normal for a particular triangle can be calculated as a weighted average of the normals of the triangles that are tangent to the particular triangle and the determined normal for only the particular triangle.

[0057] In operation 225, the edges of the triangular mesh can be determined. For each vertex, the computing device can determine the neighboring vertices (vertices one vertex away from the vertex) and neighboring vertices and whether the vertex shares a common triangle. When a pair of vertices share two triangles, the vertices are determined to be in the center of the triangular mesh. When a pair of vertices share only a single triangle, the vertices are determined to be on the edge of the triangular mesh.

[0058] For example, Figure 4 depicts a portion of an example triangular mesh 400. The computing device may determine that vertices 3 and 4 are located on the edge of the mesh because they share only a single triangle. The computing device may also determine that vertices 1 and 2 are somewhere in the center of the mesh because vertices 1 and 2 share two triangles. Vertices 3 and 4 may be designated as edge nodes, and vertices 1 and 2 may be designated as interior nodes.

[0059] In operation 230, the computing device can determine the distance to the edge for each internal node, i.e., the vertex is on the interior of the mesh. The distance can be described in various ways, including, for example, the number of hops from the vertex to reach the edge. In some implementations, the distance can be determined by adding one hop to the determined distance of the neighboring vertex. If there are multiple neighboring vertices, the minimum distance obtained from the distances of the neighboring vertices can be used. Multiple iterations of this determination can be performed for each internal node to correctly determine the distance to the edge for each internal node. In some implementations, the edge distance determination can be performed in parallel for each vertex.

[0060] After determining the edges in act 225 and the distances to the edges in act 230, the computing device may optionally perform mesh smoothing in act 235. To smooth a triangle mesh, the computing device may determine, for each vertex to be smoothed, the average location of the vertices that share a triangle with the vertex. The current location of the vertex is then modified to bring it closer to the average location of the vertices that share a triangle with the vertex.

[0061] In operation 240, the edges may be smoothed. Smoothing may be applied to each vertex that is located a determined number of hops from the edge, for example, 3 hops. The determined number of hops may be adjusted by an operator of the VAR system 100 to vary the degree of edge smoothing.

[0062] To perform edge smoothing, an average location of vertices a determined number of hops away from the edge is determined. Normals associated with the edge and locations of vertices associated with the edge can be adjusted to smooth the edge. More specifically, for each vertex located a determined number of hops from the edge, the computing device can determine an average of the locations of vertices that share a triangle with the vertex being processed and have the same distance from the edge. Based on the determined average location, location coordinates of the edge vertices can be adjusted to move the vertices closer to the average location. Normals for the triangles can also be recalculated with the new vertex locations.

[0063] In some implementations, Equation 1 may be used to perform edge smoothing. [ka]

[0064] In equation 1, v i is the current location of the vertex being processed, and v f is the location relative to which the vertex should be moved, and v n is the vector of the nth neighbor with the same distance to the edge as the vertex being processed, and N is the number of neighboring vertices with the same distance to the edge as the vertex being processed.

[0065] Figures 5A-5D show an example of edge smoothing. In Figures 5A-5D, a portion of a triangular mesh is shown. The portion includes vertices A, B, C, D, E, and F. For illustration purposes, only the two-dimensional (2D) coordinates of the vertices are shown. In Figure 5A, vertex A has coordinates of (10,10), vertex B has coordinates of (30,20), vertex C has coordinates of (50,10), vertex D has coordinates of (65,10), vertex E has coordinates of (95,25), and vertex F has coordinates of (120,10).

[0066] After one step of determining the vertex locations, edge smoothing is performed by modifying the locations of one or more vertices in the manner described above. The location of vertex B is changed from (30,20) in FIG. 5A to (30,17.14) in FIG. 5B, moving vertex B closer to the locations of nearby vertices A and C. Similarly, the location of vertex C is changed from (50,10) to (49.29,11.43), moving vertex C closer to the locations of nearby vertices B and D, the location of vertex D is changed from (65,10) to (67.14,12.14), moving vertex D closer to the locations of nearby vertices C and E, and the location of vertex E is changed from (95,25) to (94.29,20.71), moving vertex E closer to the locations of nearby vertices D and F.

[0067] 5C and 5D show two more iterations of the edge smoothing step. After performing multiple iterations, the edges of the mesh are smoothed, as can be seen by comparing FIGS. 5A-5D. The edge smoothing can be performed repeatedly for a fixed number of iterations, which can be variably set. Advantageously, the smoothing operation is effectively a filtering operation that removes noise from the mesh, thereby improving the overall quality of the rendered image.

[0068] Another example of an edge smoothing operation is shown in Figures 6A and 6B. Figure 6A shows a portion of a triangular mesh that has not been smoothed. Figure 6B shows the same portion of the triangular mesh after edge smoothing has been performed. As shown in Figures 6A and 6B, this edge smoothing operation can result in the removal of sharp and noisy edges and the creation of a smooth band of triangles around the perimeter of the mesh.

[0069] Operation 240 can be performed simultaneously for all voxels and can be repeated for a determined number of iterations, which can be set by an operator of VAR system 100. After the determined number of iterations is completed, some vertices may be located very close to each other. An example of close proximity of vertices is highlighted by dashed circle 610 shown in FIG. 6B. Such instances of close-located vertices are not necessary.

[0070] In operation 245, the mesh can be cleaned up by merging vertices that are less than a threshold distance away from each other, such as some of the vertices in circle 610. After merging the vertices, new triangles can be generated. To further clean up the mesh, operation 250 can also be performed to remove triangles, forming a sub-mesh of several triangles. A sub-mesh is a part of the mesh that consists only of edges and has no interior triangles. Operation 250 can also reduce noise in the mesh.

[0071] After the mesh is cleaned up and noise removed, the mesh can be further simplified in operation 255. To simplify the mesh, vertices and triangles that do not substantially contribute to the shape of the mesh are removed. To determine the contribution of the vertices and triangles, the computing device can identify pairs of vertices that belong to the same triangle and calculate the impact on the mesh when the two vertices are merged. The impact can be calculated by determining the change in angle between all neighboring triangles before and after merging. For example, a vertex associated with a flat surface will have no impact, while a vertex at the top of the nose will likely have a larger impact. When the amount of change is less than a threshold, the vertices can be merged.

[0072] An example of the mesh simplification operation 255 is shown in Figures 7A and 7B. Figure 7A shows a portion of a triangular mesh that has not been simplified. Figure 7B shows the same portion of the triangular mesh after mesh simplification has been performed. As shown in Figure 7B, the resulting mesh has fewer vertices, triangles, and edges.

[0073] During operation 255, vertices on the edge of the mesh are not merged, preserving the edge and the overall shape of the mesh. Vertices located near an edge can only be merged with vertices that are at the same distance to the edge. Triangles with zero area are removed. The mesh simplification operation 255 can be repeated for a determined number of iterations, which can be set by an operator of the VAR system 100.

[0074] In operation 260, a texture projection can be applied onto the mesh. To perform this operation, the computing device can determine the color and camera to use per vertex to project the color image onto the mesh. The computing device can additionally determine a camera weight for each vertex. The camera weight refers to a numerical value that corresponds to the amount that each camera of the depth sensor can contribute to the color of the vertex.

[0075] For the front of the mesh, both cameras may contribute equally, but edges will only have the contribution of one camera. The weighting of each camera per vertex can be determined by the computing device based on the angle between the mesh and the camera view direction, the difference between the camera's depth value and the actual distance from the camera to the vertex, and the distance to the edge of the image.

[0076] After performing the operations shown in FIG. 2, the computing device can store data indicating camera weights, colors, vertex locations, depth data, and any other information, such as distance to the edge of the mesh per vertex, triangle locations, that can be used to recreate the mesh. This data can be transmitted over wireless network 155 to a viewer of the mesh, such as user 130 or 180. The transmitted data can be decoded and processed by the computing device receiving the transmitted data for rendering. The mesh is recreated by the GPU of the receiving computing device, and a color image is projected on top of the mesh. The camera weights can be used to determine how the colors of the different cameras are blended together per vertex. The distance to the edge can determine the excess blending effect, such as the transparency around the edge of the mesh. The closer to the edge, the more transparent rendering can be applied, so the mesh appears to fade in space, hiding errors around the edge.

[0077] By using the above described method of rendering a holographic image, a viewer can enjoy a higher quality holographic image due to less noise in the image. In addition, since the above described operations introduce a latency of less than 50 milliseconds, the method can be used for real-time communication such as streaming HD data.

[0078] The described systems, methods, and techniques can be implemented in digital electronic circuitry, computer hardware, firmware, software, or combinations of these elements. Apparatuses implementing these techniques can include suitable input and output devices, computer processors, and computer program products tangibly embodied in machine-readable storage devices for execution by a programmable processor. Processes implementing these techniques can be performed by a programmable processor that executes a program of instructions to perform the desired functions by operating on input data and generating appropriate output. The techniques can be implemented using one or more computer programs or non-transitory computer-readable storage media that include instructions executable on a programmable system that includes at least one programmable processor coupled to receive data and instructions from, and transmit data and instructions to, a data storage system, at least one input device, and at least one output device.

[0079] Each computer program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language, as desired; in either case, the language may be a compiled or interpreted language. Suitable processors include, by way of example, both general-purpose and special-purpose microprocessors. In general, the processor will receive instructions and data from a read-only memory and / or a random-access memory. Suitable storage devices for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, by way of example, semiconductor memory devices such as erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and compact disk read-only memories (CD-ROMs). Any of the foregoing may be supplemented by, or incorporated in, specially designed ASICs (application-specific integrated circuits).

[0080] A computer-readable medium can be a machine-readable storage device, a machine-readable storage device substrate, a memory device, a composition of matter that provides a machine-readable propagated signal, or a combination of one or more of them. The term "computing device" encompasses all apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, an apparatus can include code that creates an execution environment for the computer program, such as code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to a suitable receiver device.

[0081] A computer program, also known as a program, software, software application, script, plug-in, or code, can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file, or in multiple cooperating files, holding other programs or data in a single file dedicated to the program. A computer program can be executed on one computer or on multiple computers, located at one site or distributed across multiple sites and interconnected by a communication network.

[0082] The processes and logic flows described herein can be implemented by one or more programmable processors that execute one or more computer programs and perform actions by operating on input data and generating output. The processes and logic flows can also be implemented by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit), and an apparatus can also be implemented as such.

[0083] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both.

[0084] The elements of a computer may include a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or be operatively coupled to receive data therefrom or transfer data thereto, or both. In addition, a computer may be embedded in another device, such as a tablet computer, a mobile phone, a personal digital assistant (PDA), a mobile audio player, a VAR system, to name only a few. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0085] Although the present specification contains many details, these should not be construed as limitations on the scope of the disclosure or what may be claimed, but rather as descriptions of features specific to certain embodiments of a particular invention. Certain features described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Also, although features may be described above as acting in a combination and may even be claimed as such, one or more features from the claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination. For example, while a mapping operation is described as a series of discrete operations, various operations may be divided into additional operations, combined into fewer operations, varied in order of execution, or eliminated, depending on the desired implementation.

[0086] Similarly, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged into multiple software products.

[0087] The terms used in this specification, and particularly in the appended claims (e.g., the body of the appended claims), are generally intended to be "open" terms (e.g., the term "including" should be interpreted as "including, but not limited to," the term "having" should be interpreted as "having at least," the term "includes" should be interpreted as "including, but not limited to," etc.).

[0088] In addition, if a specific number of introduced claim recitations is intended, such intent will be explicitly recited in the claim, and in the absence of such recitation, such intent would not exist. For example, as an aid to understanding, the following appended claims may contain the use of the introductory phrases "at least one" and "one or more" to introduce the recitation of claims. However, the use of such phrases should not be interpreted as implying that the introduction of a claim recitation with the indefinite article "a" or "an" limits any particular claim containing such introduced claim recitation to an embodiment containing only one such recitation, even when the same claim contains the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should be interpreted to mean "at least one" or "one or more"). The same applies to the use of definite articles used to introduce claim recitations.

[0089] In addition, even when a claim recitation introducing a specific number is explicitly recited, one of ordinary skill in the art will recognize that such recitation should be interpreted to mean at least the number recited (e.g., the bare recitation of "two enumerations," without other modifiers, means at least two enumerations or two or more enumerations). Furthermore, in those instances where conventional language similar to "at least one of A, B, and C, etc." or "one or more of A, B, and C, etc." is used, generally such a structure is intended to include A only, B only, C only, both A and B, both A and C, both B and C, or both A, B, and C. The term "and / or" is also intended to be interpreted in this manner.

[0090] Use of the terms "first", "second", "third", etc., is not necessarily used herein to imply a specific order or number of elements. In general, the terms "first", "second", "third", etc., are used as generic identifiers to distinguish between different elements. Unless otherwise indicated, the terms "first", "second", "third", etc., imply a specific order, these terms should not be understood to imply a specific order. Furthermore, unless otherwise indicated, the terms "first", "second", "third", etc., imply a specific number of elements, these terms should not be understood to imply a specific number of elements. For example, a first widget can be described as having a first side and a second widget can be described as having a second side. Use of the term "second side" for a second widget can distinguish such side of the second widget from the "first side" of the first widget and does not imply that the second widget has two sides.

[0091] Claims

Claims

1. 1. A computer-implemented method comprising: acquiring image data from a pair of cameras; determining a mesh of triangles from the image data; removing noise from said triangular mesh by performing mesh smoothing and edge smoothing; simplifying the triangular mesh by removing vertices and triangles from the triangular mesh that do not contribute to an overall shape of the triangular mesh within a contribution threshold, wherein simplifying the triangular mesh by removing vertices and triangles from the triangular mesh that do not contribute to an overall shape of the triangular mesh within the contribution threshold includes identifying a pair of vertices that belong to the same triangle and determining an effect on the mesh if the pair of vertices is merged, wherein the contribution of the vertices and triangles to the shape of the triangular mesh is calculated by determining an effect on the mesh if the pair of vertices is merged, and determining the effect on the mesh if the pair of vertices is merged includes determining a change in angle between all neighboring triangles before and after the pair of vertices is merged; storing in a memory mesh data characterizing the mesh after the vertices and triangles are removed from the mesh; transmitting the mesh data to a rendering device configured to render a holographic image based on the mesh data; 23. A computer-implemented method comprising:

2. The computer-implemented method includes determining edges for the triangles; Determining an edge for the triangle comprises: For each vertex in the mesh, determining a neighboring vertex located one vertex away from the vertex; determining whether the neighboring vertices and the vertex share a common triangle; determining that the neighboring vertices and the vertices are interior nodes of the mesh when the neighboring vertices and the vertices are shared by at least two triangles; determining that the neighboring vertices and the vertices are exterior nodes of the mesh when the neighboring vertices and the vertices are part of a triangle; The computer-implemented method of claim 1 , wherein the method is performed by:

3. The computer-implemented method of claim 2, further comprising: determining, for each internal node, a distance to an edge of the mesh, the distance being the number of hops from the internal node to reach the edge.

4. Performing mesh smoothing and edge smoothing includes: For each vertex, determining an average location of vertices that share a triangle with said vertex; modifying the location of said vertex to more closely approximate the average location of vertices that share a triangle with said vertex; The computer-implemented method of claim 1 , comprising:

5. Performing mesh smoothing and edge smoothing includes: identifying a vertex that is a determined number of hops from a particular edge of the mesh; identifying other vertices that share the triangle and are located the determined number of hops from the particular edge; determining an average location of said other vertices; modifying the locations of the vertices so that the locations of the vertices are closer to the average location; The computer-implemented method of claim 1 , comprising:

6. The computer-implemented method comprises: determining that two vertices are separated by a distance that is less than a threshold distance; Merging the two vertices; The computer-implemented method of claim 1 , comprising:

7. The computer-implemented method of claim 1, wherein simplifying the triangular mesh by removing vertices and triangles from the triangular mesh that do not contribute to the overall shape of the triangular mesh within the contribution threshold further includes determining that the impact on the mesh is less than the contribution threshold.

8. The computer-implemented method of claim 1, comprising removing one or more triangles from the mesh prior to removing the vertices and the triangles by removing one or more triangles that form a submesh within the mesh.

9. The computer-implemented method of claim 1 , wherein the mesh data comprises data indicative of color in one or more images and depth from the pair of cameras to pixels in the one or more images.

10. 1. A virtual reality or augmented reality system, comprising: a frame structure comprising a pair of depth sensors configured to acquire image data; A computing device coupled to the frame structure, the computing device comprising: acquiring image data from the pair of depth sensors; determining a mesh of triangles from the image data; removing noise from said triangular mesh by performing mesh smoothing and edge smoothing; simplifying the triangular mesh by removing vertices and triangles from the triangular mesh that do not contribute to an overall shape of the triangular mesh within a contribution threshold, the simplifying the triangular mesh by removing vertices and triangles from the triangular mesh that do not contribute to an overall shape of the triangular mesh within the contribution threshold includes identifying a pair of vertices that belong to the same triangle and determining an effect on the mesh if the pair of vertices is merged, the contribution of the vertices and triangles to the shape of the triangular mesh is calculated by determining an effect on the mesh if the pair of vertices is merged, and determining the effect on the mesh if the pair of vertices is merged includes determining a change in angle between all neighboring triangles before and after the pair of vertices is merged; A computing device configured to: a memory configured to store mesh data characterizing the mesh after the vertices and the triangles are removed from the mesh; and a headset configured to render a holographic image based on the mesh data; A virtual reality or augmented reality system comprising:

11. 11. The virtual reality or augmented reality system of claim 10, wherein the computing device is configured to transmit the mesh data to one or more rendering devices configured to render the holographic image based on the mesh data, the one or more rendering devices comprising the headset.

12. To determine edges for the triangles, the computing device comprises: For each vertex in the mesh, determining a neighboring vertex located one vertex away from the vertex; determining whether the neighboring vertices and the vertex share a common triangle; determining that the neighboring vertices and the vertices are interior nodes of the mesh when the neighboring vertices and the vertices are shared by at least two triangles; determining that the neighboring vertices and the vertices are exterior nodes of the mesh when the neighboring vertices and the vertices are part of a triangle; 11. The virtual reality or augmented reality system of claim 10 configured to:

13. To perform the mesh smoothing and edge smoothing, the computing device comprises: For each vertex, determining an average location of vertices that share a triangle with said vertex; modifying the location of said vertex to more closely approximate the average location of vertices that share a triangle with said vertex; 11. The virtual reality or augmented reality system of claim 10 configured to:

14. To perform the mesh smoothing and edge smoothing, the computing device comprises: identifying a vertex that is a determined number of hops from a particular edge of the mesh; identifying other vertices that share the triangle and are located the determined number of hops from the particular edge; determining an average location of said other vertices; modifying the locations of the vertices so that the locations of the vertices are closer to the average location; 11. The virtual reality or augmented reality system of claim 10 configured to:

15. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more computing devices, acquiring image data from a pair of cameras; determining a mesh of triangles from the image data; removing noise from said triangular mesh by performing mesh smoothing and edge smoothing; simplifying the triangular mesh by removing vertices and triangles from the triangular mesh that do not contribute to an overall shape of the triangular mesh within a contribution threshold, the simplifying the triangular mesh by removing vertices and triangles from the triangular mesh that do not contribute to an overall shape of the triangular mesh within the contribution threshold includes identifying a pair of vertices that belong to the same triangle and determining an effect on the mesh if the pair of vertices is merged, the contribution of the vertices and triangles to the shape of the triangular mesh is calculated by determining an effect on the mesh if the pair of vertices is merged, and determining the effect on the mesh if the pair of vertices is merged includes determining a change in angle between all neighboring triangles before and after the pair of vertices is merged; storing in a memory mesh data characterizing the mesh after the vertices and triangles are removed from the mesh; transmitting the mesh data to a rendering device configured to render a holographic image based on the mesh data; 23. A non-transitory computer-readable storage medium that causes the one or more computing devices to perform operations including:

16. the one or more computing devices are configured to determine edges for the triangles; Determining an edge for the triangle comprises: For each vertex in the mesh, determining a neighboring vertex located one vertex away from the vertex; determining whether the neighboring vertices and the vertex share a common triangle; determining that the neighboring vertices and the vertices are interior nodes of the mesh when the neighboring vertices and the vertices are shared by at least two triangles; determining that the neighboring vertices and the vertices are exterior nodes of the mesh when the neighboring vertices and the vertices are part of a triangle; The non-transitory computer-readable storage medium of claim 15, wherein the non-transitory computer-readable storage medium is

17. Performing mesh smoothing and edge smoothing includes: For each vertex, determining an average location of vertices that share a triangle with said vertex; modifying the location of said vertex to more closely approximate the average location of vertices that share a triangle with said vertex; 20. The non-transitory computer readable storage medium of claim 15, comprising:

18. Performing mesh smoothing and edge smoothing includes: identifying a vertex that is a determined number of hops from a particular edge of the mesh; identifying other vertices that share the triangle and are located the determined number of hops from the particular edge; determining an average location of said other vertices; modifying the locations of the vertices so that the locations of the vertices are closer to the average location; 20. The non-transitory computer readable storage medium of claim 15, comprising:

19. The non-transitory computer-readable storage medium of claim 15, wherein simplifying the triangular mesh by removing vertices and triangles from the triangular mesh that do not contribute to the overall shape of the triangular mesh within the contribution threshold further includes determining that the impact on the mesh is less than the contribution threshold.

Images