Augmented reality effects with real-time depth maps to sense geometry

Abstract

The technology of introducing virtual objects into the physical environment of the AR system includes shifting the vertices of the mesh representing the physical environment based on the real-time depth map. For example, the AR system generates a mesh template, that is, an initial mesh with vertices representing the physical environment, and a depth map indicating the geometric structure of the real objects in the physical environment. The AR system is configured to represent the real objects in the physical environment by shifting the vertices of the mesh based on the depth values ​​of the depth map and the parameter values ​​of the pinhole camera model. The depth value can be obtained from the perspective of the illumination source in the physical environment.

Description

Technical Field

[0001] The present description relates to operating an augmented reality system in which virtual objects interact with real objects in a physical environment. Background Art

[0002] Augmented reality (AR) is an interactive experience of a physical environment (i.e., a scene with real objects) in which the objects residing in the physical environment are enhanced by computer-generated sensory information (including visual information). Some AR systems include features such as the merging of real and virtual worlds, real-time interaction, and accurate 3D registration of virtual and real objects. Summary of the invention

[0003] An embodiment provides a geometry-aware AR system that generates a mesh template, i.e., an initial mesh with vertices representing a physical environment, and a depth map indicating the geometry of a real object in the physical environment. The connectivity of the mesh is determined based on the generated indexes, which represent vertices arranged in triplets in a specified winding order to produce a set of triangles that constitute the mesh. The AR system is configured to represent real objects in the physical environment by shifting the vertices of the mesh based on the depth values ​​of the depth map and the parameter values ​​of the pinhole camera model. The depth value can be obtained from the perspective of the illumination source in the physical environment. The mesh template and the depth map can be generated in the central processing unit (CPU) of the AR system and copied to the graphics processing unit (GPU), and the AR engine can efficiently perform shadow mapping and physical phenomenon simulation on the graphics processing unit (GPU). The depth map can be generated in real time and updated in the GPU. Shadow mapping and physical phenomenon simulation also depend on the connectivity of the mesh that does not change over time.

[0004] In one general aspect, a method may include generating a triangular mesh representing a physical environment and a depth map of the physical environment, the triangular mesh including a plurality of vertices, the depth map including a plurality of depth values. The method may also include performing a shift operation on the plurality of vertices of the triangular mesh to generate a plurality of shifted vertices representing a geometry of at least one real object within the physical environment, the shift operation being based on the depth map. The method may also include receiving virtual object data representing a virtual object, the virtual object being configured to be displayed with at least one real object in the physical environment. The method may also include displaying the virtual object in the physical environment on a display to generate a displayed virtual object, the displayed virtual object having a difference from the virtual object based on the plurality of shifted vertices.

[0005] In another general aspect, a computer program product includes a non-transitory storage medium, the computer program product including code that, when executed by a processing circuit of a computing device, causes the processing circuit to perform a method. The method may include generating a triangular mesh representing a physical environment and a depth map of the physical environment, the triangular mesh including a plurality of vertices, the depth map including a plurality of depth values. The method may also include performing a shift operation on the plurality of vertices of the triangular mesh to generate a plurality of shifted vertices representing a geometry of at least one real object within the physical environment, the shift operation being based on the depth map. The method may also include receiving virtual object data representing a virtual object, the virtual object being configured to be displayed with at least one real object in the physical environment. The method may also include displaying the virtual object in the physical environment on a display to generate a displayed virtual object, the displayed virtual object having a difference with the virtual object according to the plurality of shifted vertices.

[0006] In another general aspect, an electronic device configured to generate a re-crawl strategy includes a memory and a control circuit coupled to the memory. The control circuit can be configured to generate a triangular mesh representing a physical environment and a depth map of the physical environment, the triangular mesh including a plurality of vertices, and the depth map including a plurality of depth values. The control circuit can also be configured to perform a shift operation on the plurality of vertices of the triangular mesh to generate a plurality of shifted vertices representing the geometry of at least one real object within the physical environment, the shift operation being based on the depth map. The control circuit can also be configured to receive virtual object data representing a virtual object, the virtual object being configured to be displayed with at least one real object in the physical environment. The control circuit can also be configured to display the virtual object in the physical environment on a display to generate a displayed virtual object, the displayed virtual object having a difference with the virtual object according to the plurality of shifted vertices.

[0007] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1A is a diagram illustrating an example real-world space.

[0009] Figure 1B is a diagram illustrating another example real-world space.

[0010] Figure 1C is a diagram illustrating an example electronic environment in which the improved techniques described herein may be implemented.

[0011] Figure 2 is a flow chart illustrating an example method of operating an augmented reality system according to disclosed embodiments.

[0012] Figure 3A is a diagram illustrating an overhead view of an example physical environment imaged in an augmented reality system.

[0013] Figure 3B and 3C is a diagram illustrating a front view of an example physical environment imaged in an augmented reality system.

[0014] Figure 4A is a diagram illustrating an example depth map and a mesh template.

[0015] Figure 4B is a diagram illustrating a representation of an example grid template and its connectivity.

[0016] Figure 5 is a diagram illustrating a mesh with vertices displaced according to a depth map in accordance with disclosed embodiments.

[0017] Figure 6 is a diagram illustrating an example AR system and its components according to disclosed implementations.

[0018] Figure 7 is a diagram illustrating an example AR system with shadow tracing and its components according to disclosed embodiments.

[0019] Figure 8 is a diagram illustrating an example AR system and its components with a physical phenomenon simulator according to disclosed embodiments.

[0020] Fig. 9 is a diagram illustrating examples of computer devices and mobile computer devices that can be used to implement the described techniques. DETAILED DESCRIPTION

[0021] In some cases, an AR system introduces virtual objects associated with locations in the physical environment within a display screen in a way that the virtual objects are occluded by real objects in the physical environment. However, some AR systems do not take into account the geometry of real objects in the physical environment, resulting in virtual objects not being occluded by real objects. For example, virtual furniture and game characters may appear in front of real objects even if they are spatially located behind a sofa or table. Shadows and virtual rigid body colliders may only interact with known AR objects (such as detected horizontal or vertical planes).

[0022] Traditional means of introducing virtual objects into the physical environment in an AR system include performing a scan of the physical environment to generate a rough three-dimensional model of the physical environment. In addition, traditional means include creating shadows by calculating shadow maps based on planes detected from the camera stream.

[0023] The technical problem with the above-mentioned traditional means for introducing virtual objects into the physical environment in the AR system is that such means require too much computing resources to execute in real time. Thus, such traditional means may reduce the real-time interactive experience of the user. In addition, when the virtual object casts a shadow onto other physical objects on the plane, the shadow generated by the traditional means may cause visible artifacts.

[0024] According to the embodiments described herein, the technical solution to the above technical problem includes shifting the vertices of the mesh representing the physical environment based on the real-time depth map. For example, the AR system generates a mesh template, that is, an initial mesh with vertices representing the physical environment, and a depth map indicating the geometric structure of the real object in the physical environment. The connectivity of the mesh is determined according to the generated indexes, which represent the vertices arranged in the triples in a specified winding order to generate a set of triangles constituting the mesh. The AR system is configured to represent the real object in the physical environment by shifting the vertices of the mesh based on the depth value of the depth map and the parameter value of the pinhole camera model. The depth value can be obtained from the perspective of the illumination source in the physical environment. The mesh template and the depth map can be generated in the central processing unit (CPU) of the AR system and copied to the graphics processing unit (GPU), and the AR engine can efficiently perform shadow tracing and physical phenomenon simulation on the graphics processing unit (GPU). The depth map can be generated in real time and updated in the GPU. Shadow tracing and physical phenomenon simulation also depend on the connectivity of the mesh that does not change over time.

[0025] A technical advantage of the disclosed embodiments is that the AR system described above provides occlusion in real time. Computation time is greatly reduced by not requiring the use of surface reconstruction algorithms to trace the scene, making this approach easy to use on mobile devices. Representing depth as a screen space mesh also enables countless existing 3D assets and shader effects to interact with the real environment in AR. In addition, because the mesh template and real-time depth map are operated by the GPU with built-in shaders and physical phenomenon simulation capabilities, the calculation speed is very fast; for example, vertex displacement can be performed in parallel.

[0026] Figure 1A A real-world space 10 is illustrated and a user 13 in the real-world space 10 is illustrated. Real-world objects and AR objects are illustrated together in the figure as they would be seen by the user 13 through the mobile device. The scene (e.g., the scene of a room) viewed by the user 13 of the AR system is illustrated with dotted lines. The real-world space 10 may include at least one real-world object 15. The AR system associated with the mobile device may be configured to place an AR object 14 in the real-world space 10. Figure 1AThe AR object 14 is illustrated as being placed at a depth behind the real-world object 15. However, based on the depth and position of the real-world object 15 as compared to the depth and position of the AR object 14, only a portion of the AR object 16 (greyed out) is behind the real-world object 15.

[0027] Figure 1B Again, the AR object 14 is shown in the real world space 10. Figure 1B , the AR object has been repositioned and placed in front of the real-world object 15 at a depth.

[0028] Figure 1C 1 is a diagram illustrating an example electronic environment 100 in which the above-described technical solution may be implemented. The electronic environment 100 includes a computer 120 configured to introduce virtual objects into a physical environment in an AR system.

[0029] The computer 120 includes a network interface 122, one or more processing units 124, a memory 126, and a display interface 128. The network interface 122 includes, for example, an Ethernet adapter and / or a token ring adapter, etc., for converting electronic and / or optical signals received from the network 150 into electronic form for use by the computer 120. The group of processing units 124 includes one or more processing chips and / or components, including a central processing unit (CPU) 192 and a graphics processing unit (GPU) 194. In some embodiments, the GPU 194 is optimized to handle mesh data representing a three-dimensional object. The memory 126 includes both volatile memory (e.g., RAM) and non-volatile memory, input one or more ROMs, disk drives, and solid-state drives, etc. The group of processing units 124 and the memory 126 together form a control circuit that is configured and arranged to perform various methods and functions as described herein.

[0030] In some implementations, one or more components of the computer 120 may be or include a processor (e.g., processing unit 124) configured to process instructions stored in the memory 126. Examples of such instructions as shown in FIG. 1 may include a mesh manager 130, a depth map manager 140, a vertex shift manager 150, a virtual object manager 160, a shadow generation manager 170, a collision volume mesh manager 180, and a rendering manager 190. Also, as shown in FIG. 1, the memory 126 is configured to store various data, which is described with respect to the response manager using such data. Note that in some implementations, the entity page corresponds to a quote page that includes a quote for selling a product.

[0031] The grid manager 130 is configured to generate, receive, or obtain grid data 132. In some embodiments, the grid manager 130 generates grid data 132 for a grid template based on a uniform grid over a camera field of view from which the physical environment can be seen. In some embodiments, the grid manager 130 is configured to receive or obtain image data 132 from a display device 170 via a network interface 122, i.e., via a network such as network 190. In some embodiments, the image manager 130 is configured to receive or obtain image data 132 from a local storage (e.g., a disk drive, a flash drive, an SSD, etc.).

[0032] In some implementations, the mesh manager 130 is configured to generate the mesh data 132 on the CPU 192. In this case, the mesh manager 130 is configured to copy the mesh data 132 from the CPU 192 to the GPU 194. In this way, the AR system can efficiently process vertex displacements through the computer 120. In some implementations, the mesh manager 130 is configured to procedurally generate the mesh data 132 on the GPU 194.

[0033] The mesh data 132 represents a triangular mesh sampled at regular positions in the physical environment, which in turn is the physical environment. The mesh data 132 includes vertex data 133 and index data 134.

[0034] Vertex data 133 represents a plurality of vertices, each vertex initially located at a regular sampling position within the physical environment. Four adjacent vertices may form a pixel of a triangular mesh. In some embodiments, each vertex in the plurality of vertices is represented by a pair of numbers representing coordinates in a coordinate plane. In some embodiments, each vertex in the plurality of vertices is represented by a triple of numbers, one of the triples being set to zero. In some embodiments, the coordinates are real numbers. In some embodiments, the coordinates are integers derived from a quantization process.

[0035] Index data 134 indicates connectivity of the triangular mesh. In some embodiments, index data 134 includes a plurality of indices, each of the plurality of indices corresponding to one of the plurality of vertices. The plurality of indices are arranged into a plurality of triplets of numbers, each triplet corresponding to one triangle of the mesh, such as Figure 4B Each triplet is arranged in the specified winding order. For example, when the normal of each triangle faces outward, each triplet is arranged in a clockwise direction.

[0036] The depth map manager 140 is configured to generate the depth map data 142 in real time. In some embodiments, the depth map manager 140 does not perform a three-dimensional scan of the physical environment, as this may interrupt the real-time information flow and degrade the user experience. Instead, in some embodiments, the depth map manager 140 uses a dedicated time-of-flight (ToF) depth sensor available on some mobile devices. In some embodiments, the depth map manager 140 uses a stereo camera to generate the depth map data 142. In some embodiments, the depth map manager 140 uses a monocular camera in conjunction with software to generate the depth map data 142. In some embodiments, the depth map manager 140 generates the depth map data 142 in the GPU 194 at short but regular intervals (e.g., every 10-20 ms).

[0037] The depth map data 142 includes a real-time mesh representation of a depth map in the GPU 194. A depth map is a perspective camera image that includes depth values ​​rather than color / grayscale values ​​in each pixel. In some embodiments, each pixel of the depth map corresponds to a corresponding pixel of a triangular mesh. Figure 4A An example of such a depth map is found in . Depth map data 142 includes depth value data 143 and camera / source data 144 .

[0038] The depth value data 143 represents the depth value at each pixel of the grid representing the depth map. In some embodiments, the depth value is a measure of the light from an illumination source (e.g., a lamp, the sun) illuminating the physical environment along the trajectory. In some embodiments, the depth value is a real number representing the depth measurement. In some embodiments, the depth value is an integer resulting from a quantization process or an integer representing a plurality of pixels.

[0039] The camera / source data 144 represents the position and direction from which the depth value data 143 is generated. In some embodiments, the camera / source data 144 includes a triple of a real number representing the illumination source and a pair of real numbers (e.g., polar angle and azimuth angle) or a pair of real numbers and a signed value representing the direction (e.g., direction cosine). In some embodiments, the depth map manager 140 is configured to subtract the minimum distance to the illumination source from the depth value at each pixel. In some embodiments, the depth value at each pixel is the longitudinal component of the distance along the illumination ray.

[0040] In some embodiments, the camera / source data 144 also includes a set of real or integer numbers (i.e., a number of pixels) representing camera parameter values. In some embodiments, the camera parameters represent a pinhole camera model. The camera pinhole model assumes that the camera is a pinhole camera (i.e., a box with a pinhole on one side and an image plane on the other side). The parameters of the camera pinhole model include image width, image height, focal length in the image width direction, focal length in the image height direction, principal point in the image width direction, and principal point in the image height direction. In some embodiments, the focal length in both directions is equal to the distance between the pinhole and the image plane. The principal point is the location of the intersection between the line through the center of the pinhole and the image plane.

[0041] Vertex shift manager 150 is configured to generate shifted vertex data 152 based on mesh data 132 and depth map data 142, in particular vertex data 133, depth value data 143, and camera / source data 144. Vertex shift manager 150 shifts vertices according to the following formula:

[0042]

[0043]

[0044] Wherein, (X, Y) is the coordinate of the mesh template vertex (i.e., vertex data 133), (X′, Y′) is the coordinate of the shifted vertex (i.e., shifted vertex data 152), Z represents the depth value at the pixel corresponding to the pixel of the vertex contained at (X, Y), (F x ,F y ) represents the focal length of the pinhole camera defined above, and (F x ,F y ) represents the principal point of the pinhole camera defined above.

[0045] Shifted vertex data 152 represents shifted vertices according to equations (1) and (2). In some embodiments, shifted vertex data 152 includes triplets of numbers (real integers) representing the coordinates of the shifted vertices. In some embodiments, the resulting mesh represented by shifted vertex data 152 has connectivity defined by index data 134, because in such embodiments it is assumed that the connectivity of the mesh does not change over time.

[0046] In some embodiments, the vertex shifting manager 150 is configured to remove a vertex from the triangle mesh when the depth map satisfies a criterion. In some embodiments, the criterion includes a difference greater than a threshold between: (i) a depth value of the depth map corresponding to the vertex and (ii) an average of the depth value of the depth map corresponding to the vertex and depth values ​​corresponding to a set of neighboring vertices in the plurality of vertices.

[0047] The virtual object manager 160 is configured to generate, receive or obtain virtual object data 162 to enable the GPU 194 to place the virtual object represented by the virtual object data 162 to interact with the mesh containing the displaced vertices represented by the displaced vertex data 152. By displacing the vertices, the computer 120 is able to occlude the virtual object with the real object, rather than simply placing the virtual object in front of the real object. Figures 3A-3C In addition, the computer 120 can render shadows on real and virtual objects and simulate the physical phenomenon of collision between real and virtual objects. In this way, the virtual object manager 160 changes the virtual object according to the displaced vertices.

[0048] The shadow generation manager 170 is configured to generate shadow data 172 representing shadows generated by real and virtual objects based on the illumination source. In some embodiments, the shadow generation manager 170 is configured to modify the rendering parameters of the mesh on the display to only receive shadows on meshes that are otherwise transparent. In some embodiments, the shadow generation manager 170 is configured to render shadows after the initial camera feed is displayed but before any objects are shown.

[0049] The shadow data 172 represents the shadows cast on real and / or virtual objects in the physical environment. In some embodiments, the shadow data 172 includes a color value that is the product of the initial color value of the pixel in the grid representing the physical environment and the color value of the shadow. In some embodiments, the color value of the shadow is zero (black).

[0050] The mesh collider manager 180 is configured to generate mesh collider data 182 (e.g., a cooking collision mesh) that supports kinematic objects of arbitrary shapes by allowing virtual rigid body objects to collide with, bounce off of, and splash on real objects in the physical environment. In some embodiments, the mesh collider manager 180 is configured to generate mesh collider data 182 only when a rigid body is introduced or manipulated into the field of view (FoV) of a scene in the physical environment. In some embodiments, the mesh collider manager 180 is configured to extend the boundaries of the mesh collider toward the image plane of the camera to prevent the rigid body from disappearing from the display. In some embodiments, the mesh collider manager 180 is configured to calculate the normals of the collider mesh near a vertex using the cross product of the vectors formed by adjacent vertices.

[0051] The mesh collision volume data 182 represents a mesh collision volume for simulating collision physics in a physical environment between a kinematic virtual object (e.g., a rigid body) and a real object. The mesh collision volume data 182 includes mesh collision volume boundary data 182, which represents the boundary of the mesh collision volume that can extend toward the camera image plane.

[0052] The components (e.g., modules, processing unit 124) of the user device 120 can be configured to operate based on one or more platforms (e.g., one or more similar or different platforms), which can include one or more types of hardware, software, firmware, operating systems, and / or runtime libraries, etc. In some embodiments, the components of the computer 120 can be configured to operate within a device cluster (e.g., a server farm). In such embodiments, the functions and processing of the components of the computer 120 can be distributed to several devices of the device cluster.

[0053] The components of computer 120 may be or may include any type of hardware and / or software configured to process attributes. In some embodiments, one or more portions of the components shown in the components of computer 120 in FIG. 1 may be or may include hardware-based modules (e.g., digital signal processors (DSPs), field programmable gate arrays (FPGAs), memory), firmware modules, and / or software-based modules (e.g., computer code modules, sets of computer-readable instructions that can be executed on a computer). For example, in some embodiments, one or more portions of the components of computer 120 may be or may include software modules configured to be executed by at least one processor (not shown). In some embodiments, the functionality of the components may be included in different modules and / or different components than those shown in FIG. 1, including combining the functionality of two components illustrated as a single component.

[0054] Although not shown, in some embodiments, the components of the computer 120 (or its portion) may be configured to operate within, for example, a data center (e.g., a cloud computing environment), a computer system, and / or one or more server / host devices, etc. In some embodiments, the components of the computer 120 (or its portion) may be configured to operate within a network. Therefore, the components of the computer 120 (or its portion) may be configured to function in various types of network environments that may include one or more devices and / or one or more server devices. For example, the network may be or may include a local area network (LAN) and / or a wide area network (WAN), etc. The network may be or may include a wireless network and / or a wireless network implemented using, for example, a gateway device, a bridge, and / or a switch, etc. The network may include one or more segments and / or may have portions based on various protocols (e.g., Internet Protocol (IP) and / or proprietary protocols). The network may include at least a portion of the Internet.

[0055] In some embodiments, one or more components of the computer 120 may be or include a processor configured to process instructions stored in a memory. For example, the mesh manager 130 (and / or a portion thereof), the depth map manager 140 (and / or a portion thereof), the vertex shift manager 150 (and / or a portion thereof), the virtual object manager 160 (and / or a portion thereof), the shadow generation manager 170 (and / or a portion thereof), and the mesh collision manager 180 may be a combination of a processor and a memory configured to execute instructions associated with the process to implement one or more functions.

[0056] In some embodiments, the memory 126 can be any type of memory, such as random access memory, disk drive memory, and / or flash memory, among others. In some embodiments, the memory 126 can be implemented as more than one memory component (e.g., more than one RAM component or disk drive memory) associated with a component of the VR server computer 120. In some embodiments, the memory 126 can be a database memory. In some embodiments, the memory 126 can be or can include a non-local memory. For example, the memory 126 can be or can include a memory shared by multiple devices (not shown). In some embodiments, the memory 126 can be associated with a server device (not shown) within a network and configured to serve the components of the computer 120. As shown in FIG. 1 , the memory 126 is configured to store various data, including mesh data 132, depth map data 142, displaced vertex data 152, virtual object data 162, shadow data 172, and mesh collision volume data 182.

[0057] Figure 2 1 is a flowchart depicting an example method 200 for introducing a virtual object into a physical environment in an AR system according to the above-described improved technique. The method 200 may be performed by the software structure described in conjunction with FIG. 1 , which resides in the memory 126 of the computer 120 and is collectively executed by the processing unit 124 .

[0058] At 202, the mesh manager 130 generates a triangular mesh (e.g., mesh data 132) representing the physical environment and the depth map manager 140 generates a depth map (e.g., depth map data 142) of the physical environment, the triangular mesh including a plurality of vertices (e.g., vertex data 133) and the depth map including a plurality of depth values ​​(e.g., depth value data 143). In some embodiments, the mesh manager 130 generates a grid of regularly spaced vertices as a triangular mesh on the FoV of a camera viewing a scene in the physical environment. In some embodiments, the mesh manager 130 also generates a plurality of indices (e.g., index data 134) representing connectivity of the mesh. In some embodiments, the mesh manager 130 generates depth values ​​represented by the depth map data 142 along the direction of a ray emitted from an illumination source (e.g., camera / source data 144).

[0059] At 204, the vertex shift manager 150 performs a shift operation on a plurality of vertices of the triangle mesh to generate a plurality of shifted vertices (e.g., shifted vertex data 152), which represent a geometry of at least one real object within the physical environment, the shift operation being based on the depth map. In some embodiments, the vertex shift manager 150 is configured to remove a vertex from the triangle mesh when the depth map satisfies a criterion. In some embodiments, the criterion includes a difference greater than a threshold, the difference being between: (i) a depth value of the depth map corresponding to the vertex; and (ii) an average of the depth value of the depth map corresponding to the vertex and depth values ​​corresponding to a set of neighboring vertices in the plurality of vertices.

[0060] At 206, the virtual object manager 160 receives virtual object data (e.g., virtual object data 162) representing a virtual object that is configured to be displayed with at least one real object in a physical environment. In some embodiments, the virtual object is defined as having shape and texture parameter values. In some embodiments, the virtual object is defined using a mesh.

[0061] At 208, the computer 120 displays the virtual object in the physical environment on the display to generate a displayed virtual object, the displayed virtual object having a difference with the virtual object according to the plurality of displaced vertices. In one example, the displayed virtual object may be occluded by the real object. In another example, the displayed virtual object may have a shadow cast thereon by the real object or may cast a shadow on the real object. In another example, the displayed virtual object may be splashed after colliding with the real object.

[0062] Figure 3A 3 is a diagram illustrating a top view of an example scene within a physical environment 300 imaged in an AR system on a display 350. Figure 3AAs shown, physical environment 300 includes real objects 310(1) and 310(2). AR system has generated and inserted virtual object 320 in display 350 so as to appear to be arranged within virtual environment 300. In this case, virtual object 320 appears between real objects 310(1) and 310(2). Providing the perspective of the scene is camera 330 and illuminating the scene is illumination source 340. Based on the perspective of the camera, virtual object 320 should be occluded by real object 310(2).

[0063] Figure 3B 3 is a diagram illustrating a front view of a physical environment 300 imaged by a camera 330 within an AR system that does not use depth map information to define real objects 310(1) and 310(2). In this case, a virtual object 320 appears in front of the real object 310(2) even though it is placed between the real objects 310(1) and 310(2).

[0064] Figure 3C 3 is a diagram illustrating a front view of a physical environment 300 imaged by a camera 330 within an AR system that uses depth map information along rays emitted from an illumination source 340 to define real objects 310(1) and 310(2). In this case, virtual object 320 appears to be occluded by real object 310(2) because it is placed between real objects 310(1) and 310(2).

[0065] Figure 4A 4 is a diagram illustrating an example representation 400 of a physical environment in an AR system. The representation 400 includes a depth map 410 and a mesh template 420.

[0066] Depth map 410 is a perspective camera image that contains a depth value in each pixel instead of a color. Depth map 410 can be used directly in AR rendering, for example in a fragment shader to selectively hide parts of a virtual object that are occluded by a real object. Depth map 410 is shown as an array of pixels with various shades. Each shade represents a depth value. For example, if there are 128 gray levels, there are 128 integer possible depth values. Each pixel of depth map 410 corresponds to a position in a regular grid on the FoV of the scene in the physical environment.

[0067] The mesh template 420 is an initial mesh generated by the AR system on the FoV of the scene in the physical environment. Before any vertex shifting operation, the vertices of the mesh template 420 are arranged in a regular grid. Each vertex of the mesh template 420 corresponds to a pixel of the depth map 410.

[0068] Figure 4B4 is a diagram illustrating an example representation 400 of a physical environment in an AR system including connectivity of vertices of a template mesh 420. The connectivity between vertices representing scene depth values ​​(vertex connectivity) corresponds to the connectivity of the pixel grid of the depth map. Connectivity is represented by pairs of triangles 460 between quadruplets of vertices forming a square. The connectivity of the mesh template 420 is then represented by a series of triangles to form a triangle mesh 470.

[0069] Vertex connectivity does not change over time. The vertices of the template mesh 420 are stored in a vertex buffer. The connectivity of these vertices is represented in a triangle or index buffer, which contains a sequence of indices corresponding to the vertices in the vertex buffer. Each three indices in the index buffer describe a triangle primitive. The order within the triplet conforms to the winding order of the graphics framework in which the AR system runs. For example, a clockwise winding order renders outward-facing triangles. The vertex buffer and index buffer are initially stored in the memory of the CPU 192 and are copied to the memory of the GPU 194 only once during initialization.

[0070] Figure 5 is a diagram illustrating an example representation of a physical environment 500 having a mesh 510 with vertices displaced according to a depth map (e.g., depth map 410). Given the x and y pixel positions of the current depth pixel, the depth value, and the camera parameters as described in equations (1) and (2) above, the value of the new vertex position can be calculated in the vertex shader based on the camera pinhole model. No additional data transfer between the CPU 192 and the GPU 194 is required during rendering time, which provides greater efficiency for this method of introducing virtual objects into the physical environment in the AR system.

[0071] Figure 6 is a diagram illustrating an example process 600 of introducing a virtual object into a physical environment of an AR system.

[0072] Starting from start 602 , process 600 begins a step 612 of shifting vertices, which takes as input a template mesh 604 , a depth map 606 , depth source intrinsic parameters (ie, pinhole camera model) parameters 608 , and a time-delay adjusted six degrees of freedom (6DoF) pose 610 .

[0073] Vertex displacement 612 includes reprojection 614 of depth using depth map 606 and depth source intrinsic parameters 608. That is, the displacement of the vertices is based on the pinhole camera model parameter values. The depth values ​​are measured along the rays from the illumination source. Then, at 616, the displaced vertices are transformed from sensor space (i.e., a two-dimensional mesh) to world space (i.e., a three-dimensional mesh) based on the time-delay adjusted six degrees of freedom (6DoF) pose 610.

[0074] At 618, the process 600 generates an environment depth mesh. Based on the environment depth mesh 618, the process 600 sets custom FX shader and material parameters 620. The shader and material parameters are then input into a renderer 624 along with data 622 defining the virtual object.

[0075] Also from start 602, process 600 includes rendering 628 of a background based on RGB image 626. The output of rendering 628 and the output of rendering 624 are combined to produce a composite image 630. The composite image is input 632 into a graphics buffer and shown 634 on a display.

[0076] Figure 7 is a diagram illustrating an example process 700 for introducing shadows into the physical environment of an AR system.

[0077] Starting from start 702 , process 700 begins a step 712 of shifting vertices, which takes as input a template mesh 704 , a depth map 706 , depth source intrinsic parameters (ie, pinhole camera model) parameters 708 , and a time-delay adjusted six degrees of freedom (6DoF) pose 710 .

[0078] Vertex displacement 712 includes reprojection 714 of depth using depth map 706 and depth source intrinsic parameters 708. That is, the displacement of the vertices is based on the pinhole camera model parameter values. The depth values ​​are measured along the rays from the illumination source. Then, at 716, the displaced vertices are transformed from sensor space (i.e., a two-dimensional mesh) to world space (i.e., a three-dimensional mesh) based on the time-delay adjusted six degrees of freedom (6DoF) pose 710.

[0079] At 718, process 700 generates an environment depth mesh. Based on environment depth mesh 718, process 700 sets mesh rendering parameters to render a transparent mesh that receives shadows 720. The mesh rendering parameters are then input into a renderer 724 along with data defining a virtual object 722.

[0080] Also from start 702, process 700 includes rendering 728 of a background based on RGB image 726. The output of rendering 728 and the output of rendering 724 are combined to produce a composite image 730. The composite image is input 732 into a graphics buffer and shown 734 on a display.

[0081] Figure 8 is a diagram illustrating an example process 800 for introducing collision physics simulation into the physical environment of an AR system.

[0082] From start 802 and a trigger from user 803 (e.g., pressing a button or touching the screen), process 800 begins step 812 of shifting vertices, which takes as input a template mesh 804, a depth map 806, depth source intrinsic parameters (i.e., pinhole camera model) parameters 808, and a delay-adjusted six degrees of freedom (6DoF) pose 810.

[0083] Vertex displacement 812 includes reprojection 814 of the depth using the depth map 806 and the depth source intrinsic parameters 808. That is, the displacement of the vertices is based on the pinhole camera model parameter values. The depth values ​​are measured along the rays from the illumination source. Then, at 817, the displaced vertices are transformed from sensor space (i.e., a two-dimensional mesh) to world space (i.e., a three-dimensional mesh) based on the time-delay adjusted six degrees of freedom (6DoF) pose 810. However, if it is determined at 815 that the displaced vertex is on the boundary of the mesh, then at 816, the boundary is extended toward the image plane of the camera.

[0084] The vertex displacement 812 is repeated to create an ambient depth mesh 818. This mesh is used as input to a collision mesh cooking operation 819 that generates a mesh collider 820. The mesh collider 820 is then configured to operate a physics simulation 822.

[0085] Another result of the user trigger at 803 is that an instantiation 826 of a virtual mesh and a rigid body object defined by the virtual mesh is performed using additional virtual model prefab input 824. The parameter isKinematic (if enabled, it indicates that the rigid body object will not be driven by the physics engine, but can only be manipulated by transformations) is then set to false at 828. At 830, a virtual rigid body collider is generated; the virtual rigid body collider is fed into and out of the physics simulation 822. If it is determined at 832 that the rigid body is in a dormant state (i.e., the rigid body is not moving), the parameter isKinematic is set to true at 834. That is, in some embodiments, the rigid body is not awakened; this keeping the rigid body dormant can improve efficiency. Otherwise, the process 800 obtains the transformation of the rigid body object at 836 and generates a virtual body mesh at 838 using the input from the instantiation of the virtual mesh and rigid body object at 828. The virtual object mesh is rendered at 840.

[0086] Process 800 also includes rendering 844 of a background based on RGB image 842. The output of rendering 840 and the output of rendering 844 are combined to produce a composite image 846. The composite image is input 848 to a graphics buffer and shown 850 on a display.

[0087] Fig. 9Illustrated are examples of a general purpose computer device 900 and a general purpose mobile computer device 950 that may be used with the techniques described herein. Computer device 900 is one example configuration of computer 120 of FIG.

[0088] like Fig. 9 As shown, computing device 900 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 950 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are meant to be exemplary only and are not meant to limit the implementation of the inventions described and / or claimed in this document.

[0089] The computing device 900 includes a processor 902, a memory 904, a storage device 906, a high-speed interface 908 connected to the memory 904 and a high-speed expansion port 910, and a low-speed interface 912 connected to a low-speed bus 914 and the storage device 906. Each of the components 902, 904, 906, 908, 910 and 912 are interconnected using various buses and can be installed on a common motherboard or in other appropriate ways. The processor 902 can process instructions for execution within the computing device 900, including instructions stored on the memory 904 or the storage device 906, to display graphical information of a GUI on an external input / output device (such as a display 916 coupled to the high-speed interface 908). In other embodiments, multiple processors and / or multiple buses and multiple memories and memory types can be used as appropriate. In addition, multiple computing devices 900 can be connected, each device providing a portion of the necessary operations (for example, as a server group, a group of blade server groups, or a multi-processor system)

[0090] The memory 904 stores information within the computing device 900. In one embodiment, the memory 904 is one or more volatile memory units. In another embodiment, the memory 904 is one or more non-volatile memory units. The memory 904 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0091] The storage device 906 can provide mass storage for the computing device 900. In one embodiment, the storage device 906 can be or include: a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state storage device or device array, which device array includes devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that perform one or more methods (such as those described above) when executed. The information carrier is a computer or machine-readable medium, such as a memory 904, a storage device 906, or a memory on a processor 902.

[0092] The high-speed controller 908 manages bandwidth-intensive operations of the computing device 900, while the low-speed controller 912 manages less bandwidth-intensive operations. This allocation of functions is exemplary only. In one embodiment, the high-speed controller 908 is coupled to the memory 904, the display 916 (e.g., through a graphics processor or accelerator), and to the high-speed expansion port 910 that can accept various expansion cards (not shown). In an embodiment, the low-speed controller 912 is coupled to the storage device 906 and the low-speed expansion port 914. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices (such as keyboards, pointing devices, scanners) or networking devices (such as switches or routers), for example, through a network adapter.

[0093] The computing device 900 can be implemented in a variety of different forms, as shown. For example, it can be implemented as a standard server 920, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 924. In addition, it can be implemented in a personal computer, such as a laptop computer 922. Alternatively, components from the computing device 900 can be combined with other components in a mobile device such as device 950 (not shown). Each such device can contain one or more computing devices 900, 950, and the entire system can be composed of multiple computing devices 900, 950 that communicate with each other.

[0094] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to a storage system, at least one input device, and at least one output device to receive data and instructions from it and to transmit data and instructions to it.

[0095] These computer programs (also referred to as programs, software, software applications or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium", "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0096] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, voice, or tactile input.

[0097] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described in this specification), or includes any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0098] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship between client and server is generated by computer programs running on the respective computers and having a client-server relationship to each other.

[0099] A number of implementations have been described, however, it will be appreciated that various modifications can be made without departing from the spirit and scope of the description.

[0100] It should also be understood that when an element is referred to as being on, connected to, electrically connected to, coupled to, or electrically coupled to another element, it can be directly on, connected to, or coupled to another element, or one or more intermediate elements may be present. On the contrary, when an element is referred to as being directly on, directly connected to, or directly coupled to another element, there is no intermediate element. Although the term directly on, directly connected to, or directly coupled to may not be used throughout the detailed description, the element that is shown as being directly on, directly connected to, or directly coupled to may be referred to in this way. The claims of the present application may be modified to narrate the exemplary relationship described in the specification or shown in the accompanying drawings.

[0101] Although certain features of the described embodiments have been shown as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. Therefore, it should be understood that the appended claims are intended to cover all such modifications and variations that fall within the scope of the invention. It should be understood that they are presented only as examples and not limitations, and various changes in form and detail may be made. In addition to mutually exclusive combinations, any part of the apparatus and / or method described herein may be combined in any combination. The embodiments described herein may include various combinations and / or sub-combinations of the functions, components and / or features of the different implementations described.

[0102] In addition, the logic flows depicted in the figures do not require the particular order shown or sequential order to achieve the desired results. In addition, other steps can be provided, or steps can be removed from the described processes, and other components can be added to the described systems or removed therefrom. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A method for displaying a virtual object, comprising: Generate a triangular mesh representing a physical environment and a depth map of the physical environment, wherein the triangular mesh includes a plurality of vertices and the depth map includes a plurality of depth values; performing a shift operation on the plurality of vertices of the triangular mesh to produce a plurality of shifted vertices representing a geometry of at least one real object within the physical environment, performing the shift operation comprising removing vertices from the plurality of vertices of the triangular mesh in response to the depth map satisfying a criterion; receiving virtual object data representing the virtual object, the virtual object being configured to be displayed with the at least one real object in the physical environment; as well as The virtual object in the physical environment is displayed on a display to produce a displayed virtual object having a difference from the virtual object according to the plurality of displaced vertices.

2. The method according to claim 1, wherein: Performing the shift operation includes: Each of the plurality of vertices of the triangular mesh is displaced according to a pinhole camera model.

3. The method according to claim 1, wherein: The criterion includes a difference greater than a threshold, the difference being between: the depth value of the depth map corresponding to the vertex and an average of the depth value of the depth map corresponding to the vertex and the depth values ​​corresponding to a set of neighboring vertices in the plurality of vertices.

4. The method according to claim 1, wherein: The triangle mesh further includes a plurality of indices indicating connectivity of the triangle mesh, each of the plurality of indices corresponding to a respective vertex of the plurality of vertices of the triangle mesh, the connectivity of the triangle mesh remaining constant over time.

5. The method according to claim 4, wherein: The plurality of indices are arranged in a plurality of index triplets, each of the plurality of index triplets representing a triangle of the triangle mesh and arranged in an order representing a specified winding order.

6. The method according to claim 4, wherein: Generating the triangular mesh includes: storing the plurality of vertices and the plurality of indices of the triangle mesh in a first buffer in a memory of a central processing unit CPU; and The plurality of vertices and the plurality of indices are copied to a second buffer in a memory of a graphics processing unit GPU.

7. The method according to claim 1, wherein: Generating the depth map includes: The depth values ​​of the depth map are generated along rays emitted from light sources proximate to the physical environment.

8. The method of claim 1, further comprising, before receiving the virtual object data: rendering the triangular mesh with the plurality of displaced vertices as a transparent mesh to the display; and A first shadow is rendered on the transparent mesh based on the depth map.

9. The method according to claim 8, wherein: The virtual object displayed on the display in the physical environment includes: A second shadow is rendered on the triangular mesh based on the displayed virtual object.

10. The method according to claim 1, further comprising: A mesh collision volume is generated based on the triangular mesh, the mesh collision volume being configured to detect a collision between the at least one real object and the virtual object, the mesh collision volume comprising a set of vertices.

11. The method according to claim 10, further comprising: determining a field of view FOV of a camera displaying the virtual object in the physical environment; The boundaries of the mesh collision volume are extended in response to the virtual object moving out of the FOV of the camera.

12. The method according to claim 10, wherein: Generating the mesh collision body includes: A cross product of orthogonal vectors formed by adjacent vertices in the vertex set is calculated as a normal of the mesh collision volume near the vertex in the vertex set.

13. A computer program product comprising a non-transitory storage medium, the computer program product comprising code which, when executed by a processing circuit of a server computing device, causes the processing circuit to perform a method comprising: Generate a triangular mesh representing a physical environment and a depth map of the physical environment, wherein the triangular mesh includes a plurality of vertices and the depth map includes a plurality of depth values; performing a shift operation on the plurality of vertices of the triangular mesh to produce a plurality of shifted vertices representing a geometry of at least one real object within the physical environment, performing the shift operation comprising removing vertices from the plurality of vertices of the triangular mesh in response to the depth map satisfying a criterion; receiving virtual object data representing a virtual object, the virtual object being configured to be displayed with the at least one real object in the physical environment; as well as The virtual object in the physical environment is displayed on a display to produce a displayed virtual object having a difference from the virtual object according to the plurality of displaced vertices.

14. The computer program product of claim 13, wherein: The criterion includes a difference greater than a threshold, the difference being between: the depth value of the depth map corresponding to the vertex and an average of the depth value of the depth map corresponding to the vertex and the depth values ​​corresponding to a set of neighboring vertices in the plurality of vertices.

15. The computer program product of claim 13, wherein: The triangle mesh further includes a plurality of indices indicating connectivity of the triangle mesh, each of the plurality of indices corresponding to a respective vertex of the plurality of vertices of the triangle mesh, the connectivity of the triangle mesh remaining constant over time.

16. The computer program product of claim 13, wherein: Generating the depth map includes: The depth values ​​of the depth map are generated along rays emitted from light sources proximate to the physical environment.

17. The computer program product of claim 13, further comprising: A mesh collision volume is generated based on the triangular mesh, the mesh collision volume being configured to detect a collision between the at least one real object and the virtual object, the mesh collision volume comprising a set of vertices.

18. An electronic device, comprising: Memory; as well as A control circuit coupled to the memory, the control circuit being configured to: Generate a triangular mesh representing a physical environment and a depth map of the physical environment, wherein the triangular mesh includes a plurality of vertices and the depth map includes a plurality of depth values; performing a shift operation on the plurality of vertices of the triangular mesh to produce a plurality of shifted vertices representing a geometry of at least one real object within the physical environment, performing the shift operation comprising removing vertices from the plurality of vertices of the triangular mesh in response to the depth map satisfying a criterion; receiving virtual object data representing a virtual object configured to be displayed with the at least one real object in the physical environment; as well as The virtual object in the physical environment is displayed on a display to produce a displayed virtual object having a difference from the virtual object according to the plurality of displaced vertices.

19. The electronic device according to claim 18, wherein: The criterion includes a difference greater than a threshold, the difference being between: the depth value of the depth map corresponding to the vertex and an average of the depth value of the depth map corresponding to the vertex and the depth values ​​corresponding to a set of neighboring vertices in the plurality of vertices.

20. The electronic device according to claim 18, wherein: The triangle mesh further includes a plurality of indices indicating connectivity of the triangle mesh, each of the plurality of indices corresponding to a respective vertex of the plurality of vertices of the triangle mesh, the connectivity of the triangle mesh remaining constant over time.

Images