Systems, methods, and media for generating visualization of physical environment in artificial reality

Abstract

In one embodiment, a method performed by a computing system comprising a color camera and two monochromatic cameras respectively associated with two eyes of a user includes: computing a point cloud corresponding to a visible environment based at least on two stereoscopic grayscale images respectively acquired by the two monochromatic cameras; generating a grid corresponding to the visible environment based on the calculated point cloud; and generating two stereoscopically colorized images to be displayed to the two eyes of the user, respectively, where each of the two stereoscopically colorized images is generated using: (1) a mesh; (2) luminance information from one of the two stereoscopic grayscale images, the one stereoscopic grayscale image being associated with the eye to which the stereoscopic colorized image is to be displayed; and (3) color information from the color input image captured by the color camera.

Description

[0001] priority

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 378,949, filed on October 10, 2022, under 35 U.S.C. §119(e), which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to computer graphics and 3D reconstruction techniques. Background Art

[0004] Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content, or generated content combined with captured content (e.g., photos of the real world). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that gives the viewer a three-dimensional effect). Artificial reality may be associated with an application, product, accessory, service, or some combination thereof, which is used, for example, to create content in artificial reality and / or to be used in artificial reality (e.g., to perform activities in artificial reality).

[0005] Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including head-mounted devices (HMDs) connected to a host computer system, independent HMDs, mobile devices or computing systems, or any other hardware platform capable of providing artificial reality content to one or more viewers. When a user wears an HMD, the user's vision of the physical environment is blocked by the physical structure of the HMD. For example, the display of the HMD can be positioned directly in front of the user's eyes and very close to the user's eyes. Therefore, whenever the user needs to view the user's physical environment, the user needs to remove the HMD. Even if the removal of the HMD is temporary, doing so will cause inconvenience and interference to the user experience. Summary of the invention

[0006] "Passthrough" is a feature that allows a user wearing an HMD to view the user's physical environment by displaying visual information captured by the HMD's forward-facing camera. However, simply displaying the captured image does not achieve the desired effect. Because the position of the camera is inconsistent with the position of the user's eyes, the image captured by the camera does not accurately reflect the user's perspective. In addition, because the image has no depth, if the user moves away from the location where the image was taken, simply displaying the image will not provide the user with a proper parallax effect. Incorrect parallax combined with user motion may cause motion sickness. Therefore, in order to generate correct parallax, specific embodiments of the passthrough feature extract information about the environment (e.g., depth information) from the captured image, use this information to generate a 3D model (geometric scene representation) of the environment, and reconstruct the scene of the modeled environment from the user's current viewpoint.

[0007] Another problem solved by the perspective feature is the photon-to-vision latency. The delay between a photon hitting the camera and the photon appearing on the screen (as part of the captured image) determines the visual comfort of interacting in a dynamic world. Certain embodiments of the perspective feature overcome this problem by updating a 3D model representation of the environment based on images captured at a sufficiently high rate (e.g., 30 Hz, 60 Hz, etc.), and rendering the image based on the user's latest known head pose.

[0008] In order to meet performance requirements and system constraints (e.g., power, processing, memory, thermal budget), existing perspective systems can use monochrome cameras to capture the user's surroundings in grayscale mode. However, the resulting perspective experience is also in grayscale mode, which is not an ideal user experience.

[0009] In certain embodiments, a computing system may generate a colorized stereoscopic perspective image based on a plurality of stereoscopic monochrome images and a color input image. Current designs of artificial reality (e.g., AR / VR) devices may include image sensors (e.g., cameras) with only monochrome cameras to implement features such as head-mounted viewer (headset) tracking, depth detection, and monochrome perspective. However, in order to improve the user experience, users may want color perspective so that users can see images that are closer to the real world. Replacing the current image sensor with a monochrome camera with a color camera may be costly. Current algorithms (e.g., head-mounted viewer tracking, depth detection, etc.) may also rely on data received from a monochrome camera. In order to implement colorized stereoscopic perspective, the computing system disclosed herein may include two monochrome cameras and one color camera. Each monochrome camera may correspond to one eye. The computing system may calculate a point cloud corresponding to a visible environment based on at least two stereo grayscale images captured by corresponding monochrome cameras in the two monochrome cameras. In certain embodiments, the computing system may also include a structured light projector that projects an infrared (IR) pattern at a configured frequency. The two monochrome cameras may be synchronized with the structured light projector such that the two monochrome cameras capture IR monochrome images at a configured frequency, the IR monochrome images containing reflections of the projected IR pattern. Between two consecutive monochrome image captures when the structured light projector is active, each of the two monochrome cameras may capture j monochrome images, when the projector is inactive and the projected IR pattern reflections are not visible, where j is a preconfigured positive integer. The computing system may also compute a point cloud based on the reflections of the projected IR pattern in the IR monochrome images captured by the two monochrome cameras. In a particular embodiment, the computing system may subsample the points in the point cloud to reduce the number of points in the point cloud. In a particular embodiment, in two stereo grayscale images, when the first region has a higher gradient than the second region, the first sampling rate corresponding to the first region may be higher than the second sampling rate corresponding to the second region. In a particular embodiment, the computing system may filter the following points in the point cloud: the estimated likelihood of the point being a non-noise point is lower than a threshold likelihood associated with the position of the point. When a first position of the point cloud is within a first threshold distance from the two monochrome cameras and a second position exceeds the first threshold distance from the two monochrome cameras, a first threshold likelihood associated with the first position may be lower than a second threshold likelihood associated with the second position. In certain embodiments, the estimated likelihood that the point is a non-noise point may be estimated based on the number of times the point has been observed consecutively in a predetermined number of previous frames. The threshold likelihood associated with a position may be represented by a minimum number of times the point has been observed consecutively in a predetermined number of previous frames.

[0010] In a particular embodiment, the computing system may generate a mesh corresponding to the visible environment based on the calculated point cloud. In a particular embodiment, the computing system may generate multiple meshes corresponding to the visible environment based on the calculated point cloud. For example, the computing system may generate a mesh for each monochrome camera or each eye. To generate a mesh corresponding to the point cloud, the computing system may initialize the mesh corresponding to the point cloud as a planar mesh at a predetermined distance from two monochrome cameras. For each point in the point cloud, the computing system may perform the following steps to generate a three-dimensional mesh, and the portion of the three-dimensional mesh covering nearby objects is expanded. The computing system may identify the vertex corresponding to the point in the mesh. The computing system may warp the identified vertex to the position of the point in the point cloud. The computing system may determine whether the point in the point cloud is within a second threshold distance. When the computing system determines that the point in the point cloud is within the second threshold distance, the computing system may warp the k vertices closest to the identified vertex to the position of the point, where k is a predetermined positive integer.

[0011] In certain embodiments, a computing system may generate two stereo colorized images for display to two eyes of a user, respectively. Each of the two stereo colorized images may be generated using: (1) a mesh; (2) brightness information from one of two stereo grayscale images, the one stereo grayscale image being associated with the eye to which the stereo colorized image is to be displayed; and (3) color information from a color input image captured by a color camera. To render each vertex in the mesh for generating each of the two stereo colorized images, the computing system may identify a first corresponding pixel in the corresponding grayscale image by performing a raycast from the vertex to a corresponding monochrome camera that captured the corresponding grayscale image. The computing system may also identify a second corresponding pixel in the color input image by performing a raycast from the vertex to the color camera. The computing system may convert a first red, green, blue (RGB) value associated with the second corresponding pixel into second color information in a second color space. A brightness level indicator may be separate from one or more color information indicators in the second color space. In certain embodiments, the second color space may be a CIELAB color space in which the brightness level indicator may be orthogonal to the chromaticity indicator value. The computing system may replace the brightness level indicator in the second color information with a value corresponding to the brightness level associated with the first corresponding pixel in the corresponding grayscale image. The computing system may convert the second color information into a second RGB value. The computing system may render the vertex with the second RGB value.

[0012] In certain embodiments, the computing system may also include two displays.The computing system may present each of the two stereoscopic colorized images to a corresponding display of the two displays.

[0013] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited to these embodiments. A specific embodiment may include all or some of the multiple components, elements, features, functions, operations or steps of the embodiments disclosed herein, or may not include the components, elements, features, functions, operations or steps of the embodiments disclosed herein. The attached claims relating to methods, storage media, systems and computer program products particularly disclose embodiments according to the present invention, wherein any feature mentioned in one claim category (e.g., method) may also be claimed in another claim category (e.g., system). The dependencies or back references in the attached claims are selected only for formal reasons. However, any subject matter resulting from intentionally back-referenced any prior claim (especially multiple references) may also be claimed, so that any combination of the claims and their features is disclosed and can be claimed, regardless of the dependencies selected in the attached claims. The subject matter that can be claimed includes not only the combination of multiple features stated in the attached claims, but also any other combination of multiple features in the claims, wherein each feature mentioned in the claims can be combined with any other feature or combination of other features in the claims. Furthermore, any of the various embodiments and features described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or in any combination with any feature in the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1A An example artificial reality system worn by a user is shown in accordance with certain embodiments.

[0015] Figure 1B Examples of see-through features according to certain embodiments are shown.

[0016] Figure 2 An example logical architecture of a computing system for colorized stereoscopic viewing is shown.

[0017] Figure 3 An example diagram for generating a colorized stereoscopic perspective image is shown.

[0018] Figure 4 An example method for generating a colorized stereoscopic perspective image based on two grayscale images and one color image is shown.

[0019] Figure 5 An example network environment associated with a VR or social networking system is shown.

[0020] Figure 6 An example computer system is shown. DETAILED DESCRIPTION

[0021] "Perspective" is a feature that allows a user wearing an HMD to view the user's physical environment by displaying visual information captured by the HMD's forward-facing camera. However, simply displaying the captured image does not achieve the desired effect. Because the position of the camera is inconsistent with the position of the user's eyes, the image captured by the camera does not accurately reflect the user's perspective. In addition, because the image has no depth, if the user moves away from the location where the image was taken, simply displaying the image will not provide the user with a proper parallax effect. Incorrect parallax combined with user motion may cause motion sickness. Therefore, in order to generate correct parallax, specific embodiments of the perspective feature extract information about the environment (e.g., depth information) from the captured image, use this information to generate a 3D model (geometric scene representation) of the environment, and reconstruct the scene of the modeled environment from the user's current viewpoint.

[0022] Another problem solved by the perspective feature is the photon-to-vision latency. The delay between a photon hitting the camera and the photon appearing on the screen (as part of the captured image) determines the visual comfort of interacting in a dynamic world. Certain embodiments of the perspective feature overcome this problem by updating a 3D model representation of the environment based on images captured at a sufficiently high rate (e.g., 30 Hz, 60 Hz, etc.) and rendering the image based on the user's latest known head pose.

[0023] In order to meet performance requirements and system constraints (e.g., power, processing, memory, thermal budget), existing perspective systems can use monochrome cameras to capture the user's surroundings in grayscale mode. However, the resulting perspective experience is also in grayscale mode, which is not an ideal user experience.

[0024] In certain embodiments, a computing system may generate a colorized stereoscopic perspective image based on multiple stereoscopic monochrome images and a color input image. Current designs of artificial reality (e.g., AR / VR) devices may include image sensors (e.g., cameras) with only monochrome cameras to enable features such as head-mounted viewer tracking, depth detection, and monochrome perspective. However, in order to improve the user experience, users may want color perspective so that users can see images that are closer to the real world. Replacing the current image sensor with a monochrome camera with a color camera may be costly. Current algorithms (e.g., head-mounted viewer tracking, depth detection, etc.) may also rely on data received from a monochrome camera. In order to achieve colorized stereoscopic perspective, the computing system disclosed herein may include two monochrome cameras and one color camera. Each monochrome camera may correspond to one eye.

[0025] Figure 1A An example of an artificial reality system 100 worn by a user 102 is shown. In certain embodiments, the artificial reality system 100 may include a head mounted device (“HMD”) 104, a controller 106, and a computer 108. The HMD 104 may be worn over the user's eyes and provide visual content to the user 102 via an internal display (not shown). The HMD 104 may have two separate internal displays, one for each eye of the user 102. Figure 1A As shown, HMD 104 can completely cover the user's field of view. As the only provider of visual information for user 102, HMD 104 achieves the goal of providing an immersive artificial reality experience. However, one consequence of this is that user 102 will not be able to see the physical environment around him because the user's vision is blocked by HMD 104. Therefore, the perspective feature described herein is needed to provide the user with real-time visual information about the user's physical environment. HMD 104 can include several externally facing cameras 107A to 107C. In a specific embodiment, cameras 107A and 107B can be monochrome cameras, while camera 107C can be a color camera.

[0026] Figure 1BAn example of a perspective feature is shown. User 102 can be wearing HMD 104 and immersed in a virtual reality environment. A physical table 150 is located in the physical environment around user 102. However, since user 102's vision is blocked by HMD 104, user 102 cannot directly see table 150. In order to help the user perceive the physical environment around him while wearing HMD 104, the perspective feature uses, for example, the aforementioned externally facing cameras 107A to 107C to collect information about the physical environment. Although HMD 104 has three externally facing cameras 107A to 107C, any combination of cameras 107A to 107C can be used to perform the functions described herein. As an example and not limitation, camera 107A and camera 107B can be used to perform one or more functions described herein. The collected information can then be reprojected to the user 102 based on the user's viewpoint. In a particular embodiment in which the HMD 104 has a right display 160A for the user's right eye and a left display 160B for the user's left eye, the system 100 can respectively (1) render a re-projected view 150A of the physical environment for the right display 160A based on the viewpoint of the user's right eye and (2) render a re-projected view 150B of the physical environment for the left display 160B based on the viewpoint of the user's left eye.

[0027] Reference again Figure 1A , the HMD 104 may have an externally facing camera, e.g. Figure 1A 107A to 107C. Although only three forward-facing cameras 107A to 107C are shown, the HMD 104 can have any number of cameras facing in any direction (e.g., an upward-facing camera for capturing ceiling or room lighting, a downward-facing camera for capturing a portion of the user's face and / or body, a rear-facing camera for capturing a portion of the environment behind the user, and / or an internal camera for capturing the user's eye gaze for eye tracking purposes). The external-facing cameras are configured to capture the physical environment around the user, and such capture can be performed continuously to generate a series of frames (e.g., as a video).

[0028] A 3D representation may be generated based on depth measurements of a physical object observed by camera 107A and camera 107B. Depth may be measured in a variety of ways. In certain embodiments, depth may be calculated based on stereo images. For example, camera 107A and camera 107B may share overlapping fields of view and may be configured to capture images simultaneously. Thus, the same physical object may be captured simultaneously by camera 107A and camera 107B. For example, in an image captured by camera 107A, a particular feature of the object may appear at a pixel p AIn the image captured by camera 107B, the same feature may appear at another pixel p B The depth measurement system can use triangulation techniques to calculate the depth of the observed feature as long as it knows that the two pixels correspond to the same feature. For example, based on the position of camera 107A in 3D space and p A The pixel position relative to the field of view of camera 107A can be obtained from camera 107A and through pixel p A A similar line can be projected from another camera 107B and passes through pixel p B Since it is assumed that the two pixels correspond to the same physical feature, the two lines should intersect. The two intersecting lines, and the imaginary line drawn between the two cameras 107A and 107B, form a triangle that can be used to calculate the distance of the observed feature from the camera 107A or 107B or from the point in space where the observed feature is located.

[0029] In certain embodiments, the pose (e.g., position and orientation) of the HMD 104 within the environment may be required. For example, as the user 102 moves around in the virtual environment, in order to render an appropriate display to the user 102, the system 100 will need to determine the user's position and orientation at any time. Based on the pose of the HMD, the system 100 may also determine the viewpoint of any one of the cameras 107A to 107C or the viewpoint of any one of the user's eyes. In certain embodiments, the HMD 104 may be equipped with an inertial-measurement unit ("IMU"). The data generated by the IMU and the stereo images captured by the externally facing cameras 107A and 107B allow the system 100 to calculate the pose of the HMD 104 using, for example, simultaneous localization and mapping (SLAM) or other suitable techniques.

[0030] In certain embodiments, the artificial reality system 100 may also have one or more controllers 106 that enable the user 102 to provide input. The controller 106 may communicate with the HMD 104 or a separate computing unit 108 via a wireless connection or a wired connection. The controller 106 may have any number of buttons or other mechanical input mechanisms. In addition, the controller 106 may have an IMU so that the position of the controller 106 can be tracked. The controller 106 may also be tracked based on a predetermined pattern on the controller. For example, the controller 106 may have several infrared light emitting diodes (LEDs) or other known observable features that together form a predetermined pattern. Using a sensor or camera, the system 100 may be able to capture images of the predetermined pattern on the controller. Based on the orientation of these observed patterns, the system can calculate the position and orientation of the controller relative to the sensor or camera.

[0031] The artificial reality system 100 may also include a computer unit 108. The computer unit may be a stand-alone unit that is physically separated from the HMD 104, or may be integrated with the HMD 104. In an embodiment where the computer 108 is a separate unit, the computer may be communicatively coupled to the HMD 104 via a wireless link or a wired link. The computer 108 may be a high-performance device (e.g., a desktop or laptop computer), or a resource-constrained device (e.g., a mobile phone). High-performance devices may have a dedicated graphics processing unit (GPU) and a high-capacity or constant power supply. On the other hand, resource-constrained devices may not have a GPU and may have a limited battery capacity. Therefore, the algorithms that the artificial reality system 100 can actually use depend on the capabilities of its computer unit 108.

[0032] Figure 2An example logical architecture 200 of a computing system for colorized stereoscopic perspective is shown. The computing system 100 may include an RGB camera 210 and two monochrome cameras 220. The computing system 100 may also include an image signal processor (ISP) 212 that performs post-processing on the signal output from the RGB camera 210. In a particular embodiment, the computing system 100 may also include a structured light projector 230 that projects an infrared (IR) pattern. In a particular embodiment, the computing system 100 may include a display 270 positioned close to the user's eyes. The computing system 100 may include a sensor service 240, a system service 250, and an MR service 260 for colorized stereoscopic perspective. Each service may include one or more units that run an algorithm. The sensor service 240 may include an ISP control unit 241 that controls the ISP 212, and a sensor control unit 243 that controls the RGB camera 210, the monochrome camera 220, and the structured light projector 230. The system service 250 may include an RGB camera control unit 251 that controls the RGB camera 210 through the sensor control unit 243. The RGB camera control unit 251 may also receive color image data captured by the RGB camera 210. The system service 250 may also include a monochrome camera control unit 253, which controls the monochrome camera 220 through the sensor control unit 243. The monochrome camera control unit 253 may also receive grayscale image data captured by the monochrome camera 220. The system service 250 may also include a structured light projector pattern control unit 255, which controls the structured light projector 230 through the sensor control unit 243. The system service may also include a depth calculation unit 257, which calculates a point cloud based on the IR stereo monochrome image and / or the non-IR stereo monochrome image captured by the monochrome camera 220. The system service may also include a mesh calculation unit 259, which calculates a three-dimensional mesh based on the point cloud generated by the depth calculation unit 257. The MR service 260 may include a perspective rendering unit 262 that renders a colorized stereoscopic perspective image based on the grid generated by the grid computing unit 259 and the color image captured by the RGB camera 210 and the stereo grayscale image captured by the monochrome camera 220. The colorized stereoscopic perspective image may be presented on the display 270. Although the present disclosure describes a specific logical architecture of a computing system for colorized perspective, the present disclosure contemplates any suitable logical architecture of a computing system for colorized perspective.

[0033] Perspectives consist of sensitive data (i.e., camera images, 3D meshes, and rendered images). The software architecture ensures data privacy in the following ways:

[0034] Hidden from client applications. Sensitive data is generated / consumed within MR service 260 and system service 250. There is no API for client applications to access images, rendered eye buffers, and 3D data. Sensitive data is never provided to the client application's storage space, but is retained in the system service.

[0035] Sensitive data can be generated by the camera / intermediate algorithms and consumed on the fly. No data can be saved on disk.

[0036] Sensitive data cannot be sent to the server as telemetry data. Only non-sensitive telemetry data consisting of calculation rate and latency is sent.

[0037] In certain embodiments, computing system 100 may include one color camera 210 and two monochrome cameras 220. Each monochrome camera may correspond to one eye. Figure 3 An example diagram for generating a colorized stereoscopic perspective image is shown. The computing system 100 can calculate a point cloud corresponding to a visible environment based on at least two stereo grayscale images 312 and 314 respectively captured by corresponding monochrome cameras of the two monochrome cameras 220. As an example and not a limitation, the depth calculation unit 257 can generate a point cloud based on the stereo monochrome images 312 and 314 captured by the monochrome camera 220. The acquisition rate of the monochrome camera 220 can be controlled by the monochrome camera control unit 253. In a specific embodiment, the monochrome camera 220 can capture stereo monochrome images at, for example, 30 Hz. The depth calculation unit 257 can use a portion of these frames to generate a point cloud. Another portion of these frames can be used by the perspective rendering unit 262 for perspective texture. For example, the portion of these frames captured by the monochrome camera 220 for point cloud computing ("depth frame") can be every odd frame (e.g., 15 Hz), and the portion of these frames captured by the monochrome camera 220 for perspective texture ("texture frame") can be every even frame (e.g., 15 Hz). Although this disclosure describes computing a point cloud based on stereo grayscale images in a particular manner, this disclosure contemplates computing a point cloud based on stereo grayscale images in any suitable manner.

[0038] In a particular embodiment, the computing system 100 may include a structured light projector 230 that projects an IR pattern at a frequency configured by the structured light projector pattern control unit 255 through the sensor control unit 243. The sensor control unit 243 may synchronize two monochrome cameras 220 with the structured light projector 230 so that the two monochrome cameras 220 capture IR monochrome images containing reflections of the projected IR pattern at the configured frequency. Between two consecutive monochrome image captures when the structured light projector is active, each of the two monochrome cameras may capture j monochrome images, where the projector is inactive and the projected IR pattern reflections are not visible, where j is a preconfigured positive integer. The computing system 100 may also calculate a point cloud based on the reflections of the projected IR pattern in the IR monochrome images captured by the two monochrome cameras 220. As an example and not limitation, the monochrome camera 220 may be configured to capture three non-IR image frames between two consecutive IR image frames, where the IR images indicate images containing reflections of the projected IR pattern and the non-IR images indicate images without reflections of the projected IR pattern. IR image acquisition can be synchronized with the projection of the IR pattern. Therefore, the IR image can contain reflections of the projected IR pattern. The depth calculation unit 257 can also use the IR image frame containing the reflection of the IR pattern to calculate the point cloud. Although the present disclosure describes that the point cloud is calculated based on the IR image containing the reflection of the projected IR pattern in a specific manner, the present disclosure contemplates calculating the point cloud based on the IR image containing the reflection of the projected IR pattern in any suitable manner.

[0039] In a particular embodiment, the computing system 100 may subsample the points in the point cloud to reduce the number of points in the point cloud. In a particular embodiment, in two stereo grayscale images, when the first area has a higher gradient than the second area, the first sampling rate corresponding to the first area may be higher than the second sampling rate corresponding to the second area. As an example and not limitation, after generating the point cloud, the depth calculation unit 257 may subsample the points from the generated point cloud to reduce the number of points in the point cloud from 30,000 points to between 6,000 and 15,000 points by retaining points in high contrast areas. High contrast areas may be more likely to belong to edges. Subsampling can reduce the number of points to a number that is easy to handle, so that subsequent temporal outlier suppression steps can be performed within an acceptable latency range. Although the present disclosure describes subsampling points in a point cloud in a particular manner, the present disclosure contemplates subsampling points in a point cloud in any suitable manner.

[0040] In certain embodiments, computing system 100 may filter points in a point cloud whose estimated likelihood of being a non-noise point is lower than a threshold likelihood associated with the location of the point. In certain embodiments, the likelihood of a point being a non-noise point may be estimated based on temporal consistency in the most recent previous frames. The estimated likelihood of a point being a non-noise point may be determined based on the number of times a point is observed consecutively in a predetermined number of previous frames. When the point has appeared in the most recent previous frames, the likelihood of the point being a non-noise point is high. The threshold likelihood associated with a location may be represented by a minimum number of times the point is observed consecutively in a predetermined number of previous frames. In certain embodiments, when a first location of the point cloud is within a first threshold distance from two monochrome cameras and a second location exceeds the first threshold distance from the two monochrome cameras, a first threshold likelihood associated with the first location may be lower than a second threshold likelihood associated with the second location. A nearby object may move faster than another object farther from the camera. Therefore, a point corresponding to a nearby object may not have been observed for a long time in the previous frames. Computing system 100 may cause points within the first threshold distance to have a low threshold likelihood. By way of example and not limitation, the first threshold distance may be 60 cm. Typically, the user's hand can be observed within 60 cm from the monochrome camera 220. The depth calculation unit 257 can check previous frames to determine whether to filter out points from the point cloud of the current frame. If a point has been observed in two consecutive frames in the multiple previous frames examined, the depth calculation unit 257 can record an upvote for the point. When a point is observed in one frame and is not observed in an immediately adjacent frame, the depth calculation unit 257 can record a downvote for the point. For example, when a point has six or more upvotes in the last 30 previous frames, the depth calculation unit 257 can keep the point within 60 cm. When a point has 20 or more upvotes in the last 30 previous frames, the depth calculation unit 257 can keep the point outside 60 cm. In a specific embodiment, if the downvotes of a point exceed a threshold number associated with the area of ​​the point, the depth calculation unit 257 can discard the point from the point cloud. Although this disclosure describes filtering points in a point cloud based on temporal consistency in a particular manner, this disclosure contemplates filtering points in a point cloud based on temporal consistency in any suitable manner.

[0041] In certain embodiments, the computing system 100 may generate a mesh 322 corresponding to the visible environment based on the calculated point cloud. When a portion of the mesh 322 corresponds to a foreground object, the computing system 100 may expand the portion so as to be able to more accurately represent the foreground object. To generate the mesh 322 corresponding to the point cloud, the computing system 100 may initialize the mesh 322 corresponding to the point cloud as a planar mesh at a predetermined distance from the two monochrome cameras 220. For each point in the point cloud, the computing system 100 may perform the following steps to generate a three-dimensional mesh 322, the portion of which covering nearby objects is expanded. The computing system 100 may identify the vertex corresponding to the point in the mesh 322. The computing system 100 may warp the identified vertex to the position of the point in the three-dimensional space. The computing system 100 may determine whether the point in the point cloud is within a second threshold distance. When the computing system 100 determines that the point is within the second threshold distance, the computing system 100 may warp the k vertices closest to the identified vertex to the position of the point in the three-dimensional space, where k is a predetermined positive integer. In certain embodiments, the computing system 100 may generate a plurality of meshes corresponding to the visible environment based on the calculated point cloud. For example, the computing system 100 may generate a mesh for each monochrome camera or each eye. After generating a single mesh 322 based on the calculated point cloud, the computing system 100 may distort the mesh 322 to generate two distorted meshes: one for the left eye (i.e., the distorted mesh represents what the left monochrome camera would observe), and another distorted mesh for the right eye (i.e., the distorted mesh represents what the right monochrome camera would observe). As an example and not a limitation, the mesh computing unit 259 may initialize a planar mesh at 2 meters from the monochrome camera 220. For each point in the point cloud, the mesh computing unit 259 may distort the corresponding vertex in the mesh 322 toward the point in the three-dimensional space. When a point is within 60 cm of the monochrome camera 220, the mesh computing unit 259 may also distort the four closest vertices of the corresponding point in the mesh toward the position of the point in the three-dimensional space. In an embodiment, the four closest vertices may be distorted to a position close to the point. The effect of these steps is to distort larger mesh patches corresponding to nearby objects.Although this disclosure describes generating a three-dimensional mesh based on a point cloud in a particular manner, this disclosure contemplates generating a three-dimensional mesh based on a point cloud in any suitable manner.

[0042] In certain embodiments, the computing system 100 can generate two stereo colorized images to be displayed to the user's two eyes, respectively. Each of the two stereo colorized images can be generated using: (1) the grid 322; (2) brightness information from one of the two stereo grayscale images 332 and 334, the one stereo grayscale image being associated with the eye to which the stereo colorized image is to be displayed; and (3) color information from the color input image 336 captured by the color camera 210. The stereo colorized images are "stereo" because the rendering uses brightness or grayscale intensity for the left image from the left monochrome camera and the right image from the right monochrome camera. If the rendering is not performed using brightness or grayscale intensity, the parallax effect and immersion of the output colorized images may be reduced. The stereo colorized images are "colorized" because the grayscale intensity is mixed with the chroma component of the color input image 336 captured by the color camera 210. To render each vertex in the mesh 322 to generate each of the two stereo colorized images, the computing system 100 may identify a first corresponding pixel in the corresponding grayscale image by performing a ray cast from the vertex to a corresponding monochrome camera that captured the corresponding grayscale image. The computing system 100 may also identify a second corresponding pixel in the color input image by performing a ray cast from the vertex to the color camera. By way of example and not limitation, to render the left colorized image 342, the perspective rendering unit 262 may perform a ray cast from a position corresponding to a vertex in the mesh 322 to the left monochrome camera to identify a first pixel in the left grayscale image 332 corresponding to the vertex of the left colorized perspective image. The perspective rendering unit 262 may also perform a ray cast from a position corresponding to the same vertex in the mesh 322 to the RGB camera 210 to identify a second pixel in the color input image 336 corresponding to the vertex of the left colorized perspective image. Although the present disclosure describes identifying a pixel from a corresponding grayscale image and identifying a pixel from a color input image in a particular manner for use in blending color information of vertices in a mesh, the present disclosure contemplates identifying a pixel from a corresponding grayscale image and identifying a pixel from a color input image in any suitable manner for use in blending color information of vertices in a mesh.

[0043] In a particular embodiment, the computing system 100 may convert the first RGB value associated with the second corresponding pixel into second color information in a second color space. The brightness level indicator may be separate from one or more color information indicators in the second color space. In a particular embodiment, the second color space may be a CIELAB color space, wherein the brightness level indicator may be orthogonal to the chromaticity indicator value. In the CIELAB color space, L* represents brightness, a* represents red / green value, and b* represents blue / yellow value. The computing system 100 may replace the brightness level indicator in the second color information with a value corresponding to the brightness level associated with the first corresponding pixel in the corresponding grayscale image. The computing system 100 may convert the second color information into a second RGB value. The computing system 100 may render the vertex with the second RGB value. As an example and not limitation, continuing with the previous example, the perspective rendering unit 262 may convert the RGB value of the second pixel in the color input image 336 into a CIELAB color space value, wherein L* indicates the brightness of the second pixel in the color input image 336. The perspective rendering unit 262 may replace the L* value in the converted CIELAB color space value with a value indicating the brightness level of the first pixel in the left grayscale image 332. The perspective rendering unit 262 may convert the CIELAB color space value into an RGB color space value. The output RGB color space value may include brightness information of the first pixel from the left grayscale image 332 and color information of the second pixel from the color input image 336. The perspective rendering unit 262 may use the output RGB color space value to render the vertices in the mesh 322 to generate the left colorized perspective image 342. The right colorized perspective image 344 may be generated in a similar manner using the right grayscale image 334 and the color input image 336. Although the present disclosure describes rendering the vertices of the mesh using brightness information from the grayscale input image and color information from the color input image in a particular manner, the present disclosure contemplates rendering the vertices of the mesh using brightness information from the grayscale input image and color information from the color input image in any suitable manner.

[0044] In certain embodiments, computing system 100 may also include two displays 270, each display 270 corresponding to one eye. Computing system 100 may present each of the two stereoscopic colorized images to a corresponding display of the two displays. As an example and not limitation, the left colorized image may be presented on the left display of the two displays 270. Additionally, the right colorized image may be presented on the right display of the two displays 270. Although the present disclosure describes presenting the stereoscopic colorized images to the two displays corresponding to the user's eyes in a particular manner, the present disclosure contemplates presenting the stereoscopic colorized images to the two displays corresponding to the user's eyes in any suitable manner.

[0045] Figure 4 An example method 400 for generating a colorized stereoscopic perspective image based on two grayscale images and one color image is shown. The method may begin at step 410, in which a computing system including a color camera and two monochrome cameras respectively associated with two eyes of a user may compute a point cloud corresponding to a visible environment based on at least two stereo grayscale images respectively captured by the two monochrome cameras. In a particular embodiment, the computing system may be embodied as an artificial reality system. At step 420, the computing system may generate a mesh corresponding to the visible environment based on the computed point cloud. At step 430, the computing system may generate two stereoscopic colorized images to be displayed to the two eyes of the user, respectively, wherein each of the two stereoscopic colorized images is generated using: (1) a mesh; (2) brightness information from one of the two stereoscopic grayscale images, the one stereoscopic grayscale image being associated with the eye to which the stereoscopic colorized image is to be displayed; and (3) color information from a color input image captured by the color camera.

[0046] Where appropriate, specific examples may be repeated Figure 4 Although the present disclosure will Figure 4 The specific steps of the method are described and shown as being performed in a particular order, but the present disclosure contemplates Figure 4 Any suitable steps of the method are performed in any suitable order. In addition, although the present disclosure describes and illustrates a method for generating a colorized stereoscopic perspective image based on two grayscale images and one color image, including Figure 4 However, where appropriate, the present disclosure contemplates any suitable method for generating a colorized stereoscopic perspective image based on two grayscale images and one color image, including any suitable steps, which may include Figure 4 In addition, although the present disclosure describes and illustrates the implementation of Figure 4 specific components, devices or systems for specific steps of the method, but the present disclosure contemplates performing Figure 4 Any suitable combination of any suitable components, devices, or systems of any suitable steps of the method.

[0047] Figure 5 An example network environment 500 associated with a virtual reality system is shown. The network environment 500 includes a user 501 interacting with a client system 530, a social networking system 560, and a third-party system 570, which are connected to each other via a network 510. Figure 5A particular arrangement of user 501, client system 530, social networking system 560, third-party system 570, and network 510 is shown, but the present disclosure contemplates any suitable arrangement of user 501, client system 530, social networking system 560, third-party system 570, and network 510. By way of example and not limitation, two or more of user 501, client system 530, social networking system 560, and third-party system 570 may be directly connected to each other, bypassing network 510. As another example, two or more of client system 530, social networking system 560, and third-party system 570 may be physically or logically co-located in whole or in part with each other. Furthermore, although Figure 5 A particular number of users 501, client systems 530, social networking systems 560, third-party systems 570, and networks 510 are shown, but the present disclosure contemplates any suitable number of client systems 530, social networking systems 560, third-party systems 570, and networks 510. By way of example and not limitation, network environment 500 may include a plurality of users 501, a plurality of client systems 530, a plurality of social networking systems 560, a plurality of third-party systems 570, and a plurality of networks 510.

[0048] The present disclosure contemplates any suitable network 510. By way of example and not limitation, one or more portions of network 510 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of a Public Switched Telephone Network (PSTN), a cellular telephone network, or a combination of two or more of these networks. Network 510 may include one or more networks 510.

[0049] Links 550 may connect client systems 530, social networking systems 560, and third-party systems 570 to communication network 510, or may connect client systems 530, social networking systems 560, and third-party systems 570 to each other. The present disclosure contemplates any suitable links 550. In particular embodiments, one or more links 550 include one or more wired (e.g., Digital Subscriber Line (DSL) or Data Over Cable Service Interface Specification (DOCSIS)) links, one or more wireless (e.g., Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)) links, or one or more optical (e.g., Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)) links. In certain embodiments, one or more links 550 each include an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a network based on cellular technology, a network based on satellite communication technology, another link 550, or a combination of two or more such links 550. The multiple links 550 need not all be identical throughout the network environment 500. In one or more aspects, the one or more first links 550 can be different from the one or more second links 550.

[0050] In certain embodiments, client system 530 may be an electronic device that includes hardware, software, or embedded logic components or a combination of two or more such components and is capable of performing appropriate functions implemented or supported by client system 530. By way of example and not limitation, client system 530 may include a computer system such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer, an e-book reader, a GPS device, a camera, a personal digital assistant (PDA), a handheld electronic device, a cellular phone, a smart phone, a virtual reality head-mounted viewer and controller, other suitable electronic devices, or any suitable combination thereof. The present disclosure contemplates any suitable client system 530. Client system 530 may enable network users at client system 530 to access network 510. Client system 530 may enable its users to communicate with other users at other client systems 530. Client system 530 may generate a virtual reality environment for users to interact with content.

[0051] In certain embodiments, the client system 530 may include a virtual reality (or augmented reality) head mounted view 532 (e.g., OCULUS RIFT and similar devices, etc.), and one or more virtual reality input devices 534 (e.g., virtual reality controllers). A user at the client system 530 may wear the virtual reality head mounted view 532 and use the one or more virtual reality input devices to interact with a virtual reality environment 536 generated by the virtual reality head mounted view 532. Although not shown, the client system 530 may also include a separate processing computer, and / or any other component of the virtual reality system. The virtual reality head mounted view 532 may generate a virtual reality environment 536, which may include system content 538 (including but not limited to an operating system), such as software or firmware updates, and the virtual reality environment 536 may also include third-party content 540, such as content from an application or content dynamically downloaded from the Internet (e.g., web content). The virtual reality head mounted view 532 may include one or more sensors 542 (e.g., accelerometers, gyroscopes, magnetometers) to generate sensor data that tracks the position of the head mounted view device 532. The head-mounted view device 532 may also include an eye tracker for tracking the position of the user's eyes or the direction in which they are viewed. The client system may use data from the one or more sensors 542 to determine the speed, orientation, and gravity relative to the head-mounted view device. One or more virtual reality input devices 534 may include one or more sensors 544 (e.g., accelerometers, gyroscopes, magnetometers, and touch sensors) to generate sensor data that tracks the position of the input device 534 and the position of the user's fingers. The client system 530 may use outside-in tracking, in which a tracking camera (not shown) is placed outside the virtual reality head-mounted view device 532 and within the line of sight of the virtual reality head-mounted view device 532. In outside-in tracking, the tracking camera may track the position of the virtual reality head-mounted view device 532 (e.g., by tracking one or more infrared light emitting diode (LED) markers on the virtual reality head-mounted view device 532). Alternatively or additionally, the client system 530 can utilize inside-out tracking, in which a tracking camera (not shown) can be placed on the virtual reality headset 532 or inside the virtual reality headset 532 itself. In inside-out tracking, the tracking camera can capture images of the real world surrounding the tracking camera, and can use the changing perspective of the real world to determine the position of the tracking camera itself in space.

[0052] In certain embodiments, a client system 530 (e.g., an HMD) may include a perspective engine 546 to provide the perspective features described herein, and the client system 530 may have one or more add-ons, plug-ins, or other extensions. A user at the client system 530 may connect to a particular server (e.g., server 562 or a server associated with a third-party system 570). The server may accept requests and communicate with the client system 530.

[0053] The third-party content 540 may include a web browser (e.g., MICROSOFT INTERNET EXPLORER, GOOGLE CHROME, or MOZILLA FIREFOX), and may have one or more add-ons, plug-ins, or other extensions (e.g., a tool bar or a YAHOO TOOLBAR). A user at the client system 530 may enter a Uniform Resource Locator (URL) or other address that directs the web browser to a particular server (e.g., server 562 or a server associated with the third-party system 570), and the web browser may generate a Hyper Text Transfer Protocol (HTTP) request and transmit the HTTP request to the server. The server may accept the HTTP request and transmit one or more Hyper Text Markup Language (HTML) files to the client system 530 in response to the HTTP request. The client system 530 may render a web interface (e.g., a web page) based on the HTML file from the server for presentation to the user. The present disclosure contemplates any suitable source file. As an example and not limitation, a web interface may be rendered based on an HTML file, an Extensible Hyper Text Markup Language (XHTML) file, or an Extensible Markup Language (XML) file, depending on specific needs. These interfaces may also execute scripts, such as but not limited to those written in JAVASCRIPT, JAVA, MICROSOFT SILVERLIGHT, and a combination of markup languages ​​and scripts such as AJAX (e.g., asynchronous JAVASCRIPT and XML). In this document, where appropriate, a reference to a web interface includes one or more corresponding source files (which a browser may use to render the web interface), and vice versa.

[0054] In certain embodiments, social networking system 560 may be a network addressable computing system that may host an online social network. Social networking system 560 may generate, store, receive, and transmit social networking data, such as user profile data, concept profile data, social graph information, or other suitable data related to an online social network. Other components of network environment 500 may access social networking system 560 directly or via network 510. By way of example and not limitation, client system 530 may access social networking system 560 directly or via network 510 using a web browser of third-party content 540 or a local application associated with social networking system 560 (e.g., a mobile social networking application, a messaging application, another suitable application, or any combination thereof). In certain embodiments, social networking system 560 may include one or more servers 562. Each server 562 may be a single server or a distributed server across multiple computers or multiple data centers. The server 562 may be of various types, such as, but not limited to, a web server, a news server, a mail server, a message server, an advertising server, a file server, an application server, an exchange server, a database server, a proxy server, another server suitable for performing the functions or processes described herein, or any combination thereof. In a particular embodiment, each server 562 may include hardware, software, or embedded logic components or a combination of two or more such components for performing the appropriate functions implemented or supported by the server 562. In a particular embodiment, the social networking system 560 may include one or more data repositories 564. The data repository 564 may be used to store various types of information. In a particular embodiment, the information stored in the data repository 564 may be organized according to a particular data structure. In a particular embodiment, each data repository 564 may be a relational database, a columnar database, a correlation database, or other suitable database. Although the present disclosure describes or illustrates a particular type of database, the present disclosure contemplates any suitable type of database. A particular embodiment may provide an interface that enables the client system 530, the social networking system 560, or the third-party system 570 to manage, retrieve, modify, add, or delete information stored in the data repository 564.

[0055] In certain embodiments, the social networking system 560 may store one or more social graphs in one or more data stores 564. In certain embodiments, the social graph may include a plurality of nodes, which may include a plurality of user nodes (each corresponding to a particular user) or a plurality of concept nodes (each corresponding to a particular concept) and a plurality of edges connecting the nodes. The social networking system 560 may provide users of the online social network with the ability to communicate and interact with other users. In certain embodiments, a user may join an online social network through the social networking system 560, and may subsequently add connections (e.g., relationships) with a plurality of other users in the social networking system 560 with whom the user wants to establish connections. In this document, the term "friend" may refer to any other user in the social networking system 560 with whom the user has formed a connection, association, or relationship through the social networking system 560.

[0056] In particular embodiments, social networking system 560 may provide users with the ability to take actions on various types of items or objects supported by social networking system 560. By way of example and not limitation, these items and objects may include groups or social networks to which users of social networking system 560 may belong, events or calendar entries that may be of interest to the user, computer-based applications that the user may use, transactions that allow the user to buy or sell multiple items through the service, interactions with advertisements that the user may perform, or other suitable items or objects. A user may interact with anything that can be represented in social networking system 560, or with anything that can be represented by an external system such as third-party system 570 that is separate from social networking system 560 and coupled to social networking system 560 via network 510.

[0057] In certain embodiments, social networking system 560 may be able to link various entities. By way of example and not limitation, social networking system 560 may enable multiple users to interact with each other and receive content from third-party systems 570 or other entities, or may allow users to interact with these entities through an application programming interface (API) or other communication channels.

[0058] In certain embodiments, third-party systems 570 may include one or more types of servers, one or more data repositories, one or more interfaces (including, but not limited to, APIs), one or more web services, one or more content sources, one or more networks, or any other suitable components (e.g., components with which a server can communicate). Third-party systems 570 may be operated by a different entity than the entity that operates social networking system 560. However, in certain embodiments, social networking system 560 and third-party systems 570 may operate in conjunction with each other to provide social networking services to users of social networking system 560 or users of third-party systems 570. In this sense, social networking system 560 may provide a platform or backbone that other systems (e.g., third-party systems 570) may use to provide social networking services and functionality to users on the Internet.

[0059] In certain embodiments, third-party system 570 may include a third-party content object provider. The third-party content object provider may include one or more content object sources that may be transmitted to client system 530. By way of example and not limitation, content objects may include information related to things or activities of interest to the user, such as movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. By way of another example and not limitation, content objects may include incentive content objects, such as coupons, discount tickets, gift certificates, or other suitable incentive objects.

[0060] In certain embodiments, social networking system 560 also includes user-generated content objects that can enhance a user's interaction with social networking system 560. User-generated content can include any content that a user can add, upload, send, or "post" to social networking system 560. By way of example and not limitation, a user transmits a post from client system 530 to social networking system 560. Posts can include data such as status updates or other textual data, location information, photos, videos, links, music, or other similar data or media. Content can also be added to social networking system 560 by third parties through "communication channels" (e.g., news feeds or streams).

[0061] In certain embodiments, the social networking system 560 may include various servers, subsystems, programs, modules, logs, and data repositories. In certain embodiments, the social networking system 560 may include one or more of the following: a web server, an action logger, an API request server, a relevance and ranking engine, a content object classifier, a notification controller, an action log, a third-party content object exposure log, an inference module, an authorization / privacy server, a search module, an advertising targeting module, a user interface module, a user profile repository, a contact repository, a third-party content repository, or a location repository. The social networking system 560 may also include suitable components, such as network interfaces, security mechanisms, load balancers, failover servers, management and network operation consoles, other suitable components, or any suitable combination thereof. In certain embodiments, the social networking system 560 may include one or more user profile repositories for storing user profiles. User profiles may include, for example, biometric information, personal background information, behavioral information, social information, or other types of descriptive information (e.g., work experience, educational history, hobbies or preferences, interests, in-law relationships, or locations). Interest information may include interests associated with one or more categories. Categories may be general or specific. As an example and not limitation, if a user "likes" an item about a shoe brand, the category may be the brand, or may be the general category "shoes" or "clothing". A contact repository may be used to store contact information about a user. The contact information may indicate users who have similar or common work experience, group membership, hobbies, educational history, or are related or share common attributes in any way. The contact information may also include user-defined connections between different users and content (both internal and external). A web server may be used to link the social networking system 560 to one or more client systems 530 or one or more third-party systems 570 via a network 510. The web server may include a mail server or other messaging functionality for receiving and routing messages between the social networking system 560 and one or more client systems 530. The API request server may allow a third-party system 570 to access information from the social networking system 560 by calling one or more APIs. An action logger may be used to receive communications from a web server about a user's actions on or outside the social networking system 560. Along with the action log, a third-party content object log that a user opens to third-party content objects may be maintained. The notification controller can provide information about the content object to the client system 530. The information can be pushed to the client system 530 as a notification, or can be pulled from the client system 530 in response to a request received from the client system 530.The authorization server may be used to enforce one or more privacy settings of users of the social networking system 560. The privacy settings of the user determine how specific information associated with the user may be shared. The authorization server may allow the user to choose to have their actions recorded by the social networking system 560 or shared with other systems (e.g., third-party systems 570), for example, by setting appropriate privacy settings. A third-party content object repository may be used to store content objects received from a third party (e.g., third-party system 570). A location repository may be used to store location information associated with a user received from the client system 530. The advertising pricing module may combine social information, current time, location information, or other suitable information to provide relevant advertisements to the user in the form of notifications.

[0062] Figure 6 An example computer system 600 is shown. In a particular embodiment, one or more computer systems 600 perform one or more steps of one or more methods described or shown herein. In a particular embodiment, one or more computer systems 600 provide functionality described or shown herein. In a particular embodiment, software running on one or more computer systems 600 performs one or more steps of one or more methods described or shown herein, or provides functionality described or shown herein. A particular embodiment includes one or more portions of one or more computer systems 600. Herein, references to computer systems may encompass computing devices, and vice versa, where appropriate. In addition, references to computer systems may encompass one or more computer systems, where appropriate.

[0063] The present disclosure contemplates any suitable number of computer systems 600. The present disclosure contemplates computer systems 600 in any suitable physical form. By way of example and not limitation, computer system 600 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or a system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a network of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these systems. Where appropriate, computer system 600 may include one or more computer systems 600; may be single or distributed, across multiple locations, across multiple machines, across multiple data centers, or located in the cloud (which may include one or more cloud components in one or more networks). Where appropriate, one or more computer systems 600 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. As an example and not limitation, one or more computer systems 600 may perform one or more steps of one or more methods described or illustrated herein in real time or in batch mode. Where appropriate, one or more computer systems 600 may perform one or more steps of one or more methods described or illustrated herein at different times or at different locations.

[0064] In a particular embodiment, computer system 600 includes a processor 602, a memory 604, a storage device 606, an input / output (I / O) interface 608, a communication interface 610, and a bus 612. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0065] In a particular embodiment, the processor 602 includes hardware for executing instructions, such as instructions that constitute a computer program. By way of example and not limitation, to execute instructions, the processor 602 may retrieve (or fetch) instructions from an internal register, an internal cache, memory 604, or storage 606; decode and execute these instructions; and then write one or more results to an internal register, an internal cache, memory 604, or storage 606. In a particular embodiment, the processor 602 may include one or more internal caches for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 602 that includes any suitable number of any suitable internal caches. By way of example and not limitation, the processor 602 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions in the instruction cache may be copies of instructions in the memory 604 or storage 606, and the instruction cache may speed up the retrieval of these instructions by the processor 602. The data in the data cache may be a copy of data in the memory 604 or storage device 606 for execution at the processor 602 to operate thereon; may be the result of a previous instruction executed at the processor 602 for access by a subsequent instruction executed at the processor 602, or for writing to the memory 604 or storage device 606; or may be other suitable data. The data cache may speed up read or write operations performed by the processor 602. The TLB may speed up virtual address translations of the processor 602. In a particular embodiment, the processor 602 may include one or more internal registers for data, instructions, or addresses. Where appropriate, the present disclosure contemplates that the processor 602 includes any suitable number of any suitable internal registers. Where appropriate, the processor 602 may include one or more arithmetic logic units (ALUs); may be a multi-core processor; or may include one or more processors 602. Although the present disclosure describes and illustrates a particular processor, the present disclosure contemplates any suitable processor.

[0066] In certain embodiments, memory 604 includes a main memory for storing instructions for processor 602 to execute or data for processor 602 to operate on. By way of example and not limitation, computer system 600 may load multiple instructions from storage device 606 or another source (e.g., another computer system 600) to memory 604. Processor 602 may then load these instructions from memory 604 to an internal register or internal cache memory. To execute the instructions, processor 602 may retrieve the instructions from the internal register or internal cache memory and decode the instructions. During or after execution of the instructions, processor 602 may write one or more results (which may be intermediate results or final results) to the internal register or internal cache memory. Processor 602 may then write one or more of these results to memory 604. In certain embodiments, processor 602 executes only instructions in one or more internal registers or internal cache memories or in memory 604 (rather than storage device 606 or elsewhere), and operates only on data in one or more internal registers or internal cache memories or in memory 604 (rather than storage device 606 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple the processor 602 to the memory 604. As described below, the bus 612 may include one or more memory buses. In a particular embodiment, one or more memory management units (MMUs) are located between the processor 602 and the memory 604 and facilitate access to the memory 604 requested by the processor 602. In a particular embodiment, the memory 604 includes a random access memory (RAM). Where appropriate, the RAM is a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port RAM or a multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, the memory 604 may include one or more memories 604. Although the present disclosure describes and illustrates specific memories, the present disclosure contemplates any suitable memory.

[0067] In a particular embodiment, the storage device 606 includes a large capacity storage device for data or instructions. As an example and not limitation, the storage device 606 may include a hard disk drive (HDD), a floppy disk drive (floppydisk drive), a flash memory, an optical disk, a magneto-optical disk, a tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these storage devices. Where appropriate, the storage device 606 may include a removable or non-removable (or fixed) medium. Where appropriate, the storage device 606 may be located inside or outside the computer system 600. In a particular embodiment, the storage device 606 is a non-volatile solid-state memory. In a particular embodiment, the storage device 606 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these ROMs. The present disclosure contemplates a large-capacity storage device 606 in any suitable physical form. Where appropriate, the storage device 606 may include one or more storage control units that facilitate communication between the processor 602 and the storage device 606. Where appropriate, the storage device 606 may include one or more storage devices 606. Although the present disclosure describes and illustrates a particular storage device, the present disclosure contemplates any suitable storage device.

[0068] In a particular embodiment, the I / O interface 608 includes the following hardware, software, or both: the hardware, software, or both provide one or more interfaces for communication between the computer system 600 and one or more I / O devices. Where appropriate, the computer system 600 may include one or more of these I / O devices. One or more of these I / O devices may be able to implement communication between a person and the computer system 600. As an example and not limitation, the I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet computer, a touch screen, a trackball, a camera, another suitable I / O device, or a combination of two or more of these I / O devices. The I / O device may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 608 for these I / O devices. Where appropriate, the I / O interface 608 may include one or more device drivers or software drivers that enable the processor 602 to drive one or more of these I / O devices. Where appropriate, I / O interface 608 may include one or more I / O interfaces 608. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0069] In a particular embodiment, the communication interface 610 includes hardware, software, or both that provide one or more interfaces for communication between the computer system 600 and one or more other computer systems 600 or with one or more networks (e.g., packet-based communication). By way of example and not limitation, the communication interface 610 may include a network interface controller (NIC) or a network adapter for communicating with an Ethernet or other wired-based network, or a wireless NIC (WNIC) or a wireless adapter for communicating with a wireless network such as a WI-FI network. The present disclosure contemplates any suitable network and any suitable communication interface 610 for the network. By way of example and not limitation, the computer system 600 may communicate with the following networks: an ad hoc network; a personal area network (PAN); a local area network (LAN); a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet, or a combination of two or more of these networks. One or more portions of one or more of these networks may be wired or wireless. As an example, the computer system 600 may communicate with: a wireless PAN (WPAN) (e.g., a Bluetooth WPAN); a WI-FI network; a WI-MAX network; a cellular telephone network (e.g., a Global System for Mobile Communication (GSM) network); or other suitable wireless networks; or a combination of two or more of these. Where appropriate, the computer system 600 may include any suitable communication interface 610 for any of these networks. Where appropriate, the communication interface 610 may include one or more communication interfaces 610. Although the present disclosure describes and illustrates particular communication interfaces, the present disclosure contemplates any suitable communication interface.

[0070] In particular embodiments, bus 612 includes hardware, software, or both that couples the various components of computer system 600 to one another. By way of example and not limitation, bus 612 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a Hypertransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBAND interconnect, a low-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus, or a combination of two or more of these. Where appropriate, bus 612 may include one or more buses 612. Although the present disclosure describes and illustrates a particular bus, the present disclosure contemplates any suitable bus or interconnect.

[0071] In this document, where appropriate, one or more computer-readable non-transitory storage media may include: one or more semiconductor-based integrated circuits (integrated circuits, ICs) or other ICs (e.g., field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical disk drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital cards (SECUREDIGITAL cards) or secure digital drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these storage media. Where appropriate, computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0072] As used herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or indicated by context. Thus, as used herein, "A or B" means "A, B, or both," unless expressly indicated otherwise or indicated by context. Furthermore, "and" is both joint and several, unless expressly indicated otherwise or indicated by context. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly indicated otherwise or indicated by context.

[0073] The scope of the present disclosure covers all changes, substitutions, variations, modifications and modifications to the example embodiments described or shown herein that will be understood by those skilled in the art. The scope of the present disclosure is not limited to the example embodiments described or shown herein. In addition, although the present disclosure describes and illustrates various embodiments including specific parts, elements, features, functions, operations or steps herein, any embodiment in these embodiments may include any combination or arrangement of any parts, elements, features, functions, operations or steps described or shown anywhere in this article that will be understood by those of ordinary skill in the art. In addition, references to devices or systems or parts of devices or systems that are suitable for, arranged to, capable of, configured to, enabled to, capable of operating as, or operable to perform specific functions in the appended claims include the device, system, component, regardless of whether the device, system, component or the specific function is activated, turned on or unlocked, as long as the device, system or component is so adapted, arranged to, capable of, configured to, enabled to, capable of operating or operable. In addition, although the present disclosure describes or illustrates specific embodiments as providing specific advantages, specific embodiments may not provide these advantages, or may provide some or all of these advantages.

Claims

1. A computing system comprising: Color camera; two monochrome cameras, the two monochrome cameras being associated with the user's two eyes respectively; one or more processors; as well as one or more computer-readable non-transitory storage media coupled to one or more of the one or more processors and comprising instructions operable when executed by one or more of the one or more processors to cause the computing system to: Calculating a point cloud corresponding to the visible environment based on at least two stereo grayscale images respectively acquired by the two monochrome cameras; generating a mesh corresponding to the visible environment based on the calculated point cloud; as well as generating two stereoscopic colorized images to be displayed to two eyes of the user, respectively, wherein each of the two stereoscopic colorized images is generated using: (1) the grid; and (2) brightness information from one of the two stereoscopic grayscale images, the one stereoscopic grayscale image being associated with the eye to which the stereoscopic colorized image is to be displayed; and (3) color information from a color input image captured by the color camera.

2. The computing system of claim 1 further comprising two displays, wherein: The one or more processors, when executing the instructions, are further operable to: Each of the two stereoscopic colorized images is presented to a corresponding one of the two displays.

3. The computing system of claim 1, wherein: Rendering each vertex in the mesh for generating each of the two stereo colorized images comprises: identifying a first corresponding pixel in the corresponding grayscale image by performing a ray casting from the vertex to a corresponding monochrome camera capturing the corresponding grayscale image; identifying a second corresponding pixel in the color input image by performing a ray casting from the vertex to the color camera; converting a first RGB value associated with the second corresponding pixel into second color information in a second color space, wherein a brightness level indicator is separate from one or more chromaticity information indicators in the second color space; replacing the brightness level indicator in the second color information with a value corresponding to a brightness level associated with the first corresponding pixel; converting the second color information into a second RGB value; and The vertex is rendered using the second RGB value.

4. The computing system of claim 3, wherein: The second color space is the CIELAB color space.

5. The computing system of claim 1, wherein: The one or more processors, when executing the instructions, are further operable to: The points in the point cloud are subsampled to reduce the number of points in the point cloud, wherein, in the two stereo grayscale images, when a first region has a higher gradient than a second region, a first sampling rate corresponding to the first region may be higher than a second sampling rate corresponding to the second region.

6. The computing system of claim 1, wherein: The one or more processors, when executing the instructions, are further operable to: Points in the point cloud whose estimated likelihood of being non-noise points is below a threshold likelihood associated with the location of the point are filtered.

7. The computing system of claim 6, wherein: When a first position of the point cloud is within a first threshold distance from the two monochrome cameras and a second position exceeds the first threshold distance from the two monochrome cameras, a first threshold likelihood associated with the first position is lower than a second threshold likelihood associated with the second position.

8. The computing system of claim 6, wherein: The estimated likelihood that the point is a non-noise point is estimated based on the number of times the point is observed consecutively in a predetermined number of previous frames, and wherein the threshold likelihood associated with the location is represented by a minimum number of times the point is observed consecutively in the predetermined number of previous frames.

9. The computing system of claim 1, wherein: Generating the grid includes: Initializing the grid corresponding to the point cloud as a planar grid at a predetermined distance from the two monochrome cameras; For each point in the point cloud: identifying a vertex in the mesh corresponding to the point; warping the identified vertices to the positions of the points in the point cloud; determining that the point in the point cloud is within a second threshold distance; and In response to determining that the point in the point cloud is within the second threshold distance, k vertices closest to the identified vertex are warped to the position of the point, where k is a predetermined positive integer.

10. The computing system of claim 1 , further comprising a structured light projector that projects an infrared (IR) pattern at a configured frequency, wherein: The two monochrome cameras are synchronized with the structured light projector so that the two monochrome cameras capture IR monochrome images, wherein the IR monochrome images include reflections of the IR pattern projected at the configured frequency, wherein each of the two monochrome cameras captures j non-IR monochrome images between two consecutive IR image captures, wherein the j non-IR monochrome images do not include reflections of the projected IR pattern, and wherein j is a preconfigured positive integer.

11. The computing system of claim 10, wherein: Computing the point cloud is also based on reflections of the projected IR pattern in the IR monochrome images acquired by the two monochrome cameras.

12. A method comprising: A computing system comprising a color camera and two monochrome cameras associated with the user's two eyes: Calculating a point cloud corresponding to the visible environment based on at least two stereo grayscale images respectively acquired by the two monochrome cameras; generating a mesh corresponding to the visible environment based on the calculated point cloud; as well as generating two stereoscopic colorized images to be displayed to two eyes of the user, respectively, wherein each of the two stereoscopic colorized images is generated using: (1) the grid; and (2) brightness information from one of the two stereoscopic grayscale images, the one stereoscopic grayscale image being associated with the eye to which the stereoscopic colorized image is to be displayed; and (3) color information from a color input image captured by the color camera.

13. The method according to claim 12, wherein: The computing system also includes two displays, and the method further includes: Each of the two stereoscopic colorized images is presented to a corresponding one of the two displays.

14. The method according to claim 12, wherein: Rendering each vertex in the mesh for generating each of the two stereo colorized images comprises: identifying a first corresponding pixel in the corresponding grayscale image by performing a ray casting from the vertex to a corresponding monochrome camera capturing the corresponding grayscale image; identifying a second corresponding pixel in the color input image by performing a ray casting from the vertex to the color camera; converting a first RGB value associated with the second corresponding pixel into second color information in a second color space, wherein a brightness level indicator is separate from one or more chromaticity information indicators in the second color space; replacing the brightness level indicator in the second color information with a value corresponding to a brightness level associated with the first corresponding pixel; converting the second color information into a second RGB value; and The vertex is rendered using the second RGB value.

15. The method according to claim 14, wherein: The second color space is the CIELAB color space.

16. The method according to claim 12, further comprising: The points in the point cloud are subsampled to reduce the number of points in the point cloud, wherein, in the two stereo grayscale images, when a first region has a higher gradient than a second region, a first sampling rate corresponding to the first region may be higher than a second sampling rate corresponding to the second region.

17. The method according to claim 12, further comprising: Points in the point cloud whose estimated likelihood of being non-noise points is below a threshold likelihood associated with the location of the point are filtered.

18. The method according to claim 17, wherein: When a first position of the point cloud is within a first threshold distance from the two monochrome cameras and a second position exceeds the first threshold distance from the two monochrome cameras, a first threshold likelihood associated with the first position is lower than a second threshold likelihood associated with the second position.

19. The method according to claim 17, wherein: The estimated likelihood that the point is a non-noise point is estimated based on the number of times the point is observed consecutively in a predetermined number of previous frames, and wherein the threshold likelihood associated with the location is represented by a minimum number of times the point is observed consecutively in the predetermined number of previous frames.

20. One or more computer-readable non-transitory storage media containing software that, when executed by a computing system including a color camera and two monochrome cameras associated with one eye of a user, is operable to: Calculating a point cloud corresponding to the visible environment based on at least two stereo grayscale images respectively acquired by the two monochrome cameras; generating a mesh corresponding to the visible environment based on the calculated point cloud; as well as generating two stereoscopic colorized images to be displayed to two eyes of the user, respectively, wherein each of the two stereoscopic colorized images is generated using: (1) the grid; and (2) brightness information from one of the two stereoscopic grayscale images, the one stereoscopic grayscale image being associated with the eye to which the stereoscopic colorized image is to be displayed; and (3) color information from a color input image captured by the color camera.

Images