Computer vision camera for infrared light detection

Abstract

A head-mounted device (HMD) is configured to include at least one computer vision camera that omits an IR filter. Thus, the computer vision sensor is able to detect IR light, including IR lasers, in the environment. The HMD is configured to generate an image of the environment using the computer vision camera. This image is then fed as input into a machine learning (ML) algorithm that identifies IR lasers, which are detected by the sensor and recorded in the image. The HMD then visually displays a notification containing information corresponding to the detected IR laser.

Description

Background Technology

[0001] Mixed reality (MR) systems, including virtual reality (VR) and augmented reality (AR) systems, have garnered significant attention for their ability to create truly unique experiences for their users. For reference, traditional VR systems create fully immersive experiences by restricting the user's field of vision to a virtual environment only. This is typically achieved using a head-mounted display (HMD) that completely blocks any view of the real world. As a result, the user is fully immersed in the virtual environment. In contrast, traditional AR systems create augmented reality experiences by visually presenting virtual objects placed in or interacting with the real world.

[0002] As used herein, VR and AR systems are described and referred to interchangeably. Unless otherwise stated, the description herein also applies to all types of MR systems, which (as detailed above) include AR systems, VR reality systems, and / or any other similar systems capable of displaying virtual content.

[0003] MR devices benefit from sensors that can improve or enhance a user's understanding of the world around them. For example, there are many situations where users want to see things in the dark. First responders, for instance, benefit from using low-light cameras to assist in search and rescue operations. Additionally, fire departments are equipped with AR goggles featuring thermal imaging systems that can sense temperature. Unfortunately, the sensors used for low-light and thermal sensing are both expensive and bulky. Each sensor costs between $1,000 and $3,000, accounting for 40% of the material cost used in an HMD. Each sensor is also approximately 1” x 1” x 1.5” in size and weighs about 30-35 grams. These sensors are also high-power sensors, with each consuming 1-1.5W of power.

[0004] Despite their high cost and size, low-light sensors have limited practicality due to high dark current and readout noise that restrict their ability to see in environments below approximately starlight illumination levels. However, one of the key functions of these sensors is the ability to observe and detect lasers (e.g., infrared (IR) lasers, such as those used for target designation in MR games) through glass. Therefore, there is a need for an improved technique to detect IR lasers at the lowest possible cost, without increasing the system's size, weight, or power.

[0005] The subject matter claimed herein is not limited to embodiments that address any shortcomings or operate only in environments such as those described above. Rather, this background is provided merely to illustrate an exemplary technical field in which some of the embodiments described herein can be practiced. Summary of the Invention

[0006] The embodiments disclosed herein relate to systems, apparatus (e.g., hardware storage devices, wearable devices, etc.) and methods for detecting infrared (IR) lasers emitted in an environment.

[0007] In some embodiments, the head-mounted device (HMD) is configured to include at least one computer vision camera with an IR filter omitted. Therefore, the computer vision sensor is capable of detecting IR light in the environment, including IR lasers. The HMD is configured to generate an image of the environment using the computer vision camera. This image is then fed as input to a machine learning (ML) algorithm that identifies IR lasers detected and recorded in the image by the sensor. The HMD then visually displays a notification containing information corresponding to the detected IR laser.

[0008] In some embodiments, the HMD configured as described above generates a first image of the environment using a computer vision camera. The HMD also uses a thermal imaging camera to generate a second image of the environment. The first image is then fed as input into a machine learning (ML) algorithm configured to identify collimated IR light detected and recorded in the first image by the computer vision camera's sensor. The HMD reprojects the second image to compensate for the parallax between the user's pupil and the thermal imaging camera. Furthermore, the HMD reprojects the identified collimated IR light to compensate for the parallax between the user's pupil and the computer vision camera. The HMD also superimposes the reprojected collimated IR light onto the reprojected second image to generate a superimposed image, which is then visually displayed.

[0009] This summary is provided to present a selection of concepts in a simplified form, which will be further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0010] Additional features and advantages will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the teachings herein. The features and advantages of the invention can be realized and obtained by the means and combinations particularly pointed out in the appended claims. The features of the invention will become clearer from the description which follows and the appended claims, or may be learned by practice of the invention as set forth below. Attached Figure Description

[0011] To describe how the above and other advantages and features can be obtained, the subject matter briefly described above will be described in more detail with reference to specific embodiments shown in the accompanying drawings. It is understood that these drawings depict only typical embodiments and should therefore not be considered as limiting the scope; the embodiments will be described and explained with additional specificity and detail using the drawings, wherein:

[0012] Figure 1 An example scenario involving a game application is shown.

[0013] Figure 2 An example head-mounted device (HMD) is shown.

[0014] Figure 3 An example implementation or configuration of HMD is shown.

[0015] Figure 4 A flowchart of an example method for detecting IR lasers in the environment is shown.

[0016] Figure 5 An example scenario of emitting an IR laser in an environment is shown.

[0017] Figure 6 An example of an IR laser emitter is shown.

[0018] Figure 7 Example computer vision images and thermal images are shown.

[0019] Figure 8 This demonstrates how machine learning (ML) algorithms can analyze computer vision images to resolve or identify IR lasers from other forms of IR light.

[0020] Figure 9 The parallax correction operation that can be performed on an image portion including detected IR laser is shown.

[0021] Figure 10 It demonstrates how parallax correction depends on the depth map.

[0022] Figure 11 This demonstrates how parallax correction can also be performed on thermal images.

[0023] Figure 12 This illustrates how pixels corresponding to the reprojected IR laser can be superimposed onto the reprojected thermal image to form a superimposed image.

[0024] Figure 13 An example through-image is shown, which includes different notifications about the detected IR laser.

[0025] Figure 14 Another flowchart is shown as an example method for detecting collimated IR light.

[0026] Figure 15 An example computer system capable of performing any publicly disclosed operation is shown. Detailed Implementation

[0027] The embodiments disclosed herein relate to systems, apparatus (e.g., hardware storage devices, wearable devices, etc.) and methods for detecting infrared (IR) lasers emitted in an environment.

[0028] In some embodiments, the HMD includes a computer vision camera that omits an IR filter. The HMD uses the computer vision camera to generate an image of the environment. This image is fed as input into an ML algorithm to identify IR lasers. The HMD then visually displays a notification describing the detected IR laser.

[0029] In some embodiments, the HMD generates a first image using a computer vision camera and a second image using a thermal imaging camera. The first image is fed as input into an ML algorithm to identify collimated IR light. The HMD reprojects the second image to compensate for parallax and reprojects the identified collimated IR light to also compensate for parallax. The HMD superimposes the reprojected collimated IR light onto the reprojected second image to generate a superimposed image, which is then visually displayed.

[0030] Examples of technological benefits, improvements, and practical applications

[0031] The following sections outline some example improvements and practical applications provided by the disclosed embodiments. However, it should be understood that these are merely examples and the embodiments are not limited to these improvements.

[0032] The disclosed embodiments provide substantial improvements, benefits, and practical applications to the art. For example, the disclosed embodiments improve how environmental conditions (e.g., the presence of an IR laser) are detected. These embodiments also advantageously provide notification regarding the detection of these environmental conditions. Furthermore, the disclosed embodiments are used to reduce the cost of HMDs, the power consumption of HMDs, and even the weight of HMDs.

[0033] In other words, the embodiments address the problem of detecting IR lasers in the environment. By reusing existing camera systems (e.g., head-tracking HeT cameras, a specific type of computer vision camera used to track HMD motion) to detect IR light, these embodiments avoid the need for adding a low-light camera, which traditionally detects IR light. Furthermore, by avoiding the need for a low-light camera, the embodiments reduce hardware costs, decrease the size and weight of the HMD, and reduce system power consumption (e.g., less power is consumed due to fewer hardware components). Therefore, by performing the disclosed operations, the embodiments significantly improve HMD operation.

[0034] Example scenarios, including game implementation schemes and HMD configurations.

[0035] Figure 1An example gaming environment 100 in which the HMD 105 operates is shown. In this scenario, the HMD 105 is presenting an MR scene to a user, which includes a hologram 110 in the form of a dragon. Furthermore, in this scene, a person wearing the HMD 105 is fighting or battling a dragon. HMDs, including the HMD 105, are typically used in low-light environments. HMDs are also frequently used to emit visible light and / or IR lasers into the environment to aid in target detection. For example, it is possible that in gaming environment 100, the user of the HMD 105 is using an IR laser emitter to aim his / her weapon at the hologram 110. Therefore, it is highly beneficial to provide a system with improved IR laser detection. The disclosed embodiments provide this widely sought-after benefit.

[0036] The HMD 105 can be configured in various different ways, such as Figure 2 and Figure 3 As shown. For example, Figure 1 The HMD 105 can be configured as Figure 2 The HMD 200 can be any type of MR system 200A, including VR system 200B or AR system 200C. It should be noted that while much of this disclosure focuses on the use of an HMD, the embodiments are not limited to the use of an HMD alone. That is, any type of scanning system can be used, even systems completely removed or separated from the HMD. Therefore, the disclosed principles should be broadly interpreted to encompass any type of scanning scenario or device. Some embodiments may even avoid actively using the scanning device itself and can simply use the data generated by the scanning device. For example, some embodiments can be practiced at least partially in a cloud computing environment.

[0037] HMD 200 is shown to include a scanning sensor 205 (i.e., a type of scanning or camera system), and HMD 200 can use the scanning sensor 205 to scan the environment, map the environment, capture environmental data, and / or generate any type of image of the environment (e.g., by generating a 3D representation of the environment or by generating a “pass-through” visualization). The scanning sensor 205 can include any number or type of scanning device without limitation.

[0038] According to the disclosed embodiments, the HMD 200 can be used to generate parallax-corrected passthrough visualizations of the user's environment. In some cases, "passthrough" visualization refers to a visualization reflecting what the user would see without wearing the HMD 200, regardless of whether the HMD 200 is included as part of an AR or VR system. In other cases, passthrough visualization reflects different or novel perspectives. In some situations, passthrough visualization identifies conditions that may be undetectable by the human eye, such as the presence of IR lasers in the environment.

[0039] To generate this pass-through visualization, the HMD 200 can use its scanning sensor 205 to scan its surroundings, map its surroundings, or otherwise record its surroundings, including any objects or IR light in the environment, and then transmit that data to the user for viewing. In many cases, the pass-through data is modified to reflect or correspond to the user's pupil's viewing angle, although the image may also reflect other viewing angles. The viewing angle can be determined using any type of eye-tracking technology or other data.

[0040] To convert raw images into pass-through images, scanning sensor 205 typically relies on its camera (e.g., any type of computer vision camera, such as a head-tracking camera, hand-tracking camera, depth camera, or any other type of camera) to obtain one or more raw images (also called texture images) of the environment. In addition to generating pass-through images, these raw images can be used to determine depth data (e.g., z-axis range or measurements) that details the distance from the sensor to any object captured in the raw images. Once these raw images are obtained, depth maps can be computed based on depth data embedded in or contained in the raw images (e.g., based on pixel disparity), and the depth maps can be used to generate pass-through images (e.g., one for each pupil) for any reprojection. In some cases, depth maps can be evaluated by 3D sensing systems, including time-of-flight, stereo, active stereo, or structured light systems. Furthermore, a visual map of the surrounding environment can be evaluated using head-tracking cameras, which typically have stereo overlap regions to evaluate 3D geometry and generate an environment map.

[0041] As used herein, a "depth map" details the positional relationships and depth of objects relative to their environment. Therefore, the arrangement, positioning, geometry, contours, and depth of objects relative to each other can be determined. From the depth map, a 3D representation of the environment can be generated. As will be described in more detail later, the depth map can be used to perform parallax correction on a pass-through image.

[0042] According to pass-through visualization, users will be able to perceive what is currently in their environment without having to remove or reposition the HMD 200. Furthermore, as will be described in more detail later, the disclosed pass-through visualization will also enhance the user's ability to view objects or situations in their environment (e.g., the presence or absence of an IR laser) (e.g., by displaying other environmental conditions or image data that the human eye may not yet be able to detect).

[0043] It should be noted that while much of this disclosure focuses on generating “one” passthrough (or overlay) image, embodiments can generate separate passthrough images for each of the user’s eyes. That is, two passthrough images are typically generated concurrently with each other. Therefore, although the generation of what appears to be a single passthrough image is frequently mentioned, embodiments are actually capable of generating multiple passthrough images simultaneously.

[0044] In some embodiments, the scanning sensor 205 includes a computer vision camera 210 (with a removable IR filter 210A) and a low-light camera 215 (although not essential, such as...). Figure 2 The scan sensor 205 includes a thermal imaging camera 220 (shown in the dashed box), a possible (but not required, ultraviolet (UV) camera 225), and a possible (but not required) point illuminator (not shown). The ellipsis 230 illustrates how any other type of camera or camera system (e.g., depth camera, time-of-flight camera, virtual camera, depth laser, etc.) can be included in the scan sensor 205.

[0045] As an example, a camera configured to detect IR wavelengths may be included in scanning sensor 205. As another example, any number of virtual cameras reprojected from an actual camera may be included in scanning sensor 205 and used to generate stereo image pairs. In this way, and as will be discussed in more detail later, scanning sensor 205 can be used to generate stereo image pairs. In some cases, stereo image pairs may be obtained or generated as a result of any one or more of the following operations: generating active stereo images via using two cameras and a point illuminator; generating passive stereo images via using two cameras; generating images using structured light via using an actual camera, a virtual camera, and a point illuminator; or generating images using a time-of-flight (TOF) sensor, where a baseline exists between a depth laser and a corresponding camera, and where the field of view (FOV) of the corresponding camera is offset relative to the illumination field of the depth laser.

[0046] Typically, the human eye can perceive light within the so-called "visible spectrum," which includes light (or more precisely, electromagnetic radiation) with wavelengths ranging from about 380 nanometers (nm) to about 740 nm. As used herein, the computer vision camera 210 includes two or more monochromatic cameras configured to capture photons within the visible spectrum. Typically, these monochromatic cameras are complementary metal-oxide-semiconductor (CMOS) type cameras, but other types of cameras (e.g., charge-coupled devices, CCDs) can also be used. These monochromatic cameras can also be extended to the NIR range (up to 1100 nm).

[0047] Monochrome cameras are typically stereo cameras, meaning that the fields of view of two or more monochrome cameras at least partially overlap each other. Using this overlapping region, the image generated by computer vision camera 210 can be used to identify differences between certain pixels, which typically represent objects captured by the two images. Based on these pixel differences, embodiments are able to determine the depth of objects located within the overlapping region (i.e., "stereo depth matching"). Therefore, computer vision camera 210 can be used not only to generate through-view visualizations but also to determine object depth. In some embodiments, computer vision camera 210 can capture both visible and IR light.

[0048] According to the disclosed embodiments, the computer vision camera 210 (also referred to in some cases as a head-tracking camera) is configured to omit an IR filter. In some cases, the IR filter is removable (e.g., as shown in the removable IR filter 210A) so that it can be attached to or detached from the HMD, or more precisely, it is detached from the computer vision camera. By removing or detaching the IR filter from the computer vision camera 210, the computer vision camera 210 will be able to detect at least some IR light. For example, IR lasers typically emit IR laser light with wavelengths between approximately 850 nanometers and 1064 nanometers. By removing the IR filter from the computer vision camera 210, these cameras are able to detect at least certain wavelengths of IR light, including the wavelength of IR laser light.

[0049] Optionally, the HMD 200 may include a low-light camera 215. In some cases, the HMD 200 may not include a low-light camera 215. When the HMD 200 does include a low-light camera 215, these cameras can be selectively operated so that they can have a default off state.

[0050] If the HMD 200 does indeed include low-light cameras 215, these cameras are configured to capture both visible and IR light. IR light is generally categorized into three distinct classes: near-IR, mid-IR, and far-IR (e.g., thermal IR). This classification is determined based on the energy of the IR light. For example, near-IR has relatively high energy due to its relatively short wavelength (e.g., between approximately 750 nm and approximately 1,100 nm). In contrast, far-IR has relatively low energy due to its relatively long wavelength (e.g., up to approximately 30,000 nm). Mid-IR energy values ​​lie between or in the middle of the near-IR and far-IR ranges. The low-light camera 215 is configured to detect or be sensitive to IR light, at least within the near-IR range.

[0051] In some embodiments, the computer vision camera 210 and the low-light camera 215 (also referred to as a low-light night vision camera) operate within substantially the same overlapping wavelength range. In some cases, this overlapping wavelength range is between about 400 nanometers and about 1,100 nanometers. Furthermore, in some embodiments, both types of cameras are silicon detectors. By removing the IR filter from the computer vision camera 210, the HMD 200 can avoid relying on the low-light camera 215 to detect at least IR “laser” light. Therefore, in a preferred embodiment, the HMD 200 omits or does not include the low-light camera 215.

[0052] Thermal imaging camera 220 is configured to detect electromagnetic radiation or IR light in the far IR (i.e., thermal IR) range, although some embodiments also enable thermal imaging camera 220 to detect radiation in the mid IR range. For clarity, thermal imaging camera 220 can be a long-wave infrared imaging camera, configured to detect electromagnetic radiation by measuring long-wave infrared wavelengths. Typically, thermal imaging camera 220 detects IR radiation with wavelengths between approximately 8 micrometers and 14 micrometers to detect blackbody radiation from the environment and people in the camera's field of view. Because thermal imaging camera 220 detects far IR radiation, it can operate unrestricted under any illumination conditions.

[0053] The UV camera 225 is configured to capture light in the UV range. The UV range includes electromagnetic radiation with wavelengths between about 150 nm and about 400 nm. The disclosed UV camera 225 should be interpreted broadly and can operate in a manner that includes reflective UV photography and UV-induced fluorescence photography.

[0054] Therefore, as used herein, references to "visible light camera" or "computer vision camera" (including "head-tracking camera") primarily refer to cameras used in computer vision to perform head tracking. These cameras can detect visible light, and even combinations of visible and IR light (e.g., a range of IR light, including IR light with wavelengths between approximately 850 nm and 1064 nm due to the removal of IR filters). In some cases, these cameras are global shutter devices with a pixel size of approximately 3 μm. Thermal / long-wavelength IR devices (i.e., thermal imaging cameras) have a pixel size of approximately 10 μm or larger and detect heat from ambient radiation. These cameras are sensitive to wavelengths in the range of 8 μm to 14 μm.

[0055] Therefore, the disclosed embodiments can be configured to utilize many different camera types. These different camera types include, but are not limited to, visible light cameras, low-light cameras, thermal imaging cameras, and UV cameras. Stereo depth matching can be performed using images generated from any of the camera types or combinations thereof listed above.

[0056] It should be noted that any number of cameras can be provided on the HMD 200 for each of the different camera types. That is, the computer vision camera 210 may include 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or more cameras. However, the number of cameras is typically at least 2, so that the HMD 200 can perform stereo depth matching, as previously described. Similarly, the low-light camera 215 (if present), the thermal imaging camera 220, and the UV camera 225 may each include 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or more corresponding cameras.

[0057] Figure 3 An example HMD 300 is shown, which represents [the HMD 300]. Figure 2 The HMD 200. The HMD 300 is shown as including several different cameras, including cameras 305, 310, 315, 320, and 325. Cameras 305-325 represent those from... Figure 2 Any number or combination of cameras, including computer vision camera 210, low-light camera 215 (if present), thermal imaging camera 220, and UV camera 225. Although Figure 3 The image only shows 5 cameras, but the HMD 300 may include more or fewer than 5 cameras.

[0058] In some cases, the camera can be positioned at a specific location on the HMD 300. For example, in some cases, the first camera (e.g., camera 320) is positioned on the HMD 300 at a location above the designated left eye of any user wearing the HMD 300 in the height direction relative to the HMD. For example, camera 320 is positioned above the pupil 330. As another example, the first camera (e.g., camera 320) is additionally positioned above the designated left eye location in the width direction relative to the HMD. That is, camera 320 is not only positioned above the pupil 330, but also in a straight line with the pupil 330. When using a VR system, the camera can be placed directly in front of the designated left eye location. For example, see reference... Figure 3 The camera can be physically positioned on the HMD 300 in the z-axis direction in front of the pupil 330.

[0059] When a second camera (e.g., camera 310) is provided, the second camera can be positioned on the HMD at a location above the designated right eye position of any user wearing the HMD in the height direction relative to the HMD. For example, camera 310 is above the pupil 335. In some cases, the second camera is additionally positioned above the designated right eye position in the width direction relative to the HMD. When using a VR system, the camera can be placed directly in front of the designated right eye position. For example, see reference... Figure 3 The camera can be physically positioned on the HMD 300 in the z-axis direction in front of the pupil 335.

[0060] When a user wears the HMD 300, the HMD 300 is placed on the user's head, and the HMD 300's display is positioned in front of the user's pupils (e.g., pupils 330 and 335). Typically, cameras 305-325 are physically offset from the user's pupils 330 and 335 by a certain distance. For example, there may be a vertical offset in the HMD's height direction (i.e., the "Y" axis), as shown by offset 340. Similarly, there may be a horizontal offset in the HMD's width direction (i.e., the "X" axis), as shown by offset 345.

[0061] As previously stated, the HMD 300 is configured to provide pass-through images for the user to view. By doing so, the HMD 300 is able to provide a real-world visualization without requiring the user to remove or reposition the HMD 300. These pass-through images effectively represent the same view the user would see without wearing the HMD 300. In some cases, the pass-through images provide enhanced images that may be imperceptible to the human eye (e.g., a representation of IR lasers). Cameras 305-325 are used to provide these pass-through images.

[0062] However, none of the cameras 305-325 are telecentrically aligned with the pupils 330 and 335. Offsets 340 and 345 effectively introduce a difference in viewing angle between the cameras 305-325 and the pupils 330 and 335. These differences in viewing angle are referred to as "parallax".

[0063] Due to the parallax created by offsets 340 and 345, the raw images (also known as texture images) generated by cameras 305-325 may not be immediately usable as pass-through images. Instead, it is beneficial to perform parallax correction (also known as image synthesis) on the raw images to transform the viewpoints embodied in these raw images to correspond to the viewpoints of the user's pupils 330 and 335. Parallax correction includes any number of corrections, which will be discussed in more detail later.

[0064] Example Method

[0065] The following discussion now involves many methods and method actions that can be performed. Although method actions can be discussed in a specific order or illustrated in a flowchart as occurring in a specific order, a specific order is not required unless specifically stated otherwise, or because performing an action depends on another action that was completed before that action.

[0066] Now let's turn our attention to... Figure 4 The illustration shows a flowchart of an example method 400 for detecting IR laser light emitted in the environment. Method 400 can be performed by an HMD discussed so far. For example, an HMD may include at least one computer vision camera that omits an IR filter, such that the sensor of at least one computer vision camera is operable to detect IR light in the environment, including IR “laser” light.

[0067] Typically, method 400 is executed in low-light environments. For example, method 400 may be triggered when ambient light conditions are at or below approximately 5 lux. For reference, twilight illumination corresponds to approximately 10 lux, while midday bright sunlight corresponds to approximately 100,000 lux.

[0068] Initially, method 400 includes the action of generating an environmental image using at least one computer vision camera (action 405). Figures 5 to 7 It is illustrative.

[0069] Figure 5 An MR environment 500, including real-world objects and a hologram 505, is shown. Here, the ambient light level 510 is at or below 5 lux. Therefore, it is possible to trigger an effect from... Figure 4 Method 400. Different sensors can be used to detect the illuminance of the MR environment 500 in order to trigger method 400.

[0070] Note that within MR environment 500, an HMD 515 is present, configured as described in the previous diagram. Additionally, MR environment 500 is shown to include an IR laser 520. The IR laser 520 is indicated by dashed lines to symbolize how it is not detectable to the naked eye. In this MR environment 500, multiple people are playing a game and fighting a dragon; this is hologram 505. Of course, other types of MR environments can be used, not just game scenes. For example, any type of training environment (e.g., for first responders) or any other type of environment can be used.

[0071] The disclosed embodiments enable the use of a multi-purpose camera that omits the IR filter or a computer vision camera to detect the presence of IR laser. Figure 6 An example laser emitter 600 is shown, which can be configured to emit a visible laser 605 (e.g., possibly a red or green laser) and / or an IR laser 610. In some implementations, the laser emitter 600 is configured to emit only the IR laser 610. In any case, the laser emitter 600 can be used in... Figure 5 The MR environment 500 helps facilitate target detection, especially in low-light gaming or other training scenarios. The IR laser 610 has the wavelengths discussed earlier (e.g., at least between approximately 850 nm and 1064 nm), enabling the HMD's computer vision camera to detect the IR laser 610.

[0072] Figure 7 The image shown is a computer vision image 700 generated by the computer vision camera of the HMD. Note that the computer vision image 700 includes pixels representing the IR laser 705, where the IR laser 705 corresponds to... Figure 5 The IR laser 520 is invisible to the naked eye, but the IR laser 705 is identifiable in the computer vision image 700. In some cases, the computer vision image 700 may capture additional IR light, such as ambient IR light 710. This ambient IR light 710 may be emitted from other sources in the environment, while the IR laser 705 is emitted from... Figure 6 The laser emitter 600 emits light. Note the stark contrast in intensity between the IR light forming the IR laser 705 and the IR light forming the ambient IR light 710. Furthermore, the computer vision image 700 appears dark because it was captured in a low-light environment.

[0073] Typically, the intensity of the IR light forming the IR laser 705 will be significantly higher than the intensity of the IR light forming the ambient IR light 710. In other words, the intensity of the IR light forming the IR laser 705 is relatively higher than the intensity of the IR light forming the ambient IR light 710. Figure 7It also shows how one of HMD's thermal imaging cameras can generate thermal images 720.

[0074] In some cases, the process of generating an environmental image (e.g., one of computer vision image 700 or thermal image 720) is triggered based on the detection of specific environmental conditions. For example, image generation may be in response to the ambient light level of the environment (e.g., Figure 5 The system is triggered when the ambient light level (510) is at or below a threshold lux value. In some cases, the threshold lux value is approximately 5 lux. That is, the embodiment can be triggered to generate the above image when the ambient light level is at or below approximately 5 lux.

[0075] like Figure 7 As shown, computer vision image 700 is shown as having data from... Figure 5 The environment 500 has a specific viewpoint 715. This viewpoint 715 corresponds to the optical axis of the computer vision camera used to generate the computer vision image 700.

[0076] Some embodiments are also configured to use a thermal imaging camera (e.g., possibly a thermal imaging camera). Figure 2 One or more thermal imaging cameras 220 generate thermal image 720. The thermal imaging camera may generate thermal image 720 before, after, or simultaneously with the computer vision camera generating computer vision image 700. Typically, thermal image 720 and computer vision image 700 are generated simultaneously or at least within overlapping time periods. Figure 7 It also shows how the thermal image 720 has a viewing angle 725, which corresponds to the optical axis of the thermal imaging camera used to generate the thermal image 720. Because the thermal imaging camera is located on the HMD at a different position than the computer vision camera, the viewing angle 725 will be different from the viewing angle 715.

[0077] Return to Figure 4 Method 400 also includes an action (action 410) of feeding an image (e.g., a computer vision image) as input to a machine learning (ML) algorithm. The ML algorithm is configured to identify IR lasers detected and recorded in the image by the sensor of the computer vision camera.

[0078] Any type of ML algorithm, model, or machine learning can be used for method action 410. In fact, as used herein, references to “machine learning” or ML models can include any type of machine learning algorithm or device, neural network (e.g., convolutional neural network, multilayer neural network, recurrent neural network, deep neural network, dynamic neural network, etc.), decision tree model (e.g., decision tree, random forest, and gradient boosting tree), linear regression model or logistic regression model, support vector machine (“SVM”), artificial intelligence device, or any other type of intelligent computing system. Any amount of training data (and potentially improved later) can be used to train the machine learning algorithm to dynamically perform the disclosed operations.

[0079] Figure 8 This is a description of method action 410. Specifically, Figure 8 Image 800 is displayed, which represents the image from action 405 and represents Figure 7 The computer vision image 700 is fed as input to the ML algorithm 805. The ML algorithm 805 is configured to identify pixels corresponding to any type of IR light that may be present in the scene, as captured by image 800. Once pixels corresponding to IR light are identified, the ML algorithm 805 further classifies or determines whether each pixel further corresponds to a specific phenomenon or object in the scene. For example, the ML algorithm 805 is able to identify whether an IR light pixel corresponds to ambient IR light and further is able to identify whether an IR light pixel corresponds to IR “laser” light. As an example, Figure 8 This illustrates how the ML algorithm 805 analyzes the IR light pixels included in image 800 and specifically identifies which pixels correspond to the IR laser. Pixels unrelated to the IR laser can be filtered out from the resulting image, leaving only the pixels corresponding to the IR laser, as shown in detected IR laser 810.

[0080] Note that the detected IR laser 810 comprises a row of pixels representing the laser discussed earlier. In some cases, embodiments (e.g., ML algorithms for identifying IR lasers in an image) track the IR lasers detected in the image to identify the travel path 815 of the IR laser through the environment. In this case, the travel path 815 originates from a laser held by a person in the environment and extends outward in the direction toward the dragon hologram. Of course, embodiments are capable of tracking any travel path followed by the IR laser.

[0081] In some cases, embodiments identify IR “laser” light (compared to other forms of IR light) by comparing the intensity 820 of the IR light detected in the image. Typically, the intensity of the IR laser will be relatively (or even substantially) higher than the intensity of the ambient IR light. By initially identifying pixels corresponding to all or most of the IR light in the environment, embodiments can then compare and contrast the intensity of the detected IR light to determine whether it matches or corresponds to an intensity distribution of a known IR “laser” light distribution.

[0082] In some cases, embodiments identify IR lasers based on intelligent recognition of the lines or beams that form IR light, distinguishing them from other forms of IR light that may not be compressed or form beams. The ML algorithm 805 is capable of analyzing various pixels and identifying when IR light pixels form beams or lines. Based on this, the ML algorithm 805 can intelligently determine which pixels are likely to constitute an IR laser beam.

[0083] After the ML algorithm 805 detects an IR laser in the scene, the embodiment then performs several operations to display information corresponding to the detected IR laser. For example, returning to Figure 4 Method 400 includes an action (action 415) in which the HMD visually displays a notification including information corresponding to the detected IR laser.

[0084] To provide this notification, it is often beneficial to generate an overlay image in which the detected IR laser, or at least a notification describing the detected IR laser, is superimposed on another image to form a through or overlay image. Figure 9-13 It is illustrative.

[0085] As a preliminary consideration and as previously stated, the computer vision camera has a different viewing angle than the user's pupil. To provide the user with an accurate depiction of the location of the detected IR laser, the embodiment is capable of performing a parallax correction 90° operation, such as... Figure 9 As shown.

[0086] To perform parallax correction 900, the example first generates a depth map of the environment. Briefly turn to Figure 10 This figure illustrates how an embodiment generates or accesses depth map 1000.

[0087] In some cases, a depth map 1000 can be calculated using a rangefinder 1005. In other cases, a depth map 1000 can be calculated by performing stereo depth matching 1010. The ellipsis 1015 illustrates how other techniques can be used to calculate the depth map 1000, and is not limited to these. Figure 10 The two technologies shown.

[0088] In some implementations, depth map 1000 can be a full depth map 1000A, where each pixel in the depth map is assigned a corresponding depth value. In some implementations, depth map 1000 can be a single-pixel depth map. In some implementations, depth map 1000 can be a planar depth map 1000B, where each pixel in the depth map is assigned the same depth value. In any case, Figure 10 The depth map 1000 represents one or more depths of objects located in the environment. The embodiment can use... Figure 10 Execution of depth map 1000 Figure 9 The parallax correction is 900.

[0089] Figure 9 This details how parallax correction 900 is performed to translate or transform the viewing angle of the detected IR laser to match another viewing angle (e.g., possibly from...). Figure 7 The thermal image 720 (viewpoint, or possibly the user's pupil viewpoint) is aligned, matched, or overlapped. Recall that some embodiments filter the computer vision image to retain only the pixels corresponding to the detected IR laser. Parallax correction 900 can be performed on these remaining pixels.

[0090] By performing or executing the alignment described above, the embodiment can then selectively superimpose a portion (or all) of the detected IR laser onto another image (e.g., possibly from...). Figure 7 On the thermal image 720, accurate alignment between the content of the IR laser and other images is ensured. As will be described in more detail later, some embodiments reproject the IR laser to align with the user's pupil perspective and also reproject the thermal image to align with the user's pupil perspective. Once these two reprojections are performed, a superimposed image is generated by overlaying the reprojected IR laser pixels onto the reprojected thermal image. On the other hand, some embodiments reproject the infrared laser pixels to align with the thermal image perspective and then generate a superimposed image. After generating the superimposed image, it is reprojected to align with the user's pupil perspective. Therefore, various different reprojections can be performed to align the perspective of the superimposed image with the user's pupil perspective.

[0091] Figure 9 Image 905 is shown, and its representation includes Figure 8The image shown is of the detected IR laser 810. Image 905 includes 2D keypoints 910 and corresponding 3D points 915. After determining intrinsic camera parameters 920 (e.g., camera focal length, principal point, and lens distortion) and extrinsic camera parameters 925 (e.g., camera position and orientation), the embodiment can perform a reprojection 930 operation on image 905 to reproject the viewpoint 935 represented by image 905 to a new viewpoint 940. In some cases, the new viewpoint 940 is the viewpoint of the user's pupil, while in other cases, the new viewpoint 940 may be the viewpoint of a different image (e.g., perhaps thermal image 720).

[0092] As a result of performing the reprojection 930 operation, a reprojection image 945 is generated, which includes 2D keypoints 950 corresponding to 2D keypoints 910. In effect, the reprojection 930 operation produces a synthetic camera with new extrinsic camera parameters 955 to give the illusion that the reprojection image 945 was captured by the synthetic camera from a new perspective 940. In this respect, the reprojection image 905 (which may include only pixels corresponding to the detected IR laser, as other pixels are filtered out) compensates for the distance between the computer vision camera and the user's pupil (or possibly a thermal imaging camera), and also compensates for pose or viewpoint differences between the camera and the user's pupil (or possibly a thermal imaging camera).

[0093] Therefore, in some cases, embodiments reproject the detected IR laser to transform those pixels such that the viewing angle of those pixels is aligned with the viewing angle of the user's pupil. In some cases, embodiments reproject the detected IR laser to transform those pixels such that the viewing angle of these pixels is aligned with, for example... Figure 7 The viewpoints of the two images in the thermal image 720 are aligned. In the latter case, once the two viewpoints are aligned, the detected infrared laser can then be directly superimposed onto the thermal image 720. Once this superposition is performed, another parallax correction operation can be performed to align the viewpoint of the newly generated superimposed image with the viewpoint of the user's pupil.

[0094] In the former case, the pixel corresponding to the detected IR laser is reprojected to initially align with the user's pupil. Furthermore, Figure 7 The thermal image 720 is also reprojected to align its viewing angle with the user's pupil. Now that both images are aligned with the user's pupil, the pixels of the detected infrared laser can be directly superimposed on the thermal image to form a superimposed image. Therefore, the embodiment can employ a variety of techniques to generate superimposed images.

[0095] In summary, some embodiments first reproject the pixels of the detected IR light to match the viewing angle of the thermal image, then these embodiments generate a superimposed image, and then these embodiments reproject the superimposed image to match its viewing angle with the user's pupil. Alternatively, some embodiments reproject the pixels of the detected IR light to match the viewing angle of the user's pupil, while these embodiments reproject the thermal image to align its viewing angle with the user's pupil, and then (once the two images are aligned with the user's pupil) these embodiments superimpose the detected IR laser onto the thermal image to generate a superimposed image.

[0096] Figure 11 It shows something similar to Figure 9 The parallax correction 900 is followed by another parallax correction 1100 operation, but this parallax correction 1100 is performed on the thermal image 1105. In short, the thermal image 1105 represents data from... Figure 7 The thermal image 720 is subjected to a reprojection operation 1110 to generate a reprojected image 1115, wherein the viewpoint 1120 of the thermal image 1105 is modified to a new viewpoint 1125 corresponding to the user's pupil or possibly some other novel viewpoint.

[0097] Therefore, some embodiments enable the HMD to perform parallax correction to compensate for the positional offset between the pupil of the user's eye and any of the cameras mentioned herein (e.g., computer vision cameras and / or thermal imaging cameras). In some cases, parallax correction is performed using a full depth map to perform full reprojection. In other cases, parallax correction is performed using a planar depth map to perform planar reprojection.

[0098] Figure 12 A parallax-corrected thermal image 1200 and a parallax-corrected IR laser image 1205 are shown. In this example scenario, the embodiment superimposes at least a portion of the parallax-corrected IR laser image 1205 onto the parallax-corrected thermal image 1200 to generate a superimposed image 1210. This example scenario corresponds to an implementation in which the thermal image and the detected IR laser are reprojected to match the viewing angle of the user's pupil. Of course, other techniques can also be used, first reprojecting the detected IR laser onto the thermal image (such that the detected IR laser initially matches the viewing angle of the thermal image), and then reprojecting the resulting superimposed image to match the viewing angle of the user's pupil. In these reprojection operations, the embodiment relies on... Figure 10 The depth map discussed in the text.

[0099] Figure 12 It also shows how additional indicators or notifications can be provided on the overlay image 1210, such as Figure 4The method action 415 is discussed. That is, in some cases, embodiments may provide notification of the original source 1215 of the IR laser (e.g., the laser emitter is being held by a person). In some cases, embodiments may provide notification of the path 1220 of the IR laser. In some cases, the IR laser may be emphasized in the overlay image 1210, for example by using different shadows, colors, highlights, formats (e.g., dashed lines), etc.

[0100] Some embodiments provide a notification in the form of a direction indicator to indicate where the IR laser source may be located, such as Figure 13 As shown. Specifically, Figure 13 The through image 1300 is shown, which represents Figure 12 The overlay image 1210. The through image 1300 visually illustrates the IR laser 1305. However, in this example scenario, the source of the IR laser 1305 is not included in the through image 1300. The embodiment is able to provide notification in the form of a direction indicator 1310 to indicate where or in what direction the source of the IR laser 1305 may be located.

[0101] Therefore, it can be achieved in various different ways. Figure 4 The "notification" is described in method action 415. One way to implement the notification involves displaying the tracking path of the IR laser. Another way to implement the notification involves identifying the original source of the IR laser. Yet another way to implement the notification involves providing a direction indicator indicating the direction from which the IR laser originates. As described throughout this disclosure, the notification can be overlaid on an image generated by a thermal imaging camera, such as... Figure 12 As shown, the overlay image 1210 is at least partially formed from a thermal image. Since the overlay image 1210 can be considered a pass-through image, it is appropriate to say that the notification is overlaid on the parallax-corrected video pass-through image.

[0102] Additional methods

[0103] Now pay attention Figure 14 The diagram illustrates a flowchart of an example method 1400 for detecting collimated IR light (e.g., laser light) emitted in the environment. Method 1400 can be implemented by any HMD discussed so far. For example, an HMD includes at least one computer vision camera that omits an IR filter, making the computer vision camera's sensor operable to detect IR light in the environment, including collimated IR light. The HMD also includes a thermal imaging camera.

[0104] Initially, method 1400 includes the action (action 1405) of generating a first image of the environment using a computer vision camera. Figure 7The computer vision image 700 represents the image discussed in action 1405. Before, during, or after action 1405, method 1400 includes an action (action 1410) to generate a second image of the environment using a thermal imaging camera. Thermal image 720 represents the thermal image in action 1410.

[0105] In action 1415, the first image is fed as input to a machine learning (ML) algorithm, for example... Figure 8 The ML algorithm 805. This ML algorithm is configured to identify collimated IR light detected by the sensor of a computer vision camera and recorded in the first image. For example, in Figure 8 In this example, the ML algorithm 805 detects IR lasers, as shown in the detected IR laser 810. In some cases, the ML algorithm detects the IR lines or beams that form the collimated IR light / IR laser. In other cases, the ML algorithm distinguishes IR lasers or collimated IR light from other IR lights by detecting the intensity difference between the intensity of the IR laser / collimated IR light and the intensity of other IR lights.

[0106] Action 1420 involves reprojecting the identified collimated IR light to compensate for the parallax between the user's pupil (i.e., the user wearing the HMD) and the computer vision camera. Similarly, action 1425, which can be performed before, during, or after action 1420, involves reprojecting a second image to compensate for the parallax between the user's pupil and the thermal imaging camera. In some implementations, the process of reprojecting the identified collimated IR light is performed by reprojecting only the identified collimated IR light included in the first image and avoiding the reprojection of other content included in the first image. For example, pixel content that does not correspond to the collimated IR light can be filtered out from the first image, leaving only the collimated IR light pixels in the first image. These remaining pixels (corresponding to the collimated IR light) can then be reprojected in the manner described above.

[0107] Because the content from two different images is now aligned as a result of parallax correction, the embodiment can then directly superimpose the collimated IR light pixels onto the thermal image. In this respect, action 1430 involves superimposing the reprojected collimated IR light onto the reprojected second image to generate a superimposed image. Figure 12 The overlay image 1210 illustrates this operation. Finally, method 1400 includes the action of visually displaying the overlay image (action 1435).

[0108] In some embodiments, the HMD implementing the disclosed method is configured to omit all low-light cameras. In some embodiments, the HMD may include low-light cameras, but the HMD may power off those low-light cameras to conserve battery power. Configuring the HMD to include at least one thermal imaging camera is generally advantageous, so that detected IR light can be superimposed on the image generated by that thermal imaging camera, especially when the HMD is used in low-light environments (e.g., below about 5 lux).

[0109] Example computer / computer system

[0110] Now let's turn our attention to... Figure 15 , Figure 15 An example computer system 1500 is illustrated, which may include and / or be used to perform any of the operations described herein. The computer system 1500 may take many different forms. For example, the computer system 1500 may be embodied as a tablet computer 1500A, a desktop or laptop computer 1500B, a wearable device 1500C (e.g., any publicly disclosed HMD), a mobile device, a standalone device, or any other embodiment indicated by ellipsis 1500D. The computer system 1500 may also be a distributed system comprising one or more connected computing components / devices communicating with the computer system 1500.

[0111] In its most basic configuration, the computer system 1500 includes a variety of different components. Figure 15 The display computer system 1500 includes one or more processors 1505 (also referred to as "hardware processing units"), scanning sensors 1510 (e.g., ... Figure 2 The scanning sensor 205), image processing engine 1515, and storage device 1520.

[0112] Regarding processor 1505, it should be understood that the functions described herein can be performed, at least in part, by one or more hardware logic components (e.g., processor 1505). Examples, but not limited to, illustrative types of hardware logic components / processors that can be used include field-programmable gate arrays (“FPGAs”), program-specific or application-specific integrated circuits (“ASICs”), program-specific standard products (“ASSPs”), systems-on-a-chip (“SOCs”), complex programmable logic devices (“CPLDs”), central processing units (“CPUs”), graphics processing units (“GPUs”), or any other type of programmable hardware.

[0113] The computer system 1500 and the scanning sensor 1510 can use any type of depth detection. Examples include, but are not limited to, stereo depth detection (active illumination (e.g., using a point illuminator), structured light illumination (e.g., one physical camera, one virtual camera, and one point illuminator), and passive (i.e., no illumination), time-of-flight depth detection (with a baseline between the laser and the camera, where the camera's field of view does not completely overlap with the laser's illuminated area), rangefinder depth detection, or any other type of distance or depth detection.

[0114] Image processing engine 1515 can be configured to execute the combination graph method 400 and Figure 14 Method 1400 discusses any method actions. In some cases, image processing engine 1515 includes ML algorithms. That is, ML can also be utilized by the disclosed embodiments, as previously described. ML can be implemented as a specific processing unit (e.g., a dedicated processing unit as previously described) configured to perform one or more specialized operations of computer system 1500. As used herein, the terms “executable module,” “executable component,” “component,” “module,” “model,” or “engine” can refer to a hardware processing unit or a software object, routine, or method that can be executed on computer system 1500. The various components, modules, engines, models, and services described herein can be implemented as objects or processors (e.g., as separate threads) that execute on computer system 1500. ML model and / or processor 1505 can be configured to perform one or more of the disclosed method actions or other functions.

[0115] Storage device 1520 may be physical system memory, which may be volatile, non-volatile, or some combination of both. The term "memory" may also be used herein to refer to non-volatile mass storage such as physical storage media. If the computer system 1500 is distributed, then processing, memory, and / or storage capacity may also be distributed.

[0116] Storage device 1520 is shown to include executable instructions (i.e., code 1525). Executable instructions refer to instructions that can be executed by processor 1505 of computer system 1500 (or even possibly graphics processing engine 1515) to perform the disclosed operations, such as those described in the various methods.

[0117] The disclosed embodiments may include or utilize a special-purpose or general-purpose computer, including computer hardware such as one or more processors (e.g., processor 1505) and system memory (e.g., storage device 1520), as discussed in more detail below. Embodiments also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible to a general-purpose or special-purpose computer system. A computer-readable medium that “stores” computer-executable instructions in data form is a “physical computer storage medium” or “hardware storage device.” A computer-readable medium carrying computer-executable instructions is a “transmission medium.” Therefore, by way of example and not limitation, the current embodiments may include at least two distinct types of computer-readable media: computer storage media and transmission media.

[0118] Computer storage media (also known as “hardware storage devices”) are computer-readable hardware storage devices, such as RAM, ROM, EEPROM, CD-ROM, RAM-based solid-state drives (“SSDs”), flash memory, phase-change memory (“PCM”), or other types of memory, or other optical disc storage, disk storage, or other magnetic storage devices, or any other medium that can be used to store computer-executable instructions, data, or data structures in the form of required program code modules that can be accessed by a general-purpose or special-purpose computer.

[0119] Computer system 1500 can also be connected (via wired or wireless connection) to external sensors (e.g., one or more remote cameras) or devices via network 1530. For example, computer system 1500 can communicate with any number of devices or cloud services to acquire or process data. In some cases, network 1530 itself can be a cloud network. Furthermore, computer system 1500 can also be connected via one or more wired or wireless networks 1530 to a remote / standalone computer system configured to perform any of the processes described regarding computer system 1500.

[0120] A “network,” such as network 1530, is defined as one or more data links and / or data switches that enable the transmission of electronic data between computer systems, modules, and / or other electronic devices. When information is transmitted or provided to a computer via a network (wired, wireless, or a combination of wired and wireless), the computer correctly treats the connection as a transmission medium. Computer system 1500 will include one or more communication channels for communicating with network 1530. The transmission medium includes a network that can be used to carry data or required program code modules in the form of computer-executable instructions or data structures. Furthermore, these computer-executable instructions can be accessed by general-purpose or special-purpose computers. Combinations of the above should also be included within the scope of computer-readable media.

[0121] Upon arrival at various computer system components, modules of program code in the form of computer-executable instructions or data structures can be automatically transferred from the transmission medium to the computer storage medium (or vice versa). For example, computer-executable instructions or data structures received via a network or data link can be cached in the RAM within a network interface module (e.g., a network interface card or "NIC") and then ultimately transferred to the computer system RAM and / or less volatile computer storage media within the computer system. Therefore, it should be understood that computer storage media can be included in computer system components that also (or even primarily) use the transmission medium.

[0122] Computer-executable (or computer-interpretable) instructions include, for example, instructions that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a particular function or group of functions. Computer-executable instructions can be, for example, binary files, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. Rather, the described features and actions are disclosed as examples of implementing the claims.

[0123] Those skilled in the art will recognize that embodiments can be implemented in network computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframes, mobile phones, PDAs, pagers, routers, switches, etc. These embodiments can also be practiced in distributed system environments, where local and remote computer systems, linked by a network (via hardwired data links, wireless data links, or a combination of hardwired and wireless data links), each perform tasks (e.g., cloud computing, cloud services, etc.). In a distributed system environment, program modules may reside in local and remote storage devices.

[0124] This invention may be embodied in other specific forms without departing from its spirit or characteristics. The described embodiments should be considered illustrative rather than restrictive in all respects. Therefore, the scope of the invention is indicated by the appended claims rather than the foregoing description. All variations falling within the meaning and scope of the equivalents of the claims should be included within their scope.

Claims

1. A head-mounted device (HMD) configured to detect infrared (IR) laser light emitted from an environment, the HMD comprising: At least one computer vision camera, which omits an IR filter, such that the sensor of the at least one computer vision camera is operable to detect IR light, including IR laser, in the environment; One or more processors; as well as One or more computer-readable hardware storage devices storing instructions executable by the one or more processors to cause the HMD to perform at least the following operations: The environment is generated using the at least one computer vision camera; The image is fed as input to a machine learning (ML) algorithm configured to identify IR lasers detected and recorded in the image by the sensor of the at least one computer vision camera, wherein the ML algorithm is configured to distinguish the IR laser from other IR lights by detecting lines of IR light forming the IR laser, the IR laser being represented by a first set of pixels in the image, and the other IR lights being represented by a second set of pixels in the image; and The system visually displays a notification that includes information corresponding to the detected IR laser.

2. The HMD according to claim 1, wherein, The generation of the image of the environment is triggered in response to determining that the ambient light level of the environment is at or below 5 lux.

3. The HMD according to claim 1, wherein, The execution of the instruction further causes the HMD to perform parallax correction to compensate for the positional offset between the pupil of the user's eye and the at least one computer vision camera.

4. The HMD according to claim 3, wherein, The parallax correction is performed by performing full reprojection using a full depth map.

5. The HMD according to claim 3, wherein, The parallax correction is performed by performing planar reprojection using a planar depth map.

6. The HMD according to claim 1, wherein, The execution of the instruction further enables the HMD to: In response to the ML algorithm identifying the IR laser in the image, the detected IR laser in the image is tracked to identify the travel path of the IR laser through the environment.

7. The HMD according to claim 6, wherein, The notification includes a display of the tracked path of the IR laser.

8. The HMD according to claim 6, wherein, The notification includes an identifier of the original source of the IR laser.

9. The HMD according to claim 6, wherein, The notification includes a direction indicator that indicates the direction of the IR laser origin.

10. The HMD according to claim 1, wherein, The IR filter is a detachable filter, allowing it to be removed from the at least one computer vision camera.

11. A method for detecting infrared (IR) laser emitted in an environment, the method being performed by a head-mounted device (HMD) configured to include at least one computer vision camera, the at least one computer vision camera omitting an IR filter, such that the sensor of the at least one computer vision camera is operable to detect IR light, including IR laser, in the environment, the method comprising: The environment is generated using the at least one computer vision camera; The image is fed as input to a machine learning (ML) algorithm configured to identify IR lasers detected and recorded in the image by the sensor of the at least one computer vision camera, wherein the ML algorithm is configured to distinguish the IR laser from other IR lights by detecting lines of IR light forming the IR laser, the IR laser being represented by a first set of pixels in the image, and the other IR lights being represented by a second set of pixels in the image; and A visual notification is displayed, which includes information corresponding to the detected IR laser.

12. The method according to claim 11, wherein, The at least one computer vision camera is a head-tracking camera.

13. The method according to claim 11, wherein, The notification is overlaid on an image generated by a thermal imaging camera.

14. The method according to claim 11, wherein, The notification is overlaid on the parallax-corrected video passthrough image.

15. The method according to claim 11, wherein, The HMD does not include a low-light camera.

Images