System and method for separating dual system optical alignment of cameras

Abstract

Systems and methods for separating dual system optical alignment of cameras are disclosed. A first image is generated by an integrated camera. A pose of a computer system is determined based on the image, and a timestamp is determined. A second image is generated by a separate camera. The second image is aligned with the first image. An overlay image is generated by overlaying the second image onto the first image based on the alignment. A pose difference between a current pose of the computer and an initial pose is then identified. Accordingly, late stage re-projection (LSR) is performed on the overlay image to account for the pose difference. The LSR corrected overlay image is then displayed.

Description

Background Technology

[0001] Mixed reality (MR) systems, including virtual reality (VR) and augmented reality (AR) systems, have garnered significant attention for their ability to create truly unique experiences for their users. For reference, conventional VR systems create fully immersive experiences by restricting the user's field of vision solely to a virtual environment. This is often achieved through the use of head-mounted displays (HMDs) that completely block any view of the real world. Thus, the user is fully immersed in the virtual environment. In contrast, conventional AR systems create augmented reality experiences by visually presenting virtual objects that are placed within or interact with the real world.

[0002] As used herein, VR and AR systems are described and referenced interchangeably. Unless otherwise stated, the descriptions herein also apply to all types of MR systems (as detailed above), including AR systems, VR reality systems, and / or any other similar systems capable of displaying virtual content.

[0003] MR systems can also employ different types of cameras to display content to users, such as in the form of passthrough images. Passthrough images or views can help users avoid disorientation and / or safety hazards when transitioning to and / or navigating within an MR environment. MR systems can present views captured by cameras in a variety of ways. However, the process of using images captured by a world-facing camera to provide a view of the real-world environment presents many challenges.

[0004] Several of these challenges arise when attempting to align image content from multiple cameras. Typically, this alignment process requires detailed timestamp and pose information to perform. However, sometimes timestamp data, or even pose data, is unavailable because different cameras may operate in different time domains, resulting in time offsets. Furthermore, sometimes timestamp data is simply unavailable because cameras may operate remotely to each other, and the timestamp data is not transmitted. Aligning image content provides substantial benefits, particularly in hologram placement and generation, and these problems pose a significant obstacle to the field. Therefore, there is a substantial need in this area to improve how images are aligned with each other.

[0005] The subject matter claimed herein is not limited to addressing any shortcomings or embodiments that operate only in environments such as those described above. Rather, this background is provided merely to illustrate an exemplary area of ​​technology in which some of the embodiments described herein can be practiced. Summary of the Invention

[0006] The embodiments disclosed herein relate to systems, devices (e.g., wearable devices, hardware storage devices, etc.) and methods for aligning and stabilizing images generated by an integrated camera physically mounted to a computer system (e.g., possibly an HMD) with images generated by a separate camera physically unmounted from the computer system.

[0007] In some embodiments, a first image is generated using the integrated camera. This first image is used to determine a first pose of the computer system. Additionally, a first timestamp is determined for the first image. The embodiment also acquires a second image generated by the separate camera. The second image is aligned with the first image. An overlay image is generated by overlaying the second image onto the first image based on the alignment process. A pose difference is identified between the current pose of the computer system at the current timestamp and the first pose determined using the first image at the first timestamp. Later reprojection (LSR) is applied to the overlay image to transform the pixels in the overlay image to account for the pose difference identified between the current pose associated with the current timestamp and the first pose associated with the first timestamp. After applying the LSR to the overlay image, the embodiment displays the overlay image.

[0008] This summary is provided to introduce a selection of concepts in a simplified form, which are further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to assist in determining the scope of the claimed subject matter.

[0009] Additional features and advantages will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practice of the teachings herein. The features and advantages of the invention can be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. The features of the invention will become more apparent from the description which follows and the appended claims, or may be learned by practice of the invention as described below. Attached Figure Description

[0010] To describe how the above and other advantages and features can be obtained, a more specific description of the subject matter briefly described above will be presented with reference to specific embodiments shown in the accompanying drawings. It will be understood that these drawings depict only typical embodiments and therefore should not be considered limiting in scope. The embodiments will be described and explained with additional specificity and detail using the drawings, in which:

[0011] Figure 1 An exemplary head-mounted device (HMD) is illustrated.

[0012] Figure 2The illustration shows some of the different features of the camera on the HMD.

[0013] Figure 3 The illustration shows how the HMD and remote camera (i.e., separate camera) can operate in a consistent manner to provide the user with enhanced images.

[0014] Figure 4 The illustration shows examples of integrated and separated camera images.

[0015] Figure 5 The illustration shows how the pose of a computer system can be determined based on images generated by an integrated camera.

[0016] Figure 6 An exemplary alignment process is illustrated, in which image correspondences between integrated camera images and separate camera images are identified so that the two images are aligned together, regardless of the timestamp or orientation of the separate camera images.

[0017] Figure 7 An exemplary alignment process is illustrated, in which an integrated camera image is aligned with a separate camera image based on an estimated attitude determined using an inertial measurement unit (IMUS).

[0018] Figure 8 The flowchart illustrates the process of estimating pose alignment images based on IMU.

[0019] Figure 9 The illustration shows an example of an overlay image generated as a result of overlaying a separate camera image onto an integrated camera image based on a previously performed alignment.

[0020] Figure 10 The illustration shows that the HMD has shifted its pose slightly since the integrated camera image was generated, thus enabling the late reprojection (LSR) process to be performed on the image pixels to account for the new pose / viewpoint.

[0021] Figure 11 Another exemplary illustration of the LSR process is provided.

[0022] Figure 12 The diagram illustrates a generalized flowchart of the disclosed process for aligning image content.

[0023] Figure 13A and Figure 13B A flowchart illustrating an exemplary method for aligning and stabilizing (e.g., via post-reprojection (LSR)) an image generated by an integrated camera physically mounted to a computer system (e.g., an HMD) with an image generated by a separate camera physically unmounted from the computer system.

[0024] Figure 14The illustration provides a high-level overview of an exemplary computer system configured to perform any of the disclosed operations. Detailed Implementation

[0025] The embodiments disclosed herein relate to systems, apparatus (e.g., wearable devices, hardware storage devices, etc.) and methods for aligning and stabilizing (e.g., possibly via post-reprojection (LSR)) images generated by an integrated camera physically mounted to a computer system (e.g., possibly an HMD) with images generated by a separate camera physically unmounted from the computer system.

[0026] In some embodiments, a first image is generated using the integrated camera. This first image is used to determine a first pose of the computer system. Additionally, a first timestamp is determined for the first image. A second image generated by the separate camera is acquired. The second image is aligned with the first image to generate an overlay image. The pose difference between the current pose of the computer system at the current timestamp and the first pose is identified. Later reprojection (LSR) is applied to the overlay image to account for the pose difference. After applying the LSR, the embodiment displays the overlay image.

[0027] Examples of technological benefits, improvements, and practical applications

[0028] The following sections outline some exemplary improvements and practical applications provided by the disclosed embodiments. However, it will be appreciated that these are merely examples, and the embodiments are not limited to only these improvements.

[0029] The disclosed embodiments provide substantial improvements, benefits, and practical applications to the art. By way of example, the disclosed embodiments improve how images are generated and displayed, and improve how image content is aligned, even without using timestamp data.

[0030] In other words, the embodiments address the problem of not having a precise timestamp of an image when attempting to align the content of a remote or separate camera image with another image to create a single composite or overlay image. There could be various reasons why the timestamp information might be unknown. For example, asynchronous wireless communication may occur between multiple devices operating in different time domains, resulting in a situation where the timestamp is unknown. Despite the potential lack of information, the embodiments are still able to perform image alignment because they do not need to perform image matching based on timestamp data. Therefore, the embodiments provide an improvement in the technical field by enabling image matching to be performed without the need for time data.

[0031] Exemplary MR systems and HMDs

[0032] Now pay attention Figure 1The illustration shows an example of a head-mounted device (HMD) 100. The HMD 100 can be any type of MR system 100A, including a VR system 100B or an AR system 100C. It should be noted that although much of this disclosure focuses on the use of an HMD, the embodiments are not limited to the use of an HMD alone. That is, any type of scanning or camera system can be used, even systems that are completely removed or detached from the HMD. Thus, the disclosed principles should be interpreted broadly to cover any type of scanning scenario or device. Some embodiments may not even require the active use of the scanning device itself and can simply use data generated by the scanning device. For example, some embodiments can be practiced at least partially in a cloud computing environment.

[0033] HMD 100 is shown to include one or more scanning sensors 105 (i.e., a type of scanning or camera system), and HMD 100 is capable of using one or more scanning sensors 105 to scan the environment, map the environment, capture environmental data, and / or generate any type of image of the environment (e.g., by generating a 3D representation of the environment or by generating a “penetration” visualization). The one or more scanning sensors 105 may include, but are not limited to, any number or type of scanning device.

[0034] According to the disclosed embodiments, HMD 100 can be used to generate a see-through visualization of a user's environment. As used herein, "see-through" visualization refers to a visualization that reflects the environment from the HMD's point of view, regardless of whether HMD 100 is included as part of an AR or VR system. To generate this see-through visualization, HMD 100 may use its one or more scanning sensors 105 to scan, map, or otherwise record its surrounding environment, including any objects in the environment, and transmit that data to the user for viewing.

[0035] To generate a penetration image, one or more scanning sensors 105 typically rely on their cameras (e.g., head-tracking cameras, hand-tracking cameras, depth cameras, or any other type of camera) to acquire one or more raw images (aka texture images) of the environment. In addition to generating the penetration image, these raw images can also be used to determine depth data that details the distance (e.g., z-axis range or measurement) from the sensor to any object captured by the raw images. Once these raw images are obtained, depth maps (e.g., based on pixel differences) can be calculated based on the depth data embedded in or contained within the raw images, and penetration images can be generated using depth maps for arbitrary reprojection (e.g., one for each pupil) if needed.

[0036] From the described penetration visualization, the user will be able to perceive the content currently in his / her environment without having to remove or reposition the HMD 100. Furthermore, as will be described in more detail later, the disclosed penetration visualization will also enhance the user's ability to view objects within his / her environment (e.g., by displaying additional environmental conditions that may be imperceptible to the human eye). As used herein, the so-called "overlay image" can be one type of penetration image.

[0037] It should be noted that although the main focus of this disclosure is on generating "one" transmission image, the embodiments actually generate separate transmission images for each eye in the user's eyes. That is, two transmission images are typically generated simultaneously with each other. Therefore, although it is often mentioned that the generation appears to be a single transmission image, the embodiments are actually capable of generating multiple transmission images simultaneously.

[0038] In some embodiments, the scanning sensor(s) 105 includes one or more visible light cameras 110, one or more low-light cameras 115, one or more thermal imaging cameras 120, and potentially (although not necessarily, as provided in...) Figure 1 The dashed box in the image represents one or more ultraviolet (UV) cameras 125 and potentially (though not necessarily, as indicated by the dashed box) point illuminators 130. The ellipsis 135 indicates how any other type of camera or camera system (e.g., depth camera, time-of-flight camera, virtual camera, depth laser, etc.) can be included in one or more scanning sensors 105.

[0039] As an example, a camera configured to detect mid-infrared wavelengths can be included within one or more scanning sensors 105. As another example, any number of virtual cameras reprojected from an actual camera can be included within one or more scanning sensors 105 and can be used to generate stereo image pairs. In this way, one or more scanning sensors 105 can be used to generate the stereo image pairs. In some cases, the stereo image pairs can be obtained or generated as a result of any one or more of the following operations: active stereo image generation via two cameras and a point illuminator (e.g., point illuminator 130); passive stereo image generation via two cameras; image generation using structured light via an actual camera, a virtual camera, and a point illuminator (e.g., point illuminator 130); or image generation using a time-of-flight (TOF) sensor, wherein a baseline exists between the depth laser and the corresponding camera, and wherein the field of view (FOV) of the corresponding camera is offset relative to the illumination field of the depth laser.

[0040] Typically, the human eye can perceive light within the so-called "visible spectrum," which includes light (or more precisely, electromagnetic radiation) with wavelengths ranging from approximately 380 nanometers (nm) to approximately 740 nm. As used herein, the visible light camera 110 comprises two or more red, green, and blue (RGB) cameras configured to capture photons within the visible spectrum. Typically, these RGB cameras are complementary metal-oxide-semiconductor (CMOS) type cameras, but other camera types (e.g., charge-coupled devices, CCDs) may also be used.

[0041] RGB cameras are typically stereo cameras, meaning that the fields of view of two or more RGB cameras at least partially overlap each other. Using this overlapping region, images generated by the visible light camera(s) 110 can be used to identify differences between specific pixels that typically represent objects captured by the two images. Based on these pixel differences, embodiments can determine the depth of objects located within the overlapping region (i.e., "stereoscopic depth matching"). Thus, the visible light camera(s) 110 can be used not only to generate penetration visualizations but also to determine object depth. In some embodiments, the visible light camera(s) 110 can capture both visible light and IR light.

[0042] One or more low-light cameras 115 are configured to capture visible light and IR light. IR light is often classified into three different categories, including near-IR, mid-IR, and far-IR (e.g., thermal IR). These categories are determined based on the energy of the IR light. For example, near-IR has relatively high energy due to its relatively short wavelength (e.g., between about 750 nm and about 1000 nm). Conversely, far-IR has relatively low energy due to its relatively long wavelength (e.g., up to about 30,000 nm). Mid-IR has energy values ​​between or in between the near-IR and far-IR ranges. One or more low-light cameras 115 are configured to detect IR light at least in the near-IR range or be sensitive to IR light.

[0043] In some embodiments, one or more visible light cameras 110 and one or more low-light cameras 115 (also known as low-light night vision cameras) operate within substantially the same overlapping wavelength range. In some cases, this overlapping wavelength range is between approximately 400 nanometers and approximately 1000 nanometers. Additionally, in some embodiments, both types of cameras are silicon detectors.

[0044] One distinguishing feature between these two types of cameras relates to the illumination conditions or illumination range(s) under which they actively operate. In some cases, the visible light camera(s) 110 is a low-power camera and operates in environments with illumination between approximately 10 lux and approximately 100,000 lux, or more precisely, the illumination range starts at approximately 10 lux and increases to over 10 lux. In contrast, the low-light camera(s) 115 consumes more power and operates in environments with the illumination range between approximately 1 millilux and approximately 10 lux.

[0045] On the other hand, one or more thermal imaging cameras 120 are configured to detect electromagnetic radiation or IR light in the far IR (i.e., thermal IR) range, but some embodiments also enable one or more thermal imaging cameras 120 to detect radiation in the mid IR range. To clarify, one or more thermal imaging cameras 120 may be long-wave infrared imaging cameras configured to detect electromagnetic radiation by measuring long-wave infrared wavelengths. Typically, one or more thermal imaging cameras 120 detect IR radiation with wavelengths between approximately 8 micrometers and 14 micrometers. Because one or more thermal imaging cameras 120 detect far IR radiation, they are able to operate without limitation under any illumination conditions.

[0046] In some (though not all) cases, one or more thermal imaging cameras 120 include an uncooled thermal imaging sensor. The uncooled thermal imaging sensor uses a detector design based on a specific type of calorimeter, a device that measures the amplitude or power of an incident electromagnetic wave / radiation. To measure the radiation, the calorimeter uses a thin layer of absorbing material (e.g., metal) connected to a thermal reservoir via a thermal chain. The incident wave impacts and heats the material. In response to the heating of the material, the calorimeter detects a temperature-dependent resistance. Changes in ambient temperature cause changes in the temperature of the calorimeter, and these changes can be converted into an electrical signal, thereby generating a thermal image of the environment. According to at least some of the disclosed embodiments, the uncooled thermal imaging sensor is used to generate any number of thermal images. The calorimeter of the uncooled thermal imaging sensor is capable of detecting electromagnetic radiation across a broad spectrum, spanning the mid-IR spectrum, the far-IR spectrum, and even waves up to millimeter dimensions.

[0047] One or more UV cameras 125 are configured to capture light in the UV range. The UV range includes electromagnetic radiation with wavelengths between about 10 nm and about 400 nm. The disclosed one or more UV cameras 125 should be interpreted broadly and can operate in a manner that includes both reflective UV photography and UV-induced fluorescence photography.

[0048] Therefore, as used herein, a “visible light camera” (including a “head-tracking camera”) is a camera primarily used in computer vision to perform head tracking. These cameras are capable of detecting visible light, or even a combination of visible light and IR light (e.g., the range of IR light, including IR light with wavelengths of approximately 850 nm). In some cases, these cameras are global shutter devices with a pixel size of approximately 3 μm. On the other hand, low-light cameras are cameras that are sensitive to both visible light and near-IR. These cameras are larger and may have pixels of approximately 8 μm or larger. These cameras are also sensitive to wavelengths to which silicon sensors are sensitive, which are between approximately 350 nm and 1100 nm. Thermal / long-wavelength IR devices (i.e., thermal imaging cameras) have a pixel size of approximately 10 μm or larger and detect heat radiated from the environment. These cameras are sensitive to wavelengths in the range of 8 μm to 14 μm. Some embodiments also include mid-IR cameras configured to detect at least mid-IR light. These cameras often comprise non-silicon materials (e.g., InP or InGaAs) that detect light in the wavelength range of 800 nm to 2 μm.

[0049] Therefore, the disclosed embodiments can be configured to utilize many different camera types. These different camera types include, but are not limited to, visible light cameras, low-light cameras, thermal imaging cameras, and UV cameras. Images generated from any one or a combination of camera types listed above can be used to perform penetration image generation and even stereo depth matching.

[0050] Typically, one or more low-light cameras 115, one or more thermal imaging cameras 120, and one or more UV cameras 125 (if present) consume relatively more power than one or more visible light cameras 110. Therefore, when not in use, the one or more low-light cameras 115, one or more thermal imaging cameras 120, and one or more UV cameras 125 are typically in a power-off state, where these cameras are either turned off (and therefore do not consume power) or in a reduced operability mode (and therefore consume significantly less power than those cameras when fully operational). In contrast, the one or more visible light cameras 110 are typically in a power-on state, where these cameras are fully operational by default.

[0051] It should be noted that any number of cameras can be provided on the HMD 100 for each of the different camera types. That is, one or more visible light cameras 110 may include 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or more than 10 cameras. However, the number of cameras is typically at least two, so that the HMD 100 can perform penetration image generation and / or stereo depth matching, as previously described. Similarly, one or more low-light cameras 115, one or more thermal imaging cameras 120, and one or more UV cameras 125 may each include 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or more corresponding cameras.

[0052] Figure 2 The illustration shows an exemplary HMD 200, which represents a source... Figure 1 The HMD 100 is shown. The HMD 200 is shown as including multiple different cameras, including cameras 205, 210, 215, 220, and 225. Cameras 205-225 indicate that they are from... Figure 1 Any number or combination of one or more visible light cameras 110, one or more low-light cameras 115, one or more thermal imaging cameras 120, and one or more UV cameras 125. Although in Figure 2 The image only shows 5 cameras, but the HMD 200 can include more or fewer than 5 cameras.

[0053] In some cases, the camera can be positioned at a specific location on the HMD 200. For example, in some cases, the first camera (e.g., camera 220) is positioned on the HMD 200 at a location above the designated left eye position of any user wearing the HMD 200 in the height direction relative to the HMD. For example, camera 220 is positioned above the pupil 230. As another example, the first camera (e.g., camera 220) is additionally positioned above the designated left eye position in the width direction relative to the HMD. That is, camera 220 is not only positioned above the pupil 230, but also in a straight line relative to the pupil 230. When using a VR system, the camera may be placed directly in front of the designated left eye position. For example, see reference... Figure 2 The camera can be physically positioned on the HMD200 in front of the pupil 230 along the z-axis.

[0054] When a second camera (e.g., camera 210) is provided, the second camera can be positioned on the HMD at a location above the designated right eye position of any user wearing the HMD in the height direction relative to the HMD. For example, camera 210 is above the pupil 235. In some cases, the second camera is additionally positioned above the designated right eye position in the width direction relative to the HMD. When using a VR system, the camera can be placed directly in front of the designated right eye position. For example, see reference... Figure 2 The camera can be physically positioned on the HMD 200 in front of the pupil 235 in the z-axis direction.

[0055] When a user wears the HMD 200, the HMD 200 is fitted to the user's head, and the display of the HMD 200 is positioned in front of the user's pupils (such as pupils 230 and 235). Typically, cameras 205-225 will be physically offset from the user's pupils 230 and 235 by a certain distance. For example, there may be a vertical offset in the HMD height direction (i.e., the "Y" axis), as shown by offset 240. Similarly, there may be a horizontal offset in the HMD width direction (i.e., the "X" axis), as shown by offset 245.

[0056] HMD 200 is configured to provide one or more passthrough images 250 for viewing by a user of HMD 200. In doing so, HMD 200 is able to provide a real-world visualization without requiring the user to remove or reposition HMD 200. These one or more passthrough images 250 effectively represent a view of the environment from the HMD's perspective. Cameras 205-225 are used to provide these one or more passthrough images 250. In some implementations, embodiments utilize a planar reprojection process when generating the passthrough images. Using such a planar reprojection process is acceptable when objects in the environment are sufficiently far from the HMD. Therefore, in some cases, embodiments are able to avoid performing parallax correction because objects in the environment are far enough away that this distance results in negligible errors regarding depth visualization or parallax issues.

[0057] Integrated camera and separate camera operation

[0058] Now pay attention Figure 3 The illustration shows environment 300 in which the HMD 305 is operating. The HMD 305 indicates that it originates from... Figure 2 HMD 200 in the middle.

[0059] In this scenario, the HMD 305 includes an integrated camera 310 that is physically mounted to the HMD 305. For example, the integrated camera 310 could be... Figure 2 Any of the cameras 205-225 mentioned above. Similarly, the integrated camera 310 can be in... Figure 1 Any of the cameras mentioned herein, such as one or more visible light cameras 110, one or more low-light cameras 115, one or more thermal imaging cameras 120, or even one or more UV cameras 125. The integrated camera 310 is shown scanning the environment 300 via a field of view (FOV) 315. That is, objects included in the FOV 315 will be represented in the image generated by the integrated camera 310.

[0060] Figure 3 The presence or use of the detachable camera 320 is also shown. Here, the detachable camera 320 is physically removed from the HMD 305. For example, in this particular scenario, the detachable camera 320 is strapped to or otherwise placed on the user's chest. In some scenarios, the detachable camera 320 may not be placed on the user's body, but may instead be placed on an object held by the user. As an example, suppose the detachable camera 320 is mounted on a selfie stick or another type of extension pole. In some cases, the detachable camera 320 may be attached to another device being used by the user. In some cases, the detachable camera 320 may be completely removed from the user's control, such as when the detachable camera 320 is placed on the ground or may be placed on another user.

[0061] Figure 3 This illustrates how the separate camera 320 is associated with its corresponding field of view (FOV) 325. That is, objects contained within the FOV 325 will be captured or included in images generated by the separate camera 320. One will then realize how both the integrated camera 310 and the separate camera 320 can generate still images and videos without limitation.

[0062] According to the disclosed principles, at least a portion of FOV 315 overlaps with FOV 325, as shown by overlap condition 330. This overlap 330 enables embodiments to generate multiple images and then overlay image content from one image onto another to generate a composite image or overlaid image with enhanced features that would not exist if only a single image were used.

[0063] It should be noted that although this disclosure primarily focuses on the use of two images, embodiments are capable of aligning content from two or more images with overlapping areas. For example, suppose 2, 3, 4, 5, 6, 7, 8, 9, or even 10 images have overlapping content. Embodiments are capable of examining each image and then aligning specific portions with each other. The resulting overlay image can then be a composite image formed by any combination or alignment of available images (e.g., even 10 or more images, if available). Therefore, when performing the disclosed operations, embodiments are capable of utilizing any number of images, and are not limited to just two images.

[0064] Assuming the integrated camera 310 is a low-light camera, and further assuming the separate camera 320 is a thermal imaging camera, as will be discussed in more detail later, embodiments are capable of selectively extracting image content from the thermal imaging camera image and overlaying that image content onto the image generated by the low-light camera. In this respect, the thermal imaging content can be used to enhance or supplement the low-light image content, thereby providing the user with enhanced image content. Further details regarding these characteristics will be provided later.

[0065] Figure 4 It shows the result of Figure 3 The integrated camera 310 generates a resulting image in the form of an integrated camera image 400. Provides an image for the integrated camera image 400. Figure 4 The shadow shown is used to distinguish this image from any other image. The shadow should not be interpreted as meaning that the integrated camera image 400 is an image of any particular type.

[0066] By analyzing the content contained in the integrated camera image 400, the embodiment is able to determine the HMD (e.g., from...). Figure 3 The orientation 405 of the HMD (305) in the embodiment. For example, by detecting anchor points (e.g., points identified as relatively stationary or not moving), the embodiment is able to determine the orientation or orientation 405 of the HMD relative to the surrounding environment.

[0067] Additionally, a timestamp 410 can be determined for the integrated camera image 400. The timestamp 410 identifies the time when the integrated camera image 400 was generated. Of course, the timestamp 410 can be based on any timing calculation, including absolute time such as that determined by an atomic clock, or alternatively, relative time of any type such as a processor clock cycle.

[0068] From Figure 3The integrated camera 310 generates an integrated camera image 400, and the integrated camera 310 operates at a specific refresh rate 415 to generate a new image. This refresh rate 415 can be set to any value. However, typically, the refresh rate 415 is at least between 30 Hz and 90 Hz. In some cases, the refresh rate 415 is higher than 90 Hz, such as possibly 120 Hz or higher. Typically, the refresh rate 415 is around 90 Hz.

[0069] Figure 4 It also shows the source from Figure 3 Separate camera image 420 generated by separate camera 320. Figure 4 In this illustration, the separate camera image 420 is shown to be smaller than the integrated camera image 400, but this size difference is for illustrative purposes only. In some cases, the separate camera image 420 may have a higher resolution than the integrated camera image 400, while in other cases, the separate camera image 420 may have a lower resolution than the integrated camera image 400. In some cases, the two images may have the same resolution.

[0070] The integrated camera image 400 (e.g., the "first" image) can be one of a visible light image, a low-light image, or a thermal image. The separate camera image 420 (e.g., the "second" image) can be a different image from the visible light image, the low-light image, or the thermal image, or it can even be an image of the same type as the first image.

[0071] Similar to the discussion regarding the integrated camera image 400, the embodiment can also use the separated camera image 420 to determine some additional information. Note that these operations are not strictly necessary and can be skipped or avoided in some cases. Thus, the following discussion involves some operations that may or may not be performed.

[0072] Specifically, the embodiment is able to analyze and separate the contents of the camera image 420 to determine the source of the image. Figure 3 The pose 425 of the separate camera 320. Similarly, a timestamp 430 can be determined for the separate camera image 420. In some cases, timestamp 410 is different from timestamp 430, or reflects different times, so that the two images can have a time offset.

[0073] The separate camera 320 can also have its own refresh rate 435. This refresh rate 435 can be set to any value. However, typically, the refresh rate 435 is at least between 10 Hz and 60 Hz. In some cases, the refresh rate 435 is higher than 60 Hz, such as possibly 90 Hz or 120 Hz or even higher. Typically, the refresh rate 435 is approximately 30 Hz. In some cases, the refresh rate 435 is the same as the refresh rate 415, while in other cases, the refresh rate 435 is different from the refresh rate 415. When the two refresh rates are different, then the two cameras (e.g., from...) Figure 3 The integrated camera 310 and the separate camera 320 operate in different time domains.

[0074] As briefly described earlier, in some instances, embodiments are able to avoid determining attitude 425 and timestamp 430. Black arrows 440, marked as non-dependent, indicate how embodiments can avoid determining attitude 425, and black arrows 445, marked as non-dependent, indicate how embodiments can avoid determining timestamp 430. In some cases, this non-dependency is based on a lack of information (e.g., such as in the case of data not being transmitted), or on embodiments that avoid calculating information. Further details regarding these aspects will be provided later.

[0075] Regarding attitude determination Figure 5 Some additional information was provided. Figure 5 An integrated camera 500 is shown, which represents the integrated camera discussed so far. Figure 5 The attitude 505 is also shown, and its representation comes from Figure 4 The attitude 505 refers to at least the xyz position of the integrated camera 500 relative to its environment, according to the disclosed principles.

[0076] In some cases, pose 505 may include detailed information on 6 degrees of freedom (DOF) or 6 DOF 510. Typically, 6 DOF 510 refers to the movement or position of an object in three-dimensional space. 6 DOF 510 includes surge (i.e., forward and backward along the x-axis), heave (i.e., downward along the z-axis), and sway (i.e., left and right along the y-axis). In this respect, 6 DOF 510 refers to a combination of three translations and three rotations. Any possible movement of the subject can be represented using 6 DOF 510.

[0077] In some cases, pose 505 may include information detailing 3 DOF 515. Typically, 3 DOF 515 refers to tracking only rotational motions, such as pitch (i.e., horizontal axis), yaw (i.e., normal axis), and roll (i.e., vertical axis). In this respect, 3 DOF 515 allows the HMD to track rotational motions rather than translational movements. To further explain, 3 DOF 515 allows the HMD to determine whether the user (the person wearing the HMD) is looking left or right, whether the user is rotating his / her head up or down, or whether the user is pivoting to the left or right. Unlike 6 DOF 510, when using 3 DOF 515, the HMD cannot determine whether the user has moved in a translational manner, such as by moving to a new position in the environment.

[0078] Deterministic 6 DOF 510 and 3 DOF 515 can be performed using built-in sensors such as accelerometers, gyroscopes, and magnetometers. Deterministic 6 DOF 510 can also be performed using position tracking sensors such as head-tracking sensors.

[0079] Image correspondence and alignment

[0080] Based on the disclosed principles, the embodiments are able to [achieve the following] Figure 4 The integrated camera image 400 shown is aligned with the separate camera image 420. That is, because at least a portion of the FOV of the two cameras overlaps with each other, as in... Figure 3 As described herein, at least a portion of the resulting image includes corresponding content. Therefore, corresponding content can be identified, and merged, blended, or overlaid images can then be generated based on similar corresponding content. By generating this overlaid image, the embodiment can provide the user with enhanced image content that would be unavailable if only a single image type were provided to the user. Figure 6 The illustration shows a first type of alignment 600, which can be used to align image content from two (or possibly more) different images.

[0081] Figure 6 The text indicates that the source is... Figure 4 The integrated camera image 400 and the integrated camera image 605 represent the images from the camera. Figure 4 Separated camera image 420 and separated camera image 610. These two types of images are also often referred to as "texture" images.

[0082] The embodiments are capable of analyzing textured images (i.e., performing computer vision feature detection) to attempt to find any number of feature points. As used herein, the phrase "feature detection" generally refers to the process of computing an image abstraction and then determining (e.g., a particular type) the presence of image features at any particular point or pixel in the image. Typically, corners (e.g., the corner of a wall), distinguishable edges (e.g., the edge of a table), or ridges are used as feature points because edges or corners have inherent or striking visual contrast.

[0083] Figure 6 Several exemplary feature points in the integrated camera image 605 are shown, such as feature points 615A, 620A, and 625A. Other feature points are identified using black circles but are not labeled. Note that these feature points are related to corners, edges, or other ridges, such as folds in blankets and pillows, and corners of images and walls. Any type of feature detector can be programmed to identify feature points. In some cases, the feature detector may be a machine learning algorithm.

[0084] As used herein, references to any type of machine learning can include any type of machine learning algorithm or device, one or more convolutional neural networks, one or more multilayer neural networks, one or more recurrent neural networks, one or more deep neural networks, one or more decision tree models (e.g., decision trees, random forests, and gradient boosting trees), one or more linear regression models, one or more logistic regression models, one or more support vector machines (“SVM”), one or more artificial intelligence devices, or any other type of intelligent computing system. Any amount of training data can be used (and may be refined later) to train the machine learning algorithm to dynamically perform the disclosed operations.

[0085] Figure 6 The embodiments also illustrate how the separated camera image 610 can be analyzed, examined, or reviewed to identify feature points, as indicated by the black circles. Examples include, but are not limited to, feature point 615B, feature point 620B, and feature point 625B.

[0086] Based on the disclosed principles, embodiments detect any number of feature points (e.g., 1, 2, 3, 4, 6, 7, 8, 9, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 200, 500, 1000, 2000, or more than 2000) and then attempt to identify correlations or correspondences between feature points detected in the integrated camera image 605 and feature points identified in the separated camera image 610. For example, a correspondence 615C has been identified in which feature point 615A is linked to or corresponds to feature point 615B. Similarly, a correspondence 620C has been identified in which feature point 620A is determined to correspond to feature point 620B. A correspondence 625C has been identified in which feature point 625A is determined to be aligned to or correspond to feature point 625B. Although only three correspondences should be visualized, one will realize how to identify any number of correspondences.

[0087] The sum or compilation of the identified correspondences (e.g., correspondences 615C, 620C, and 625C) constitutes one or more image correspondences 630. Thus, in some embodiments, the alignment 600 process includes identifying any number of feature points and then identifying correlations or correspondences between feature points in two (or more) different images.

[0088] Note that in this implementation, the embodiments avoid determining the pose or timestamp of the separated camera images 610. Instead, the embodiments rely on feature matching to determine whether image content should be overlaid from one image onto another. With further clarification, these embodiments are similar in pose to... Figure 4 It is independent of 440 and independent of 445 in terms of timestamp.

[0089] Some embodiments then adapt one or more feature or image correspondences 630 to motion model 635 to overlay one image onto another to form an enhanced overlay image. Motion model 635 can be any type of motion model. Typically, a motion model is a transformation matrix that allows a model, known scene, or object to be projected onto different models, scenes, or objects.

[0090] In some cases, motion model 635 can simply be a rotational motion model. Using a rotational model, embodiments can move an image by any number of pixels (e.g., possibly 5 pixels to the left and 10 pixels up) to overlay one image onto another. For example, once image(s) corresponding to 630 are identified, embodiments can identify those feature points or their corresponding pixel coordinates. Once the coordinates are identified, embodiments can use the rotational motion model scheme described above to overlay the separate camera image 610 onto the integrated camera image 605.

[0091] In some cases, the motion model 635 can be more complex, such as in the form of a similarity transformation model. This similarity transformation model can be configured to allow (i) rotating either the integrated camera image 605 (e.g., the "first" image) or the separate camera image 610 (e.g., the "second" image), (ii) scaling either the first or the second image, or (iii) homography transformation of either the first or the second image. In this regard, the similarity transformation model scheme can be used to overlay image content from one image onto another. Further details regarding this overlay process will be provided later. Therefore, in some cases, the process of aligning a separate camera image (e.g., the "second" image) with an integrated camera image (e.g., the "first" image) is performed by: (i) identifying an image correspondence between the second and first images, and then (ii) adapting the identified image correspondence to the motion model such that the second image is projected onto the first image.

[0092] Figure 7 The illustration shows another alignment 700 operation that can be performed to align the content from two images, thereby enabling the content to be overlaid to form an overlay image. Specifically, Figure 7 An integrated camera image 705 and a separate camera image 710 are shown, both representing their corresponding images discussed in the previous figures.

[0093] The integrated camera image 705 includes a texture 715. As used herein, texture 715 generally refers to information about the spatial arrangement of colors or intensities contained in the image. Similarly, the separate camera image 710 is shown as including a texture 720.

[0094] Based on the alignment 700 operation, the embodiment determines that the texture 715 in the integrated camera image 705 and / or the texture 720 in the separated camera image 710 are insufficient to perform feature matching or image correspondence matching, such as combining Figure 6 As described. For example, there may be an insufficient number of feature points detected in either of the two images. Alternatively or alternatively, a sufficient number of feature points may be detected, but an insufficient number of corresponding points may be identified. Based on this initial determination, the embodiment relies on or reverts to alignment 700 operation, which is based on a predicted or estimated attitude determined by various inertial measurement units (IMUs).

[0095] Specifically, the integrated camera that generates integrated camera image 705 is associated with a first IMU 725. Similarly, the separate camera that generates separate camera image 710 is associated with a second IMU 730. Embodiments utilize IMU 725 to determine the pose of the integrated camera, possibly based on initial guiding vision (e.g., an initial base image generated by the integrated camera) combined with IMU data generated by IMU 725. Similarly, embodiments utilize IMU 730 to determine the pose of the separate camera, possibly based on initial guiding vision (e.g., an initial base image generated by the separate camera) combined with IMU data generated by IMU 730.

[0096] Once these two poses are estimated or determined, as shown by IMU estimated pose 735 and IMU estimated pose 740, the embodiment then uses those poses to align one or more portions of the image with each other. Once aligned, one or more portions of one image (the aligned portions) are overlaid onto the corresponding portions of the other image to generate an enhanced overlaid image. In this respect, the IMU can be used to determine the pose of the corresponding camera, and these poses can then be used to perform the alignment process. Figure 8 The illustration shows the use in Figure 7 The IMU discussed herein will come from the second image (e.g., from...). Figure 7 An exemplary flowchart 800 shows the alignment of the image content of the separated camera image 710 with a first image (e.g., the integrated camera image 705).

[0097] Specifically, flowchart 800 initially includes an action (action 805) attempting to identify an image correspondence between the second image and the first image. For example, an embodiment may initially attempt to perform... Figure 6 The alignment 600 operation discussed in the paper involves feature points being attempted for alignment.

[0098] Then, flowchart 800 includes determining that one or both of the second and first images lacks sufficient or threshold amounts of texture to identify the corresponding action (action 810). For example, from Figure 7 Textures 715 or 720 may not meet the texture threshold, making it impossible to identify a sufficient number of corresponding images or the threshold number.

[0099] Then, flowchart 800 includes a first inertial measurement unit (IMU) using a computer system (e.g., possibly from...). Figure 7 The IMU 725 in the computer system is used to estimate the action (action 815) of the IMU estimated pose (e.g., IMU estimated pose 735).

[0100] In parallel or serial with action 815, flowchart 800 includes an action (action 820) to estimate the IMU estimated pose of the separated camera (e.g., IMU estimated pose 740) using a second IMU (e.g., possibly IMU 730) of the separated camera.

[0101] Then, flowchart 800 includes an action (action 825) to align the second image with the first image by aligning the IMU-estimated pose of the computer system with the IMU-estimated pose of the separate camera. In this respect, flowchart 800 generally outlines the combination of... Figure 7 The alignment process discussed in section 700. Therefore, a variety of different alignment techniques can be used to align image content or identify image correspondences.

[0102] Based on any alignment process used, the embodiment then generates an overlay image, as shown in Figure 9 As shown. Specifically, Figure 9 An overlay image 900 is shown, which includes image content 905 and image content 910. Of course, the image content can be pulled or extracted from any number of images that are aligned with each other without restriction.

[0103] From the integrated camera images discussed so far (e.g., Figure 7 Image content 905 is extracted, pulled, or drawn from the integrated camera image 705, while image content 910 is extracted, pulled, or drawn from the separate camera image discussed so far (e.g., separate camera image 710). In some cases, image content 905 includes all image content from the integrated camera image, while in other cases, image content 905 includes only a portion of the image content from the integrated camera image. Similarly, in some cases, image content 910 may include all image content from the separate camera image, while in other cases, image content 910 includes only a portion of the image content from the separate camera image.

[0104] In some cases, the amount included in image content 905 and 910 can depend on the degree or level of overlap between the FOVs of the integrated camera and the separate camera. (Reference) Figure 3 In this scenario, FOV 315 completely consumes, overlaps with, or encloses FOV 325. The resulting integrated camera image may then include all the content included in the separate camera image. If only a portion of the two images overlaps, then only the content associated with that portion can be included in the overlay image 900.

[0105] Generating this overlay image 900 is highly beneficial for many reasons. For example, suppose image content 905 is low-light or visible-light content, and suppose image content 910 is thermal imaging content. The thermal imaging content can be used to enhance or supplement the low-light or visible-light content by providing an increased amount of situational awareness about the environment.

[0106] In some cases, image content 910 and / or image content 905 may be at least partially transparent. For example, suppose image content 910 is overlaid on top of image content 905. Image content 905 may include the content currently being overlaid by image content 910. If image content 910 is at least partially transparent, both image content 905 and image content 910 will be visible, thereby providing even further visual enhancement or even further visual information. The transparency can be set to any value. For example, the transparency can be set to 1%, 5%, 10%, 15%, 20%, 25%, 50%, 75%, or even up to 99%, or any value in between.

[0107] Late reprojection

[0108] When image frames (e.g., from) Figure 9 When the overlay image (900) in the image is rendered, the embodiment can determine whether the pose depicted in that frame matches the current pose of the computer system. If the pose matches, the image can be displayed to the user. On the other hand, if the pose does not match, a late reprojection (LSR) process can be performed to transform the pixels in the image to compensate for the new pose. Typically, LSR is performed to correct only 3 DOF changes (e.g., yaw, pitch, roll) because objects are often removed from the HMD, making it possible to avoid forward or backward projection due to planar reprojection of objects in the scene or planar viewpoints (e.g., all objects can be assigned the same planar depth). However, in some cases, LSR can be performed to correct 6 DOF changes.

[0109] To clarify, the process of generating and rendering frames is not instantaneous; instead, it takes some time to execute. For example, at 60 frames per second (FPS), rendering an application or HMD takes approximately 16.667 milliseconds (ms) to render a frame. While this is a small duration in time, it's possible that the HMD has moved during that period (e.g., the user may have moved, causing the HMD to move). LSR is the process of transforming or modifying pixels in an image (e.g., overlaying image 900) to account for shifts in viewpoint or pose.

[0110] With additional clarification, in order to reduce or eliminate some rendering errors or problems caused by pose variations over time, HMD can apply post-correction to make final adjustments to the image after it has been rendered by the GPU. This process is performed before the pixels are displayed to compensate for the latest rotation, translation, and / or magnification caused by the user's head movement. This adjustment process is often referred to as "post-correction," "post-reprojection," "LSR," or "LSR adjustment." Figure 10 and Figure 11 Some useful examples of these LSR operations are provided.

[0111] Figure 10 An integrated camera 1000 is shown, representing the integrated camera discussed so far. As previously discussed, the embodiment is able to determine the pose 1005 of the integrated camera 1000 and the timestamp 1010 of the image generated by the integrated camera 1000. In this exemplary scenario, the pose 1005 and the timestamp 1010 are at time T0.

[0112] Before displaying the overlay image, the HMD has been moved, so that the integrated camera 1015 representing the integrated camera 1000 has been moved. Now, the integrated camera 1015 has a new or current pose 1020 and a new or current timestamp 1025 reflecting time T1. Previous timestamps (such as those from...) Figure 4 The timestamps 410 and 430 in the current time stamp are different from the current timestamp 1025. The current pose 1020 can be determined using any technique, including IMU data, head tracking data, or any other technique used for pose identification.

[0113] The pose difference 1030 represents the difference between pose 1005 and the current pose 1020. The pose difference 1030 can be represented using 6 DOF information or 3 DOF information. As a result of this detected pose difference 1030, an embodiment is triggered to perform LSR. Note that LSR can be performed on integrated camera images, separated camera images, or overlaid images. Figure 11 An exemplary scenario in which LSR is performed on an overlay image is shown.

[0114] Specifically, Figure 11 The overlay image 1100 is shown, which represents the image from... Figure 9 The overlay image 900. The overlay image 1100 includes any number of pixels 1105, such as pixels A, B, C, D, E, F, G, H, I, J, K, L, M, N, O and P. Figure 11 The LSR 1110 operation performed on the overlay image 1100 is also shown to take into account the detected new poses, such as those from... Figure 10 The current posture is 1020.

[0115] As a result of performing the LSR 1110 operation, an LSR-corrected overlay image 1115 is generated. Note that, as shown by the LSR-corrected pixel 1120, one, some, or all of the pixels in the overlay image 1100 have been transformed. For example, pixel A' is a transformed version of pixel A. Similarly, pixel B' is a transformed version of pixel B. Pixels C', D', E', F', G', H', I', J', K', L', M', N', O', and P' are transformed versions of pixels C, D, E, F, G, H, I, J, K, L, M, N, O, and P, respectively. By performing the LSR 1110 operation, the embodiment is able to correct or compensate for new poses detected by the HMD, including the integrated camera. Figure 12 The diagram provides a summary of the principles discussed so far.

[0116] Specifically, Figure 12 The diagram illustrates a first time domain 1200 and a second time domain 1205 during which images are generated. The first time domain 1200 can be associated with an integrated camera as discussed so far, while the second time domain 1205 can be associated with a separate camera. For example, the integrated camera can generate images at a rate of 90 Hz, while the separate camera can generate images at a rate of 30 Hz.

[0117] As shown on the "Time" axis, images 1210A, 1210B, 1210C, 1210D, 1210E, 1210F, 1210G, 1210H, and 1210I can be generated by the integrated camera over the entire time period. In this example, the separate camera generates images at a lower or reduced rate, as shown by images 1215A, 1215B, and 1215C. For example, the integrated camera generates three images for each image generated by the separate camera.

[0118] Based on the disclosed principles, the embodiments then perform the alignment process described previously to generate an overlay image throughout the time. For example, the embodiments utilize images 1210B and 1215A to generate an overlay image 1220A. Subsequently, the embodiments use image 1210C and reuse the same image 1215A to generate an overlay image 1220B. Subsequently, the embodiments use image 1210D and reuse image 1215A to generate an overlay image 1220C. In this respect, a single image (e.g., image 1215A) can be used consecutively multiple times in combination with other images to generate an overlay image. The refresh rates of the two cameras can be used to determine the number of iterations in which a single camera image can be reused. Using refresh rates of 90 Hz and 30 Hz, one of the separate camera images can be used at least three times. Other refresh rate ratios will determine the number of times a single image will be used. Note that the overlay image is generated by performing different alignment processes discussed previously.

[0119] To complete the example, overlay image 1220D is generated based on the combination of image 1210E and image 1215B. Overlay image 1220E is generated based on the combination or alignment of image 1210F and image 1215B. Overlay image 1220F is generated based on the alignment of image 1210G and image 1215B. Overlay image 1220G is generated based on the alignment of image 1210H and image 1215C. Overlay image 1220H is generated based on the combination of image 1210I and image 1215C.

[0120] Subsequently, the embodiment performs LSR operations on the overlay image, the integrated camera image, and / or the separated camera image. Figure 12 In the example shown, the embodiment performs LSR on the overlay image. For example, LSR 1225A is performed on overlay image 1220A, LSR 1225B is performed on overlay image 1220B, LSR 1225C is performed on overlay image 1220C, LSR 1225D is performed on overlay image 1220D, LSR 1225E is performed on overlay image 1220E, LSR 1225F is performed on overlay image 1220F, LSR 1225G is performed on overlay image 1220G, and LSR 1225H is performed on overlay image 1220H.

[0121] The LSR-corrected image is then displayed on a monitor for the user to view. For example, after performing LSR 1225A on overlay image 1220A, the embodiment displays the LSR-corrected image obtained at 1230A. Subsequently, the embodiment displays the next LSR-corrected image, and so on, as illustrated by displays 1230B, 1230C, 1230D, 1230E, 1230F, 1230G, and 1230H. Each of these obtained LSR-corrected images is then displayed relative to each other in time, as shown by a timeline. Similarly, the rate at which the LSR-corrected images are displayed can correspond to a faster rate for either the integrated camera or the separate camera. In this case, the integrated camera refreshes at a faster rate compared to the separate camera. Therefore, the display of the LSR-corrected images, or more precisely, the display rate of these images, can correspond to the rate of the integrated camera. In this case, the display rate of the LSR-corrected images could be 90 Hz, just like the rate of the integrated camera.

[0122] Exemplary methods

[0123] The following discussion now concerns the many methods and method actions that can be performed. Although method actions may be discussed in a specific order or illustrated in a flowchart as occurring in a specific order, a specific order is not required unless otherwise specified, or because an action depends on another action that is completed before that action is performed.

[0124] Figure 13A and Figure 13B A flowchart is illustrated for an exemplary method 1300 for aligning and stabilizing (e.g., via LSR) an image generated by an integrated camera (e.g., any integrated camera discussed so far) that is physically mounted to a computer system with an image generated by a separate camera (e.g., any separate camera discussed so far) that is physically unloaded from the computer system.

[0125] The computer system can be a head-mounted device (HMD) worn by the user. In some implementations, the integrated camera is one camera selected from a group of cameras, including visible light cameras, low-light cameras, or thermal imaging cameras. Similarly, the separate camera is also one camera selected from the group. Furthermore, the separate camera can be oriented such that its field of view (FOV) at least partially overlaps with the FOV of the integrated camera. Of course, any number of additional mounted or unmounted cameras can also be used, provided that their FOVs also overlap.

[0126] First, method 1300 includes the action (action 1305) of generating a first image using an integrated camera. The first image may represent any integrated camera image among those discussed so far.

[0127] Then, method 1300 includes an action (action 1310) to determine a first pose of the computer system using the first image. For example, from Figure 10 The posture 1005 and from Figure 4 The posture 405 in the text represents the "first" posture.

[0128] In parallel or sequentially with action 1310, method 1300 includes an action (action 1315) to determine a first timestamp of the first image. Figure 10 The timestamp 1010 and from Figure 4 The timestamp 410 in the code represents the first timestamp.

[0129] Method 1300 also includes the action of acquiring a second image generated by the separate camera (action 1320). Action 1320 may be performed before, after, or during any of actions 1305, 1310, or 1315. Furthermore, any separate camera image in the disclosed separate camera images represents the "second" image. In some cases, method 1300 includes the action (not shown) of determining that the integrated camera and the separate camera operate in different temporal domains. Based on this detected difference, embodiments are able to determine the frequency of reuse of a particular image, such as by... Figure 12 The image 1215A is shown as a repetition. As a specific example, the integrated camera can be detected as operating in conjunction with a 90 Hz refresh rate for displaying content, while the separate camera can be detected as operating in conjunction with a 30 Hz refresh rate for displaying content.

[0130] Then, method 1300 includes an action (action 1325) to align the second image with the first image. Figure 6 or Figure 7 Any alignment process described herein can be used to perform the alignment process described in action 1325. For example, in some cases, alignment of the second image with the first image is performed by identifying an image correspondence between the second image and the first image, as in... Figure 6 As illustrated, in cases where image correspondence is used to align images, the process of aligning the second image with the first image is performed without relying on the timestamp or pose associated with the second image. Alternatively, the alignment process is based solely on feature correspondence. In some cases, aligning the two images is based on IMU-estimated pose, as in... Figure 7 As shown in the diagram.

[0131] Subsequently, method 1300 includes an action (1330) to generate an overlay image by overlaying the second image onto the first image, based on the alignment process described in action 1325. Figure 12The overlay images 1220A-1220H shown in the figure and those from Figure 9 Any image in the overlay image 900 can represent the overlay image in action 1330.

[0132] Method 1300 continues in Figure 13B The process includes an action (action 1335) that identifies the pose difference between the computer system's current pose at the current timestamp and a first pose determined using a first image at the first timestamp. For example, Figure 10 The posture difference 1030 represents the posture difference described in action 1335, wherein the posture difference 1030 is based on the difference between posture 1005 determined at timestamp 1010 and current posture 1020 determined at current timestamp 1025.

[0133] Then, method 1300 includes the action of applying late reprojection (LSR) to the overlay image (action 1340). Figure 11 LSR 1110 in the text refers to the LSR operation described in action 1340. This LSR process transforms the pixels in the covered image (e.g., Figure 11 (Pixel 1105) to take into account the pose difference identified between the current pose associated with the current timestamp and the first pose associated with the first timestamp. The transformation produces Figure 11 The LSR-corrected pixel count is 1120.

[0134] After applying LSR to the overlay image, method 1300 includes an action (action 1345) to display the overlay image, which is a type of pass-through image. The image can be displayed on the HMD's display.

[0135] Therefore, the disclosed embodiments can be used to bring substantial improvements to how visual content is generated, aligned, and displayed. For example, image content from one image can be extracted and overlaid onto another image to provide an enhanced visualization for the user. This visualization will enable the user to improve his / her interaction with the computer system. Furthermore, the disclosed alignment process can be performed regardless of any timing differences between the images.

[0136] Exemplary computer / computer system

[0137] Now pay attention Figure 14The illustration depicts an exemplary computer system 1400, which may include and / or be used to perform any of the operations described herein. The computer system 1400 may take various forms. For example, the computer system 1400 may be embodied as a tablet computer 1400A, a desktop computer or laptop computer 1400B, a wearable device such as an HMD 1400C (which represents an HMD discussed herein), a mobile device, or any other type of standalone device, as indicated by ellipsis 1400D. The computer system 1400 may also be a distributed system comprising one or more connected computing components / devices communicating with the computer system 1400.

[0138] In its most basic configuration, the computer system 1400 includes a variety of different components. Figure 14 The computer system 1400 shown includes one or more processors 1405 (also known as "hardware processing units") and storage devices 1410. Although not shown, the computer system 1400 may include a combination of... Figure 1 and Figure 2 Any of the features described herein, and any other features described herein. It should be noted that none of the disclosed features are mutually exclusive, and any feature described herein may be combined with any other feature described herein.

[0139] Regarding processor 1405, it should be understood that the functions described herein can be performed at least in part by one or more hardware logic components (e.g., processor 1405). Examples, but not limited to, illustrative types of hardware logic components / processors that may be used include field-programmable gate arrays (“FPGAs”), application-specific integrated circuits (“ASICs”), application-specific standard products (“ASSPs”), systems-on-a-chip (“SOCs”), complex programmable logic devices (“CPLDs”), central processing units (“CPUs”), graphics processing units (“GPUs”), or any other type of programmable hardware.

[0140] The computer system 1400 and the scanning sensor can utilize any type of depth detection. Examples include, but are not limited to, stereo depth detection (active illumination (e.g., using a point illuminator), structured light illumination (e.g., one physical camera, one virtual camera, and one point illuminator) and passive (i.e., no illumination)), time-of-flight depth detection (with a baseline between the laser and the camera, where the camera's field of view does not completely overlap with the laser's illumination field), rangefinder depth detection, or any other type of range or depth detection.

[0141] As previously discussed, machine learning (ML) can also be used in the disclosed embodiments. ML can be implemented as a specific processing unit (e.g., a dedicated processing unit as previously described) configured to perform one or more dedicated operations of computer system 1400. As used herein, the terms “executable module,” “executable component,” “component,” “module,” “model,” or “engine” can refer to a hardware processing unit or a software object, routine, or method executable on computer system 1400. The various components, modules, engines, models, and services described herein can be implemented as objects or processors executable on computer system 1400 (e.g., as separate threads). The ML model and / or processor 1405 can be configured to perform one or more of the disclosed method actions or other functions.

[0142] Storage device 1410 may be physical system memory, which may be volatile, non-volatile, or a combination of both. The term "memory" may also be used herein to refer to non-volatile mass storage devices such as physical storage media. If the computer system 1400 is distributed, then processing, memory, and / or storage capacity may also be distributed.

[0143] Storage device 1410 is shown to include executable instructions (i.e., code 1415). Executable instructions refer to instructions that can be executed by processor 1405 of computer system 1400 (or even possibly an ML model) to perform operations such as those disclosed in the various methods.

[0144] The disclosed embodiments may include or utilize a special-purpose or general-purpose computer, including computer hardware such as one or more processors (e.g., processor 1405) and system memory (e.g., storage device 1410), as discussed in more detail below. Embodiments also include physical media and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible by a general-purpose or special-purpose computer system. A computer-readable medium storing computer-executable instructions in data form is a “physical computer storage medium” or “hardware storage device.” A computer-readable medium carrying computer-executable instructions is a “transmission medium.” Thus, by way of example and not limitation, the present embodiments may include at least two distinctly different computer-readable media: computer storage media and transmission media.

[0145] Computer storage media (also known as “hardware storage devices”) are computer-readable hardware storage devices, such as RAM, ROM, EEPROM, CD-ROM, RAM-based solid-state drives (“SSDs”), flash memory, phase-change memory (“PCM”) or other types of memory, or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired units of program code in the form of computer-executable instructions, data or data structures and that can be accessed by a general-purpose or special-purpose computer.

[0146] Computer system 1400 can also be connected to external sensors (e.g., one or more remote cameras) or devices via network 1420 (via a wired or wireless connection). For example, computer system 1400 can communicate with any number of devices or cloud services to acquire or process data. In some cases, network 1420 itself can be a cloud network. Furthermore, computer system 1400 can also be connected to remote / separate computer systems via one or more wired or wireless networks 1420, which are configured to perform any of the processing described with respect to computer system 1400.

[0147] A “network” similar to network 1420 is defined as one or more data links and / or data switches that enable the transmission of electronic data between computer systems, modules, and / or other electronic devices. When information is transmitted or provided to a computer via a network (hardwired, wireless, or a combination of hardwired and wireless), the computer correctly treats the connection as a transmission medium. Computer system 1400 will include one or more communication channels for communicating with network 1420. The transmission medium includes a network that can be used to carry data or required program code units in the form of computer-executable instructions or data structures. Furthermore, these computer-executable instructions can be accessed by general-purpose or special-purpose computers. Combinations of the above should also be included within the scope of computer-readable media.

[0148] Upon arrival at various computer system components, program code units in the form of computer-executable instructions or data structures can be automatically transferred from the transmission medium to the computer storage medium (or vice versa). For example, computer-executable instructions or data structures received via a network or data link can be buffered in the RAM within a network interface module (e.g., a network interface card or "NIC") and then eventually transferred to the computer system RAM and / or the less volatile computer storage medium within the computer system. Therefore, it should be understood that computer storage media can be contained within computer system components that also (or even primarily) utilize the transmission medium.

[0149] Computer-executable (or computer-interpretable) instructions include, for example, instructions that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a function or a set of functions. Computer-executable instructions can be, for example, binary files, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. Rather, the described features and actions are disclosed as exemplary forms for implementing the claims.

[0150] Those skilled in the art will understand that these embodiments can be practiced in network computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics devices, network PCs, minicomputers, mainframes, mobile phones, PDAs, pagers, routers, switches, etc. These embodiments can also be practiced in distributed system environments, where local and remote computer systems, linked by a network (via hardwired data links, wireless data links, or a combination of hardwired and wireless data links), each perform tasks (e.g., cloud computing, cloud services, etc.). In a distributed system environment, program modules can reside in local and remote memory storage devices.

[0151] This invention may be embodied in other specific forms without departing from its spirit or characteristics. The described embodiments are to be considered illustrative rather than restrictive in all respects. Therefore, the scope of the invention is indicated by the appended claims rather than by the foregoing description. All modifications within the equivalent meaning and scope of the claims should be included within their scope.

Claims

1. A computer system configured to align and stabilize images generated by an integrated camera physically mounted to the computer system with images generated by a detached camera physically unmounted from the computer system, wherein, The computer system is a user-worn head-mounted device (HMD), wherein the separate camera is oriented such that the field of view (FOV) of the separate camera at least partially overlaps with the FOV of the integrated camera, and the computer system includes: One or more processors; and One or more computer-readable hardware storage devices storing instructions executable by the one or more processors to cause the computer system to perform at least the following operations: The integrated camera is used to generate the first image; The first image is used to determine the first posture of the computer system; Determine the first timestamp of the first image; Acquire the second image generated by the separate camera; Align the second image with the first image; An overlay image is generated by overlaying the second image onto the first image based on the alignment. Identify the pose difference between the current pose of the computer system at the current timestamp and the first pose determined using the first image at the first timestamp; The late reprojection LSR is applied to the overlay image to transform the pixels in the overlay image to take into account the pose difference identified between the current pose associated with the current timestamp and the first pose associated with the first timestamp; and After the LSR is applied to the overlay image, the overlay image is displayed.

2. The computer system of claim 1, wherein, The execution of the instructions also causes the computer system to determine that the integrated camera and the separate camera are operating in different time domains.

3. The computer system of claim 2, wherein, The integrated camera operates in conjunction with a 90 Hz refresh rate to display content, and the separate camera operates in conjunction with a 30 Hz refresh rate to display content.

4. The computer system of claim 1, wherein, Aligning the second image with the first image is performed by identifying the image correspondence between the second image and the first image.

5. The computer system of claim 1, wherein, Aligning the second image with the first image is performed through the following operations: Attempt to identify image correspondences between the second image and the first image; Determine that one or both of the second and first images lack sufficient texture to identify the corresponding images; The first inertial measurement unit (IMU) of the computer system is used to estimate the attitude of the computer system. The second IMU of the separated camera is used to estimate the IMU estimated attitude of the separated camera; as well as The second image is aligned with the first image by aligning the IMU estimated attitude of the computer system with the IMU estimated attitude of the separate camera.

6. The computer system of claim 1, wherein, Aligning the second image with the first image is performed without relying on the timestamp associated with the second image.

7. The computer system of claim 1, wherein, Aligning the second image with the first image is performed through the following operations: Identify the image correspondence between the second image and the first image; and Based on the identified image correspondence, the correspondence is adapted to the motion model, such that the second image is projected onto the first image.

8. The computer system of claim 7, wherein, The motion model is a rotational motion model.

9. The computer system of claim 7, wherein, The motion model is a similarity transformation model, which is configured to allow: (i) rotation of the first image or the second image, (ii) scaling of the first image or the second image, or (iii) homography transformation of the first image or the second image.

10. The computer system of claim 1, wherein, The integrated camera is selected from a camera group including a visible light camera, a low-light camera, or a thermal imaging camera, and the separate camera is also selected from the camera group.

11. A method for aligning and stabilizing images generated by an integrated camera physically mounted to a computer system with images generated by a detached camera physically unmounted from the computer system, wherein, The computer system is a user-worn head-mounted device (HMD), wherein the separate camera is oriented such that the field of view (FOV) of the separate camera at least partially overlaps with the FOV of the integrated camera, the method comprising: The integrated camera is used to generate the first image; The first image is used to determine the first posture of the computer system; Determine the first timestamp of the first image; Acquire the second image generated by the separate camera; Align the second image with the first image; An overlay image is generated by overlaying the second image onto the first image based on the alignment. Identify the pose difference between the current pose of the computer system at the current timestamp and the first pose determined using the first image at the first timestamp; The late reprojection LSR is applied to the overlay image to transform the pixels in the overlay image to take into account the pose difference identified between the current pose associated with the current timestamp and the first pose associated with the first timestamp; and After the LSR is applied to the overlay image, the overlay image is displayed.

12. The method of claim 11, wherein, The LSR is configured to correct for the 3-DOF variations included in the attitude differences, such that forward or backward projection is avoided during the application of the LSR.

13. The method of claim 11, wherein, The first image is one of a visible light image, a low-light image, or a thermal image, and the second image is a different image of the visible light image, the low-light image, or the thermal image.

14. The method of claim 11, wherein, The first timestamp of the first image is different from the second timestamp of the second image.

15. The method of claim 14, wherein, Both the first timestamp and the second timestamp are different from the current timestamp.

Images