Simultaneous pass-through and recording capability on a head-mounted device

Abstract

To provide an electronic device such as a head-mounted device.SOLUTION: A method of operating a head-mounted device may include outputting, with one or more displays, pass-through content at a pass-through frame rate, activating, while outputting the pass-through content, a recording mode to capture recorded content at a recording frame rate different from the pass-through frame rate, obtaining, with one or more image sensors, an image, and warping the image to generate a plurality of warped images corresponding to different points in time of the pass-through content. The method may further comprise obtaining a bracket image associated with the image, wherein the bracket image is not displayed. The image being displayed as part of the pass-through content may be enhanced using information of at least a portion of the bracket image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This application claims priority to U.S. Patent Application No. 19 / 182,462, filed April 17, 2025, and U.S. Provisional Patent Application No. 63 / 668,659, filed July 8, 2024, which are incorporated by reference in their entireties.

[0002] This application relates generally to electronic devices, and more particularly to electronic devices such as head-mounted devices. [Background technology]

[0003] An electronic device such as a head-mounted device can include hardware and software subsystems for performing gaze tracking, hand tracking, and head pose tracking of a user. The head-mounted device can also include a front-facing camera for acquiring a video feed of the physical environment facing the head-mounted device. The video feed can be displayed to the user as a pass-through feed.

[0004] The content displayed to the user may be recorded. It is against this background that the embodiments described herein arise, as it can be difficult to record high dynamic range content with a wide depth of field without affecting the pass-through feed. Summary of the Invention

[0005] One aspect of the present disclosure provides a method for operating a head-mounted device. The method includes outputting pass-through content at a pass-through frame rate using one or more displays; activating a recording mode to capture recorded content at a recording frame rate different from the pass-through frame rate while the one or more displays are outputting the pass-through content at the pass-through frame rate; acquiring images using one or more image sensors; and warping the images to generate multiple warped images corresponding to different times for the pass-through content. The warped images may be generated based on measured or predicted poses of the head-mounted device at the different times. The method may further include acquiring bracketed images using adjusted image settings. The bracketed images are not displayed as part of the pass-through content.

[0006] One aspect of the present disclosure provides a method of operating a head-mounted device, the method including: acquiring a first image using one or more cameras; warping the first image to generate a reprojected version of the first image; outputting the reprojected version of the first image using one or more displays; acquiring the bracket images without outputting the bracket images on the one or more displays while the reprojected version of the first image is displayed to a user; and generating recorded content based at least in part on the bracket images. The method may optionally further include warping the first bracket image, the second bracket image, and the third bracket image to a common time point to generate corresponding warped bracket images, and combining the warped bracket images to generate a fused image for the recorded content. The method may optionally further include warping the first bracket image, the second bracket image, and the third bracket image to a common time point to generate corresponding warped bracket images, and enhancing the first image using information from at least some of the warped bracket images.

[0007] One aspect of the present disclosure provides a method for operating a head-mounted device, the method including: acquiring, using one or more cameras, a first image and bracketed images associated with the first image; storing, using a recording pipeline, recorded content generated at least in part based on the bracketed images; warping at least one of the bracketed images to generate at least one warped bracketed image; enhancing the first image based on the at least one warped bracketed image to generate an enhanced image; and outputting, using one or more displays, a reprojected version of the enhanced image without outputting the bracketed images. Optionally, the method may further include warping at least one of the bracketed images to a time point at which the first image is acquired. Optionally, the method may further include warping at least one of the bracketed images and the first image to a common time point after the time at least one of the bracketed images is acquired. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 illustrates a top view of an exemplary head-mounted device, according to some embodiments.

[0009] [Figure 2] FIG. 1 is a block diagram of an exemplary head-mounted device, according to some embodiments.

[0010] [Figure 3] FIG. 1 illustrates an exemplary display and recording pipeline that may be included within a head-mounted device, according to some embodiments.

[0011] [Figure 4] FIG. 1 illustrates how image frames can be processed for display and recording pipelines, according to some embodiments.

[0012] [Figure 5] 1 is a flowchart of exemplary steps for processing images for display and recording pipelines, according to some embodiments.

[0013] [Figure 6] 1 is a flowchart of illustrative steps for performing image enhancement for a display pipeline, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0014] A top view of an exemplary head-mounted device (HMD) is shown in FIG. 1. As shown in FIG. 1, a head-mounted device such as electronic device 10 may have a head-mounted support structure such as housing 12. Housing 12 may include a portion (e.g., head-mounted support structure 12T) to allow device 10 to be worn on a user's head. Support structure 12T may be formed from fabric, polymer, metal, and / or other materials. Support structure 12T may form a strap or other head-mounted support structure to help support device 10 on the user's head. The main support structure of housing 12 (e.g., a head-mounted housing such as main housing portion 12M) may support electronic components such as display 14.

[0015] Main housing portion 12M may include a housing structure formed from metal, polymer, glass, ceramic, and / or other materials. For example, housing portion 12M may have a housing wall on the front surface F and housing walls on the top, bottom, left, and right sides formed from a rigid polymer or other rigid support structure, which rigid walls may optionally be covered with electrical components, fabric, leather, or other soft materials. Housing portion 12M may also have an internal support structure, such as a frame (chassis) and / or structure that provides structural support while performing multiple functions, such as controlling airflow and dissipating heat.

[0016] The walls of housing portion 12M may also enclose internal components 38 within an interior region 34 of device 10 and may separate interior region 34 from the environment (exterior region 36) surrounding device 10. Internal components 38 may include integrated circuits, actuators, batteries, sensors, and / or other circuitry and structure for device 10. Housing 12 may be configured to be worn on a user's head and may form glasses, spectacles, hats, masks, helmets, goggles, and / or other head-mounted devices. Configurations in which housing 12 forms goggles may be described herein by way of example.

[0017] The front surface F of the housing 12 may face outward from the user's head and face. The opposing back surface R of the housing 12 may face the user. A portion of the housing 12 on the back surface R (e.g., a portion of the main housing 12M) may form a cover, such as a cover 12C (sometimes referred to as a curtain). The presence of the cover 12C on the back surface R may help hide the internal housing structure, internal components 38, and other structures within the interior region 34 from the user's view.

[0018] Device 10 may have one or more cameras, such as camera 46 of FIG. 1 . Camera 46 mounted on the front surface F and facing outward (toward the front of device 10 and away from the user) may be referred to herein as a forward-facing, front-facing, outward-facing, or external-facing camera. Camera 46 may capture visual odometry information, image information that is processed to locate objects in the user's field of view (e.g., so that virtual content can be properly registered with real-world objects), image content that is displayed in real time to the user of device 10, and / or other suitable image data. For example, a forward-facing (outward-facing) camera may enable device 10 to monitor the motion of device 10 relative to the environment surrounding it (e.g., the camera may be used in forming part of a visual odometry system or a visual inertial odometry system). An outward-facing camera may also be used to capture images of the environment that are displayed to the user of device 10. If desired, images from multiple outward-facing cameras may be merged together for the user, and / or outward-facing camera content may be merged with computer-generated content.

[0019] Device 10 may have any suitable number of cameras 46. For example, device 10 may have K cameras, where K is at least 1, at least 2, at least 4, at least 6, at least 8, at least 10, at least 12, less than 20, less than 14, less than 12, less than 10, between 4 and 10, or other suitable values. Camera 46 may be sensitive to infrared wavelengths (e.g., camera 46 may be an infrared camera), may be sensitive to visible wavelengths (e.g., camera 46 may be a visible light camera), and / or may be sensitive to other wavelengths. Optionally, camera 46 may be sensitive to both visible and infrared wavelengths.

[0020] The device 10 may have left and right optical modules 40. The optical modules 40 support electrical and optical components, such as light-emitting components and lenses, and may therefore be referred to as optical assemblies, optical systems, optical component support structures, lens and display support structures, electrical component support structures, or housing structures. Each optical module may include a respective display 14, a lens 30, and a support structure, such as a support structure 32. The support structure 32, which may also be referred to as a lens support structure, optical component support structure, optical module support structure, optical module portion, or lens barrel, may include a hollow cylindrical structure with an open end or other support structure for housing the display 14 and lens 30. The support structure 32 may include, for example, a left lens barrel supporting the left display 14 and left lens 30, and a right lens barrel supporting the right display 14 and right lens 30.

[0021] Display 14 may include an array of pixels or other display devices for generating an image. Display 14 may include, for example, organic light emitting diode pixels formed on a substrate with thin film circuitry and / or formed on a semiconductor substrate, pixels formed from crystalline semiconductor dies, liquid crystal display pixels, scanning display devices, waveguides, and / or other display components for generating an image.

[0022] The lens 30 may include one or more lens elements for providing image light from the display 14 to the respective eyebox 13. The lens may be implemented using refractive glass lens elements, using mirror lens structures (catadioptric lenses), using Fresnel lenses, using holographic lenses, and / or using other lens systems.

[0023] When the user's eyes are positioned in the eyebox 13, the display (display panel) 14 cooperates to form the display of the device 10 (e.g., the images provided by each of the left and right optical modules 40 are viewable by the user's eyes within the eyebox 13 such that a stereoscopic image is created for the user). While the display is viewed by the user, the left image from the left optical module is fused with the right image from the right optical module.

[0024] It may be desirable to monitor a user's eyes while they are located in the eyebox 13. For example, it may be desirable to capture an image of the user's iris (or other part of the user's eye) using a camera for user authentication. It may also be desirable to monitor the direction of the user's gaze. Gaze tracking information may be used as a form of user input and / or to determine locations within the image where the resolution of the image content should be locally enhanced in a foveated imaging system. To ensure that device 10 can capture sufficient eye images while the user's eyes are located in the eyebox 13, each optical module 40 may be provided with a camera, such as camera 42, and one or more light sources, such as light-emitting diodes 44, or light-emitting devices such as lasers, lamps, etc. The camera 42 and the light-emitting diodes 44 may operate at any suitable wavelength (visible, infrared, and / or ultraviolet). As an example, the diodes 44 may emit infrared light that is invisible (or nearly invisible) to the user. This allows eye monitoring operations to be performed continuously without interfering with the user's ability to view images on the display 14.

[0025] A block or schematic diagram of an exemplary electronic device, such as a head-mounted device or other wearable device, is shown in Figure 2. Device 10 of Figure 2 may be operated as a standalone device, and / or resources of device 10 may be used to communicate with external electronic devices. As an example, communications circuitry 22 within device 10 may be used to transmit user input information, sensor information, and / or other information to external electronic devices (e.g., via a wireless or wired connection). Each of these external devices may include components of the type illustrated by device 10 of Figure 2.

[0026] As shown in FIG. 2 , a head-mounted device such as device 10 can include control circuitry 20. Control circuitry 20 can include storage and processing circuitry to support the operation of device 10. The storage and processing circuitry can include storage devices such as non-volatile memory (e.g., flash memory or other electrically programmable read-only memory configured to form a solid-state drive), volatile memory (e.g., static or dynamic random access memory), etc. One or more processors within control circuitry 20 can be used to collect input from sensors and other input devices and can also be used to control output devices. The processing circuitry can be based on one or more processors, such as a microprocessor, a microcontroller, a digital signal processor, a baseband processor and other wireless communication circuitry, a power management unit, an audio chip, an application-specific integrated circuit, etc. In operation, control circuitry 20 can use display(s) 14 and other output devices in providing visual and other output to a user. Control circuitry 20 can be configured to perform operations in device 10 using hardware (e.g., dedicated hardware or circuitry), firmware, and / or software. Software code for performing operations within device 10 may be stored on storage circuitry (which may, for example, include a non-transitory (tangible) computer-readable storage medium that stores the software code). The software code is sometimes referred to as program instructions, software, data, instructions, or code. The stored software code may be executed by processing circuitry within device 20.

[0027] To support communications between device 10 and external devices, control circuitry 20 can communicate using communications circuitry 22. Communications circuitry 22 can include an antenna, radio frequency transceiver circuitry, other wireless communications circuitry, and / or wired communications circuitry. Communications circuitry 22, sometimes referred to as control circuitry and / or control and communications circuitry, can support two-way wireless communications between device 10 and external devices (e.g., companion devices such as computers, cellular telephones, or other electronic devices, pointing devices, or accessories such as controllers, computer styluses, or other input devices, speakers, or other output devices, etc.) via a wireless link.

[0028] For example, circuitry 22 may include radio frequency transceiver circuitry, such as wireless local area network (WLAN) transceiver circuitry configured to support communications over a wireless local area network (WLAN) link, near-field communications transceiver circuitry configured to support communications over a short-range communications link, cellular telephone transceiver circuitry configured to support communications over a cellular telephone link, or transceiver circuitry configured to support communications over any other suitable wired or wireless communication link. Wireless communication may be supported, for example, via a Bluetooth® link, a Wi-Fi® link, a wireless link operating at frequencies between 10 GHz and 400 GHz, a 60 GHz link, or other millimeter wave link, a cellular link, or other wireless communication link. Device 10 may also include power circuitry for transmitting and / or receiving wired and / or wireless power, if desired, and may include a battery or other energy storage device. For example, device 10 may include a coil and rectifier for receiving wireless power provided in circuitry within device 10.

[0029] Device 10 may include input / output devices, such as device 24. Input / output device 24 may be used to collect user input, collect information about the environment surrounding the user, and / or provide output to the user. Input / output device 24 may include one or more displays, such as display(s) 14. Display(s) 14 may include one or more display devices, such as an organic light emitting diode display panel (a panel with organic light emitting diode pixels formed on a polymer or silicon substrate containing pixel control circuitry), a liquid crystal display panel, a microelectromechanical systems display (e.g., a two-dimensional mirror array or scanning mirror display device), a display panel with a pixel array formed from crystalline semiconductor light emitting diode dies (sometimes referred to as microLEDs), a display including a waveguide, and / or other display devices.

[0030] The sensors 16 in the input / output device 24 may include force sensors (e.g., strain gauges, capacitive force sensors, resistive force sensors, etc.), audio sensors such as microphones, touch and / or proximity sensors such as capacitive sensors such as touch sensors forming buttons, trackpads, or other input devices, and other sensors. Optionally, the sensors 16 may include optical sensors such as optical sensors that emit and detect light, ultrasonic sensors, optical touch sensors, optical proximity sensors, and / or other touch and / or proximity sensors, monochrome and color ambient light sensors, image sensors (e.g., cameras), fingerprint sensors, iris scanning sensors, retinal scanning sensors, and other biometric sensors, temperature sensors, sensors for measuring three-dimensional contactless gestures (“air gestures”), pressure sensors, sensors for detecting information about the position, orientation, and / or movement of the device 10 and / or the posture of the user's head (e.g., accelerometers, magnetic sensors such as compass sensors, geometries, and the like). The sensors 16 may include gyroscopes, and / or inertial measurement units containing some or all of these sensors), health sensors such as blood oxygen level sensors, heart rate sensors, blood flow sensors, and / or other health sensors, radio frequency sensors, optical sensors such as three-dimensional camera systems (e.g., structured light sensors and / or depth sensors based on stereo imaging devices that capture three-dimensional images) and / or self-mixing sensors and light detection and ranging (lidar) sensors that collect time-of-flight measurements (e.g., time-of-flight cameras), humidity sensors, moisture sensors, gaze tracking sensors, electromyographic sensors that sense muscle activation, face sensors, and / or other sensors. In some configurations, device 10 may use sensors 16 and / or other input / output devices to collect user input. For example, a button can be used to collect button press input, a touch sensor overlying the display can be used to collect user touch screen input, a touchpad can be used to collect touch input, a microphone can be used to collect audio input (e.g., voice commands), an accelerometer can be used to monitor when a finger contacts the input surface and thus can be used to collect finger press input, etc.

[0031] If desired, electronic device 10 may include additional components (see, for example, other devices 18 in input / output devices 24). The additional components may include tactile output devices, actuators for moving a movable housing structure, audio output devices such as speakers, light emitting diodes for status indicators, light sources such as light emitting diodes for illuminating portions of the housing and / or display structure, other optical output devices, and / or other circuitry for collecting input and / or providing output. Device 10 may also include a battery or other energy storage device, connector ports for supporting wired communication with auxiliary equipment and receiving wired power, and other circuitry.

[0032] The display(s) 14 may be used to present various content to the user's eyes. The left and right displays 14 used to present a fused stereoscopic image to the user's eyes when viewed through the eyebox 13 may sometimes be collectively referred to as the displays 14. As an example, real-world content may be presented by the displays 14. "Real-world" content may refer to images of the physical environment captured by one or more front-facing cameras (e.g., see camera 46 in FIG. 1) and passed as a live feed to the user. Thus, the real-world content being captured by the front-facing cameras may be referred to as a camera pass-through feed, a (live) video pass-through feed, or a pass-through video feed (content).

[0033] A physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic devices. In contrast, an extended reality (XR) environment refers to a wholly or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, etc. In an XR system, a subset of a person's body movements or a representation thereof is tracked, and in response, one or more properties of one or more virtual objects simulated within the XR environment are adjusted to behave according to at least one law of physics. In some embodiments, the display 14 can be used to output extended reality (XR) content, which can include virtual reality content, augmented reality content, and / or mixed reality content. The content output on the display 14 may be referred to as a display frame.

[0034] It may be desirable to record real-world pass-through content and / or XR content displayed to a user of a head-mounted device. Pass-through content including only real-world content generated based on images captured by one or more outward-facing cameras (e.g., left and right front-facing cameras) may be recorded via operations sometimes referred to as “spatial capture” or “spatial recording” operations. XR content including computer-generated content blended with real-world pass-through content may be recorded via operations sometimes referred to as “mixed reality capture,” “extended reality capture,” “mixed reality recording,” “extended reality recording,” or “view capture” operations. Such recording operations can generate or create corresponding recordings having the recorded content. Spatial capture can generate recordings having only real-world content, while XR capture can generate recordings having both real-world content and computer-generated (virtual) content. Content presented on the display 14 can be output as display frames. A recording operation can generate recordings of the display frames.

[0035] Consider an example in which a first person is using a head-mounted device. The first person can initiate spatial or XR capture to generate a corresponding recording of the display frames. The recording may be shared (in real time or after the recording is completed) with a second person (e.g., via an additional electronic device that presents the recording to the second person), or may be subsequently played back by the first person. Thus, recording and optionally sharing in-headset experiences in this manner enables a more social experience, enables additional functionality, and so on. A user can view a playback of the recording on a head-mounted device that is the same or similar to the head-mounted device that originally generated the recording. In other situations, a user can view a playback of the recording on a different type of device, such as a mobile phone, laptop computer, tablet computer, etc. The device that presents the playback of the recording can have a stereoscopic display or a non-stereoscopic display.

[0036] Figure 3 is a diagram illustrating various hardware and software subsystems that may be included within device 10. As shown in Figure 3, device 10 may include an imaging subsystem including one or more image sensors 50, an image signal processing subsystem such as an image signal processor (ISP) 52, a user tracking subsystem including one or more position and motion sensors 54, one or more gaze detection sensors 80, an image warping subsystem such as an image warping block 60, an image fusion subsystem such as an image fusion block 66, a display pipeline including one or more displays 14, and a recording subsystem such as a recording pipeline 68.

[0037] Image sensor 50 may include one or more front-facing cameras and / or other cameras used to collect information about the external real-world environment surrounding device 10. Camera 50 may represent one or more of front-facing cameras 46 of FIG. 1. The video feed output from camera 50 may be referred to as a raw video feed or a live video pass-through feed, a live pass-through video stream, or pass-through content. Accordingly, front-facing camera 50 may be referred to herein as a pass-through camera.

[0038] Such pass-through feed output from camera 50 can be processed by an image signal processor (ISP) 52 configured to perform image signal processing functions that depend solely on the input of the live camera feed itself. For example, ISP block 52 can be configured to perform auto-exposure to control exposure settings for the pass-through feed, tone mapping, auto-focus, color correction, gamma correction, shading correction, noise reduction, black level adjustment, demosaicing, image sharpening, high dynamic range (HDR) correction, color space conversion, and / or other image signal processing functions to output a corresponding processed pass-through feed (e.g., a series of processed video frames). Processor 52 can output one or more image settings to adjust or otherwise control image sensor 50.

[0039] Gaze detection sensor 80, sometimes referred to as a gaze tracker, may be configured to collect gaze or point of gaze information. The gaze tracker may monitor the user's eyes using one or more inward-facing camera(s) (e.g., camera 42 in FIG. 1 ) and / or other gaze tracking components (e.g., an eye-facing component that emits a light beam so that reflection of the beam from the user's eye can be detected and / or other light source such as light source 44). One or more gaze tracking sensor(s) 80 may face the user's eyes and track the user's gaze. The gaze detection sensor 80 may determine the location of the user's eyes (e.g., the center of the user's pupils), the direction the user's eyes are pointing (the user's direction of gaze), the user's pupil size (e.g., to obtain light modulation and / or other optical parameters, and / or the amount of gradualness with which one or more of these parameters are spatially adjusted, and / or the area over which one or more of these optical parameters are adjusted based on pupil size), the current focus of the user's eye lenses (e.g., whether the user is focusing on near or far vision, which may be used to assess whether the user is daydreaming or thinking strategically or tactically), and / or other gaze information, and / or other gaze information. The gaze tracking device's camera is sometimes referred to as an inward-facing camera, gaze detection camera, eye tracking camera, gaze tracking camera, or eye monitoring camera. If desired, other types of image sensors (e.g., infrared and / or visible light emitting diodes and photodetectors, etc.) may also be used in monitoring the user's gaze.

[0040] According to some embodiments, gaze information output from gaze detection sensor(s) 80 may be provided to image sensor 50 to determine focus when acquiring a new image (e.g., for autofocus purposes). In some embodiments, image signal processing functions performed by ISP 52 may optionally be based on gaze tracking information from gaze detection sensor(s) 80. For example, ISP 52 may adjust the pass-through feed based on the gaze tracking information (e.g., for a foveated display) or may adjust the pass-through feed to better match the virtual content.

[0041] The position and motion sensors 54 may include accelerometers, magnetic sensors such as compass sensors, gyroscopes, and / or an inertial measurement unit (IMU) that includes some or all of these sensors. The position and motion sensors 54 may optionally include one or more cameras. The position and motion sensors 54 may track the user's head pose. The yaw, roll, and pitch of the user's head, representing three degrees of freedom (DoF), may collectively define the user's "head orientation." The user's head orientation, along with the user's position, representing three additional degrees of freedom (e.g., X, Y, and Z in three-dimensional space), may be collectively defined herein as the user's "head pose." The user's head pose therefore represents six degrees of freedom. These position and motion sensors may assume that the head-mounted device 10 is worn on the user's head. Thus, references herein to head pose, head movement, yaw of the user's head (e.g., rotation about a vertical axis), pitch of the user's head (e.g., rotation about a side-to-side axis), roll of the user's head (e.g., rotation about a front-to-back axis), etc. may be considered interchangeable with references to device pose, device movement, device yaw, device pitch, device roll, etc. In particular embodiments, the position and movement sensor 54 may also include a six degree-of-freedom (DoF) tracking sensor, which may be used to monitor both rotational movement, such as roll, pitch, and yaw, and positional / translational movement in a 3D environment.

[0042] Thus, the position and motion sensor 54 may be used to obtain the current head pose (sometimes simply referred to as "pose") of the device 10. The device 10 may optionally include a (head) pose prediction subsystem, such as a pose predictor 56 configured to predict or estimate future (head) poses of the device 10. The pose predictor 56 may predict or estimate the pose of the device 10 based on past or recent pose data / information. The pose predictor 56 may estimate the pose of the device 10 at some point in the future. Thus, the pose predictor 56 may be used to output one or more predicted poses at various points in time in the future. The pose predictor 56 is optional and may be omitted.

[0043] Although not explicitly shown, device 10 may include other types of tracking sensors, such as hand tracking sensors for monitoring the user's hand movements / gestures and / or other sensors for tracking other part(s) of the user's body part(s).

[0044] The warp subsystem, sometimes referred to herein as warp (warping) block 60, may be a software or hardware component configured to warp pass-through content and / or any virtual content based on one or more warp meshes. For example, the warp block or warper 60 may be configured to perform or apply geometric transformations, scaling, rotation, translation, distortion, deformation, warping, morphing, rippling, and / or other transformations or visual effects to the underlying image to manipulate the appearance of the captured image. Generally, warping may include a two-dimensional (2D) transformation or mapping that adjusts pixel locations within the image (e.g., changes the spatial arrangement of the image's pixels). The warping block 60 may also be configured to perform image reprojection based on pose information (e.g., based on pose data output from the position and motion sensor 54 or based on predicted pose data output from the pose predictor 56). For example, the warper 60 may perform image reprojection by mapping three-dimensional (3D) points onto a 2D plane to help align images captured from different camera viewpoints. The warped image may then be presented as a live video (pass-through) feed to a user via one or more displays 14.

[0045] To provide device 10 with recording capabilities, device 10 may include a separate recording subsystem, such as a recording pipeline 68. As shown in FIG. 3 , recording pipeline 68 may include a recorder processing block 72 and a recorder memory 74. To provide flexibility in subsequent editing and / or playback of the recording, recording pipeline 68 may record a wide variety of information associated with the pass-through or augmented reality experience. In general, any parameters, metadata, raw content, and other information acquired by one or more components within device 10 may be recorded by recording pipeline 68. In addition to being provided to image signal processor 52, the raw pass-through feed, the processed pass-through feed, and / or image sensor metadata from image sensor 50 may be provided to and recorded by recording pipeline 68. In the example of FIG. 3 , all frames being output from ISP block 52 may be temporarily stored in cache 64, sometimes referred to as an image buffer. One or more cached images may be warped by block 60 to generate a reprojected image for the display pipeline. The cached images (or image frames) may optionally be processed by an image fusion block 66 to selectively combine or fuse two or more image frames. The fusion block 66 may also provide image warping functionality (see, e.g., warping component 67).

[0046] In some embodiments, any image signal processing (ISP) parameters used by ISP 52 (e.g., color adjustment parameters, brightness adjustment parameters, distortion parameters, and / or any other parameters used in adjusting the pass-through feed) may be provided to and recorded by the recording pipeline 68. In some embodiments, virtual content output by the graphics rendering pipeline may be provided to and recorded by the recording pipeline 68 (e.g., by recording the virtual content as a single layer or as multiple layers). If desired, parameters such as color adjustment parameters, brightness adjustment parameters, distortion parameters, and / or other parameters used by the virtual content compositor to generate the virtual content may also be provided to and recorded by the recording pipeline 68. In some embodiments, head tracking information, gaze tracking information, and / or hand tracking information may also be provided to and recorded by the recording pipeline 68. In some embodiments, foveation parameters used in performing dynamic foveation may also be provided to and recorded by the recording pipeline 68. In some embodiments, compositing metadata associated with compositing the pass-through feed with the virtual content may be provided to and recorded by the recording pipeline 68. The compositing metadata used and output by the media merge compositor may include information about how the virtual content and the pass-through feed are blended (e.g., using one or more alpha values), information about video matting operations, etc. If desired, audio data obtained from one or more speakers in device 10 may be provided to and recorded by the recording pipeline 68.

[0047] Information received by recording pipeline 68 may be stored in memory 74. Before or after recording the information, recording processor 72 may optionally perform additional operations such as selecting a subset of the received frames for recording (e.g., selecting alternating frames to be recorded, selecting one of every three frames to be recorded, selecting one of every four frames to be recorded, selecting one of every five to ten frames for recording, etc.), limiting the rendered frames to a smaller field of view (e.g., limiting the X dimension of the rendered content, limiting the Y dimension of the rendered content, or otherwise constraining the size or extent of the frames to be recorded), not distorting the rendered content because the content being recorded may not be viewed through a lens during later playback, etc.

[0048] It may be desirable to use the recording pipeline 68 to capture high dynamic range (HDR) video. To achieve a high dynamic range, multiple successive images need to be acquired at different exposure levels to capture a wider range of brightness and detail. Conventionally, a single HDR image can be generated using a technique sometimes called exposure bracketing. Exposure bracketing may involve acquiring an image at a nominal exposure setting, acquiring one or more images using a lower exposure setting (e.g., to acquire an underexposed image to capture more detail in the highlights / brighter areas), and acquiring one or more images using a higher exposure setting (e.g., to acquire an overexposed image to capture more detail in the shadows / darker areas). Operating the pass-through camera while performing exposure bracketing can, if care is not taken, affect a user's overall experience viewing the displayed pass-through content. Furthermore, images acquired by the pass-through camera can be focused based on the user's gaze. For example, if a user wearing the device 10 is currently looking at a relatively close object, the background behind the object will be out of focus and blurred. Therefore, in addition to capturing a high dynamic range, it may also be desirable to record content with a depth of field that is independent of the user's gaze (e.g., so that most of the recorded video is in focus and therefore a user who later views the recorded video can clearly see any part of the video).

[0049] According to some embodiments, device 10 may be configured to generate pass-through content at a first (nominal) pass-through frame rate and, when a recording function / mode is enabled (e.g., when activating spatial capture or XR capture), record the content at a second recording frame rate using recording pipeline 68 without changing the nominal pass-through frame rate. For example, the pass-through frame rate for an in-head viewing experience may be equal to 90 fps (frames per second), 96 fps, 100 fps, 120 fps, 70 fps, 75 fps, or other pass-through frame rate. The pass-through frame rate is sometimes referred to as the camera frame rate. The pass-through frame rate may be equal to the system frame rate of device 10.

[0050] The term "system frame rate" can refer to the camera frame rate (e.g., the rate at which exposures are being performed by camera 50) and / or the display frame rate (e.g., the rate at which video frames are being output on display 14). Device 10 may have a unified system frame rate in which the camera frame rate is set equal to (or synchronized with) the display frame rate. This is exemplary. If desired, device 10 may optionally be operated using an asynchronous system frame rate in which the camera frame rate is not equal to the display frame rate. In general, device 10 can dynamically adjust to operate between three or more different system frame rates, four or more system frame rates, five to ten system frame rates, or more than ten different system frame rates.

[0051] The recording pipeline 68 can record content at a recording frame rate that differs from the pass-through (system) frame rate. A "recording frame rate" refers to the frame rate used by the recording pipeline 68 to generate the corresponding recording, and may be defined as such herein. In situations where the recording frame rate cannot operate at the pass-through / system frame rate due to system limitations (e.g., memory bandwidth limitations, encoder limitations, power and thermal limitations, etc.), the recording frame rate of the recording pipeline 68 can be or should be set equal to a percentage of the camera frame rate. For example, if the camera or system frame rate is set equal to 90 fps, the recording frame rate can be set equal to 30 fps (e.g., by capturing one out of every three frames that are displayed). As another example, if the camera or system frame rate is set equal to 100 fps, the recording frame rate can be set equal to 50 fps (e.g., by capturing one out of every two frames that are displayed). In other words, the camera or pass-through frame rate can be a multiple of the recording frame rate. If system limitations are removed, the recording frame rate may optionally be set equal to the system frame rate. The terms "fps" and "Hz" may be used interchangeably herein when referring to the frame rate of device 10.

[0052] Device 10 may be configured to acquire an initial image frame. After the initial image frame, several successive image frames may be acquired with different camera settings (e.g., using different exposure levels to capture a high dynamic range, using different focal lengths to capture an extended depth of field, using different color temperature settings to capture a wider tonal range, etc.). The successive image frames acquired using different camera settings should not be displayed to the user and, if no other action is taken, would effectively reduce the pass-through frame rate. Rather, the initial image frame and / or various versions of the initial image frame that have been reprojected based on the user's head pose while the successive frames were captured may instead be displayed to the user. Displaying the reprojected images may help mask the reduced pass-through frame rate. Reprojecting images for display in this manner may be effective because the user is expected to remain relatively stable while recording video. Thus, operating device 10 in this manner may be technically advantageous and beneficial for recording high dynamic range video content with extended depth of field (e.g., depth of field that is independent of the user's gaze) without degrading the in-headset pass-through experience.

[0053] Details of such image reprojection and bracketing operations may be further described in connection with FIG. 4 and in connection with the flowchart of FIG. 5. During the operation of block 200, device 10 may be configured to acquire a first (initial) image at a gaze distance. For example, one or more image sensors 50 may be configured to acquire the first image using a nominal exposure time (duration) T_isp determined by ISP block 52 to generate a nominal exposure level, an (auto) focal length determined based on the user's gaze D_gaze, and a color temperature setting Temp_isp determined by ISP block 52 to generate a target white balance. If the user is currently viewing an object located at a particular focal length from camera 50, the first image may be captured using that particular focal length. The first frame may be acquired at time t0 (e.g., see the frame labeled "t0" in FIG. 4). The first (initial) image may be temporarily stored or buffered in cache 64 (see FIG. 3).

[0054] During the operation of block 202, the device 10 may be configured to acquire a first bracketed image using first image (camera) settings. In the example of FIG. 4 , the one or more image sensors 50 may be configured to acquire a first bracketed image at time t0a (see frame labeled “t0a”) using first image settings such as a first adjusted exposure time (duration) T1, a first adjusted focal length D1, a first adjusted color temperature Temp1, and / or other adjusted image settings. The first adjusted exposure time T1 should be different from the nominal exposure time T_isp previously used to acquire the first (initial) image frame. The first adjusted focal length D1 should be different from the gaze-based focal length D_gaze. The first adjusted color temperature Temp1 should be different from the automatic Temp_isp previously used to acquire the first image frame. If desired, other image-related or camera settings can be adjusted when acquiring the first bracketed image at time t0a. The first bracketed image acquired using the first (adjusted) image settings is not displayed to the user but is subsequently processed for archival purposes. The first bracketed image may be temporarily stored or buffered in cache 64.

[0055] During the operation of block 204, device 10 may be configured to acquire a second bracketed image using second image (camera) settings. In the example of FIG. 4 , one or more image sensors 50 may be configured to acquire a second bracketed image at time t0b (see frame labeled “t0b”) using second image settings, such as a second adjusted exposure time T2, a second adjusted focal length D2, a second adjusted color temperature Temp2, and / or other adjusted image settings. The second adjusted exposure time T2 should be different from T_isp and T1. The second adjusted focal length D2 should be different from D_gaze and D1. The second adjusted color temperature Temp2 should be different from Temp_isp and Temp1. If desired, other image-related or camera settings can be adjusted when acquiring the second bracketed image at time t0b. The second bracketed image acquired using the second (adjusted) image settings is not displayed to the user but is subsequently processed for record purposes. The second bracketed image may be temporarily stored or buffered in cache 64 .

[0056] During the operation of block 206, device 10 may be configured to acquire a third bracketed image using third image (camera) settings. In the example of FIG. 4 , one or more image sensors 50 may be configured to acquire a third bracketed image at time t0c (see frame labeled “t0c”) using third image settings, such as a third adjusted exposure time T3, a third adjusted focal length D3, a third adjusted color temperature Temp3, and / or other adjusted image settings. The third adjusted exposure time T3 should be different from T_isp, T1, and T2. The third adjusted focal length D3 should have different D_gaze, D1, and D2. The third adjusted color temperature Temp3 should be different from Temp_isp, Temp1, and Temp2. ​​If desired, other image-related or camera settings can be adjusted when acquiring the third bracketed image at time t0c. The third bracketed image, acquired using the third (adjusted) image settings, is not displayed to the user but is subsequently processed for archival purposes and may be temporarily stored or buffered in cache 64.

[0057] Capturing multiple (bracketed) images using different exposure settings, a technique sometimes referred to as exposure bracketing, can be technically advantageous and beneficial for providing high dynamic range content for a recording pipeline. Capturing multiple (bracketed) images using different focus settings, a technique sometimes referred to as focus bracketing, can be technically advantageous and beneficial for providing extended (wider) depth of field for a recording pipeline. Capturing multiple (bracketed) images using different color temperature settings, a technique sometimes referred to as white balance bracketing, can be technically advantageous and beneficial for providing more accurate color reproduction for a recording pipeline. Additionally or alternatively, other types of image bracketing techniques can be employed, including, but not limited to, ISO bracketing (e.g., acquiring multiple images at different sensor sensitivity or ISO settings) and aperture bracketing (e.g., acquiring multiple images at different lens aperture or f-stop settings), to name a few. The examples shown and described in connection with FIGS. 4 and 5, in which three bracketed images are acquired for recording purposes, are illustrative. In general, device 10 may be configured to acquire three or more bracketed image frames, five or more bracketed image frames, seven or more bracketed image frames, or more than ten bracketed image frames to help improve the overall quality of the recorded content. All of the bracket-related operations described above may be orchestrated using a bracket control subsystem, such as bracket controller 62, as shown in the example of FIG. 3. The terms image, frame, and image frame may be used interchangeably herein.

[0058] During the operation of block 208, bracketed images previously stored in cache 64 may be communicated to recording pipeline 68 via image fusion block 66. Image fusion block 66 may include a warping subsystem 67. Warper 67 may be implemented as a hardware or software component. Because the various bracketed image frames are acquired at different times (e.g., a first bracketed image is acquired at time t, a second bracketed image is acquired at time t, and a third bracketed image is acquired at time t), if the user's head moves even slightly during recording, the three bracketed frames may be taken at slightly different head poses. Therefore, to properly combine the multiple bracketed images, warper 67 may warp the first, second, third, and / or other recently acquired bracketed images to the same new point at time t, where t is a time after time t, to the same head pose at time t. Warping from t0a to t0c+ may be based on the pose at time t0a and the pose at time t0c+. Warping from t0b to t0c+ may be based on the pose at time t0b and the pose at time t0c+. Warping from t0c to t0c+ may be based on the pose at time t0c and the pose at time t0c+. After the bracketed images have all been warped to time t0c+, the warped versions of the bracketed images, sometimes referred to herein as “warped bracketed images,” may then be combined or fused together to generate a high dynamic range (HDR) image with extended depth of field, accurate white balance, and / or other image enhancements for storage in the recording pipeline 68. In general, block 66 can selectively warp N recently acquired bracketed frames to the same time and space, fuse the warped images, and then output the fused image to the recording pipeline 68 for storage.

[0059] As described above, the bracketed images acquired between blocks 202-206 are taken with an adjusted image sensor setting and should not be displayed to the user. The bracketed image frames may be acquired at the nominal system or camera frame rate. To maintain the pass-through video experience at the pass-through frame rate, at least some video frames must be sent to the display pipeline to be displayed to the user in parallel with the acquisition of the bracketed image frames. In other words, although the operations of blocks 210, 212, and 214 are shown as occurring after the operations of blocks 202, 204, and 206, the operations of blocks 210, 212, and 214 may occur in parallel or simultaneously with the operations of blocks 202, 204, and 206.

[0060] During the operation of block 210, device 10 may warp the first image originally acquired during block 200 to generate a first warped image corresponding to time t0+1. For example, warping block 60 may warp or reproject the first image based on the difference between the detected head pose at time t0 and the detected head pose at time t0+1 to generate a first warped (reprojected) image frame labeled "t0+1," as indicated by arrow 100. This warping or reprojection may optionally be based on measured pose output from sensor 54 or predicted pose output from pose predictor 56 (see FIG. 3). Time t0+1 may be equal to or different from time t0a, which corresponds to the time at which the first bracketed frame is being acquired for the recording pipeline.

[0061] During the operation of block 212, device 10 may warp the first image originally acquired during block 200 to generate a second warped image corresponding to time t0+2, occurring some time after t0+1. For example, warping block 60 may warp or reproject the first image based on the difference between the detected head pose at time t0 and the detected head pose at time t0+2 to generate a second warped (reprojected) image frame labeled "t0+2," as indicated by arrow 102. This warping or reprojection may optionally be based on measured pose output from sensor 54 or predicted pose output from pose predictor 56. Time t0+2 may be equal to or different from time t0b, which corresponds to the time at which a second bracketed frame is being acquired for the recording pipeline.

[0062] During the operation of block 214, device 10 may warp the first image originally acquired during block 200 to generate a third warped image corresponding to time t0+3, occurring some time after t0+2. The time delta between t0+1 and t0+2 should be equal to the time delta between t0+2 and t0+3 to maintain consistent cadence of the displayed content. Warping block 60 may warp or reproject the first image based on the difference between the detected head pose at time t0 and the detected head pose at time t0+3 to generate a third warped (reprojected) image frame labeled "t0+3," as indicated by arrow 104. This warping or reprojection may optionally be based on measured pose output from sensor 54 or predicted pose output from pose predictor 56. Time t0+3 may be equal to or different from time t0, which corresponds to the time at which the third bracketed frame is being acquired for the recording pipeline.

[0063] During the operation of block 216, the warped images obtained from blocks 210, 212, and 214 may be output by a display pipeline (e.g., displayed to a user operating device 10). Displaying reprojected images (e.g., images reprojected based on head pose over successive frames) while preventing bracketed images from being shown to the user may be technically advantageous and beneficial for using reprojection to mask an otherwise reduced pass-through frame rate. In general, operations associated with the recording pipeline (e.g., blocks 202-208) and operations associated with the display pipeline (e.g., blocks 210-216) may occur in parallel.

[0064] The operations of Figure 5 may be repeated for the next (non-bracketed) image frame, as indicated by the subsequent image frame labeled "t1" in Figure 4, to process one or more additional image frames for the recording and display pipelines. The examples of Figures 4-5 are illustrative, in which three images at times t0+1, t0+2, and t0+3 are generated using reprojection for display purposes. In general, device 10 may be configured to generate one or more reprojected frames, two or more reprojected frames, three or more reprojected frames, four to ten reprojected frames, or more than ten reprojected frames for the display pipeline, depending on the pass-through frame rate and the recording frame rate.

[0065] The operations described in connection with FIG. 5 , in which the image frames being output by the display pipeline are separate from the bracketed images, are exemplary. FIG. 6 is a flowchart of exemplary steps for performing image enhancement of the display pipeline based on the bracketed images, according to some embodiments. During the operations of block 300, device 10 may be configured to acquire a first (initial) image at a gaze distance. For example, one or more image sensors 50 may be configured to acquire the first image using a nominal exposure time (duration) T_isp determined by ISP block 52 to generate a nominal exposure level, an (auto) focal length determined based on the user's gaze D_gaze, and a color temperature setting Temp_isp determined by ISP block 52 to generate a target white balance. The first frame may be acquired at time t0. The first (initial) image may be temporarily stored or buffered in cache 64 (see FIG. 3 ).

[0066] During the operations of block 302, device 10 may be configured to acquire one or more bracketed images using different image settings. For example, one or more image sensors 50 may be configured to capture multiple bracketed images using different exposure settings (e.g., varying exposure times or durations), using different focus settings (e.g., with varying focal lengths), using different color temperatures (e.g., with varying white balance settings), and / or using other adjusted image sensor settings. For example, operations of blocks 202, 204, and 206 of the type described in connection with FIG. 5 may be performed during this time.

[0067] During the operations of block 304, device 10 may be configured to selectively warp at least some (one) of the bracket images to generate one or more corresponding warped bracket images. Because the bracket images are acquired at a time after an initial time t, the bracket images may be warped based on the pose at the time they were acquired and the pose at time t. In other words, one or more bracket images may be warped or reprojected to a common time t in the past.

[0068] The bracketed images may include additional information that may be useful for enhancing the first (initial) image. During the operation of block 306, the first image may be enhanced based on one or more warped bracketed images obtained using the operation of block 304. For example, bracketed images obtained using exposure bracketing techniques may include additional information in highlight regions that may be useful for recovering detail in brighter portions of the first image and / or additional information in shadow regions that may be useful for recovering detail in darker portions. As another example, bracketed images obtained using white balance bracketing techniques may include additional color information that may be useful for correcting the white balance of an image having mixed lighting conditions.

[0069] During the operations of block 308, device 10 may warp the enhanced image based on a different pose to generate an additional warped image. For example, during this time, the operations of blocks 210, 212, and 214 described in connection with FIG. 5 may be performed, except that the reprojection is based on the enhanced version of the first image obtained from block 306. The additional warped image may be referred to and defined herein as a reprojected version of the enhanced image.

[0070] During the operation of block 310, a reprojected version of the enhanced image obtained from block 308 may be output by a display pipeline (e.g., displayed to a user operating device 10). Displaying a reprojected image (e.g., an image reprojected based on head pose over successive frames) while preventing bracketed images (e.g., bracketed images obtained during block 302) from being shown to the user may be technically advantageous and beneficial to use reprojection to mask an otherwise reduced pass-through frame rate. Although not explicitly shown in FIG. 6, the bracketed frames obtained from block 302 may be warped, fused, and sent to a recording pipeline in parallel, as described in connection with block 208 of FIG. 5.

[0071] 6, in which the bracketed images are warped to a common time in the past (e.g., time t0), is illustrative. Alternatively, one or more of the first image and the bracketed images may be warped to a common future time, sometimes referred to herein as the "presentation time," based on an expected / predicted pose at the presentation time, and image enhancement may be performed in that space.

[0072] The examples in Figures 5 and 6 are illustrative. In some embodiments, one or more of the described operations may be modified, replaced, or omitted. In some embodiments, one or more of the described operations may be performed in parallel. In some embodiments, additional processes may be added or inserted between the described operations. If desired, the order of some operations may be reversed or changed, and / or the timing of the described operations may be adjusted so that they occur at slightly different times. In some embodiments, the described operations may be distributed across a larger system.

[0073] To help protect user privacy, any personal user information collected by the sensors may be processed using best practices. These best practices include meeting or exceeding any applicable privacy regulations. Opt-in and opt-out options and / or other options may be provided that allow users to control the use of their personal data.

[0074] According to one embodiment, a method of operating a head-mounted device includes outputting pass-through content at a pass-through frame rate using one or more displays; activating a recording mode to capture recorded content at a recording frame rate different from the pass-through frame rate while the one or more displays are outputting the pass-through content at the pass-through frame rate; acquiring images using one or more image sensors; and warping the images to generate a plurality of warped images corresponding to different times for the pass-through content.

[0075] According to another embodiment, warping the image to generate a plurality of warped images optionally includes warping the image to generate a first warped image based on a first measured or predicted pose of the head-mounted device at a first time, and warping the image to generate a second warped image based on a second measured or predicted pose of the head-mounted device at a second time subsequent to the first time.

[0076] According to another embodiment, warping the image to generate a plurality of warped images optionally includes warping the image to generate a third warped image based on a third measured or predicted pose of the head-mounted device at a third time subsequent to the second time.

[0077] According to another embodiment, the method optionally includes, after acquiring the image, acquiring a first bracket image using first adjusted image settings that are different from the image settings used to acquire the image, and, after acquiring the first bracket image, acquiring a second bracket image using second adjusted image settings that are different from the first adjusted image settings.

[0078] According to another embodiment, the method optionally includes, after acquiring the second bracket image, acquiring a third bracket image using a third adjusted image setting that is different from the first adjusted image setting and the second adjusted image setting.

[0079] According to another embodiment, outputting the pass-through content at the pass-through frame rate optionally includes outputting the pass-through content without outputting the first bracketed image, the second bracketed image, and the third bracketed image.

[0080] According to another embodiment, the image settings used to acquire the images optionally include a given exposure duration, a first adjusted image setting used to acquire the first bracketed image optionally includes a first exposure duration that is different from the given exposure duration, a second adjusted image setting used to acquire the second bracketed image optionally includes the given exposure duration and a second exposure duration that is different from the first exposure duration, and a third adjusted image setting used to acquire the third bracketed image optionally includes a third exposure duration that is different from the given exposure duration, the first exposure duration, and the second exposure duration.

[0081] According to another embodiment, the image settings used to acquire the images optionally include a given focal length, the first adjusted image setting used to acquire the first bracket image optionally includes a first focal length that is different from the given focal length, the second adjusted image setting used to acquire the second bracket image optionally includes the given focal length and a second focal length that is different from the first focal length, and the third adjusted image setting used to acquire the third bracket image optionally includes a third focal length that is different from the given focal length, the first focal length, and the second focal length.

[0082] According to another embodiment, the method optionally includes using one or more gaze detection sensors to output gaze information used to determine a given focal length for acquiring the image.

[0083] According to another embodiment, the method optionally includes warping the first bracket image, the second bracket image, and the third bracket image to a common time point to generate corresponding warped bracket images, and fusing the warped bracket images to generate a fused image for the recorded content.

[0084] According to another embodiment, the pass-through frame rate is optionally greater than the recording frame rate.

[0085] According to another embodiment, the pass-through frame rate is optionally a multiple of the recording frame rate.

[0086] According to one embodiment, a method of operating a head-mounted device includes acquiring a first image using one or more cameras; warping the first image to generate a reprojected version of the first image; outputting the reprojected version of the first image using one or more displays; acquiring bracketed images without outputting the bracketed images on the one or more displays while the reprojected version of the first image is displayed to a user; and generating recorded content based at least in part on the bracketed images.

[0087] According to another embodiment, acquiring the bracket images optionally includes acquiring a first bracket image using a first camera setting, acquiring a second bracket image using a second camera setting different from the first camera setting, and acquiring a third bracket image using a third camera setting different from the first camera setting and the second camera setting.

[0088] According to another embodiment, warping the first image to generate a reprojected version of the first image optionally includes warping the first image based on a first pose of the head-mounted device at a first time point, warping the first image based on a second pose of the head-mounted device at a second time point after the first time point, and warping the first image based on a third pose of the head-mounted device at a third time point after the second time point.

[0089] According to another embodiment, the method optionally includes warping the first bracket image, the second bracket image, and the third bracket image to a common time point to generate corresponding warped bracket images, and combining the warped bracket images to generate a fused image for the recorded content.

[0090] According to another embodiment, the method optionally includes warping the first bracket image, the second bracket image, and the third bracket image to a common time point to generate corresponding warped bracket images, and enhancing the first image using information from at least a portion of the warped bracket images.

[0091] According to one embodiment, a method of operating a head-mounted device includes acquiring, using one or more cameras, a first image and bracketed images associated with the first image; storing, using a recording pipeline, recorded content generated based at least in part on the bracketed images; warping at least one of the bracketed images to generate at least one warped bracketed image; enhancing the first image based on the at least one warped bracketed image to generate an enhanced image; and outputting, using one or more displays, a reprojected version of the enhanced image without outputting the bracketed images.

[0092] According to one embodiment, the method optionally includes warping at least one of the bracketed images to the time at which the first image is acquired.

[0093] According to one embodiment, the method optionally includes warping at least one of the bracketed images and the first image to a common time point after the time at which at least one of the bracketed images was acquired.

[0094] The above is merely exemplary and various modifications may be made to the described embodiments. The above embodiments may be implemented individually or in any combination.

Claims

1. 1. A method of operating a head-mounted device, comprising: outputting the pass-through content at the pass-through frame rate using one or more displays; activating a recording mode to capture recorded content at a recording frame rate different from the pass-through frame rate while the one or more displays are outputting the pass-through content at the pass-through frame rate; acquiring images using one or more image sensors; warping the images to generate a plurality of warped images corresponding to different times for the pass-through content; A method comprising:

2. Warping the image to generate the plurality of warped images includes: warping the image to generate a first warped image based on a first measured or predicted pose of the head-mounted device at a first time; and warping the image to generate a second warped image based on a second measured or predicted pose of the head-mounted device at a second time subsequent to the first time.

3. Warping the image to generate the plurality of warped images includes:

3. The method of claim 2, further comprising: warping the image to generate a third warped image based on a third measured or predicted pose of the head-mounted device at a third time subsequent to the second time.

4. After acquiring the images, acquiring a first bracketed image using a first adjusted image setting that is different from the image setting used to acquire the images; 10. The method of claim 1, further comprising, after acquiring the first bracket image, acquiring a second bracket image using a second adjusted image setting different from the first adjusted image setting.

5. 5. The method of claim 4, further comprising, after acquiring the second bracket image, acquiring a third bracket image using a third adjusted image setting different from the first adjusted image setting and the second adjusted image setting.

6. 6. The method of claim 5, wherein outputting the pass-through content at the pass-through frame rate includes outputting the pass-through content without outputting the first bracketed image, the second bracketed image, and the third bracketed image.

7. the image settings used to acquire the image include a given exposure duration; the first adjusted image settings used to acquire the first bracketed images include a first exposure duration that is different from the given exposure duration; the second adjusted image settings used to acquire the second bracketed images include the given exposure duration and a second exposure duration different from the first exposure duration; 6. The method of claim 5, wherein the third adjusted image settings used to acquire the third bracketed image include a third exposure duration different from the given exposure duration, the first exposure duration, and the second exposure duration.

8. the image settings used to acquire the image include a given focal length; the first adjusted image setting used to acquire the first bracketed image includes a first focal length different from the given focal length; the second adjusted image setting used to acquire the second bracketed image includes the given focal length and a second focal length different from the first focal length; 6. The method of claim 5, wherein the third adjusted image settings used to acquire the third bracketed image include a third focal length different from the given focal length, the first focal length, and the second focal length.

9. The method of claim 8 , further comprising using one or more gaze detection sensors to output gaze information used to determine the given focal length for acquiring the image.

10. warping the first bracketed image, the second bracketed image, and the third bracketed image to a common time point to generate corresponding warped bracketed images; The method of claim 5 , further comprising fusing the warped bracketed images to generate a fused image for the recorded content.

11. The method of claim 1 , wherein the pass-through frame rate is greater than the recording frame rate.

12. The method of claim 11 , wherein the pass-through frame rate is a multiple of the recording frame rate.

13. 1. A method of operating a head-mounted device, comprising: acquiring a first image using one or more cameras; warping the first image to generate a reprojected version of the first image; outputting the reprojected version of the first image using one or more displays; acquiring the bracketed images without outputting the bracketed images on the one or more displays while the reprojected version of the first image is displayed to the user; generating recorded content based at least in part on the bracketed images.

14. acquiring the bracketed images acquiring a first bracketed image using a first camera setting; acquiring a second bracketed image using a second camera setting different from the first camera setting; and acquiring a third bracketed image using a third camera setting different from the first camera setting and the second camera setting.

15. Warping the first image to generate a reprojected version of the first image comprises: warping the first image based on a first pose of the head-mounted device at a first time point; warping the first image based on a second pose of the head-mounted device at a second time point after the first time point; and warping the first image based on a third pose of the head-mounted device at a third time point after the second time point.

16. warping the first bracketed image, the second bracketed image, and the third bracketed image to a common time point to generate corresponding warped bracketed images; The method of claim 13 , further comprising combining the warped bracketed images to generate a fused image for the recorded content.

17. warping the first bracketed image, the second bracketed image, and the third bracketed image to a common time point to generate corresponding warped bracketed images; The method of claim 13 , further comprising: enhancing the first image using information from at least a portion of the warped bracketed image.

18. 1. A method of operating a head-mounted device, comprising: capturing a first image and bracketed images associated with the first image using one or more cameras; storing, with a recording pipeline, recorded content generated based at least in part on the bracketed images; warping at least one of the bracketed images to generate at least one warped bracketed image; enhancing the first image based on the at least one warped bracketed image to generate an enhanced image; outputting a reprojected version of the enhanced image without outputting the bracketed images using one or more displays; A method comprising:

19. The method of claim 18 , further comprising warping the at least one of the bracketed images to a time when the first image is acquired.

20. 20. The method of claim 18, further comprising warping the at least one of the bracketed images and the first image to a common time after the time at which the at least one of the bracketed images was acquired.

Images