Camera frame extrapolation for video perspective

Abstract

Systems and techniques are provided for synchronizing a frame rate of see-through content from a camera with a display refresh rate. An example method may include obtaining a first frame and a second frame captured by an image capture device, which may be part of a device, such as an augmented reality (XR) device. A display refresh rate of the display is greater than a camera frame rate of the image capture device. The method may determine motion between the first frame and the second frame. The method may include predicting a third frame corresponding to a time after the second frame based on the second frame and the determined motion. The method may include displaying content (e.g., mixed reality content) including the predicted third frame and virtual content rendered for the XR device.

Description

Technical Field

[0001] The present disclosure generally relates to video content for extended reality video see-through. For example, aspects of the present disclosure relate to techniques and systems for camera frame extrapolation for extended reality video see-through. Background Art

[0002] Extended reality (e.g., augmented reality, virtual reality, mixed reality, video see-through, etc.) devices such as smart glasses and head-mounted displays (HMDs) typically implement cameras and sensors to track the positioning of extended reality (XR) devices and other objects within a physical environment. XR devices can use such tracking information to provide realistic XR experiences to users of XR devices. For example, XR devices can allow users to experience or interact with immersive virtual environments or content. In order to provide realistic XR experiences, XR technologies can integrate virtual content with the physical world or a video feed depicting a scene in the physical world. In some cases, XR technologies can match the relative poses and movements of objects and devices in the physical world. For example, XR devices can use tracking information to calculate the relative poses of features of devices, objects, and / or real-world environments in order to match the relative positioning and movement of devices, objects, and / or real-world environments. In some cases, XR devices can use the poses and movements of one or more devices, objects, and / or real-world environments to render content relative to the real-world environment in a convincing manner. Relative pose information can be used to match virtual content to the user's perceived motion and the spatiotemporal state of the device, objects, and real-world environment. Summary of the invention

[0003] In some examples, systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively, "systems and techniques") for synchronizing a frame rate of perspective content from a camera with a display refresh rate are described. According to at least one illustrative example, a method for synchronizing a frame rate of perspective content from a camera with a display refresh rate is provided. In some examples, the method may include: obtaining a first frame and a second frame captured by an image capture device of an electronic device, the first frame being captured before the second frame, wherein a display refresh rate of a display is greater than a camera frame rate of the image capture device; determining motion between the first frame and the second frame; predicting a third frame corresponding to a time after the second frame based on the second frame and the motion between the first frame and the second frame; and displaying content including the predicted third frame and virtual content rendered for the electronic device at the display of the electronic device.

[0004] In another illustrative example, an apparatus for synchronizing a frame rate of perspective content from a camera with a display refresh rate is provided. In some examples, the apparatus may include a memory and one or more processors coupled to the memory, the one or more processors being configured to: obtain a first frame and a second frame captured by an image capture device, the first frame being captured before the second frame, wherein a display refresh rate of a display is greater than a camera frame rate of the image capture device; determine motion between the first frame and the second frame; predict a third frame corresponding to a time after the second frame based on the second frame and the motion between the first frame and the second frame; and display content including the predicted third frame and virtual content rendered for the apparatus at the display.

[0005] In another illustrative example, a non-transitory computer-readable medium for synchronizing a frame rate of perspective content from a camera with a display refresh rate is provided. In some examples, the non-transitory computer-readable medium may include instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: obtain a first frame and a second frame captured by an image capture device of an electronic device, the first frame being captured before the second frame, wherein a display refresh rate of a display is greater than a camera frame rate of the image capture device; determine a motion between the first frame and the second frame; predict a third frame corresponding to a time after the second frame based on the second frame and the motion between the first frame and the second frame; and display content including the predicted third frame and virtual content rendered for the electronic device at the display of the electronic device.

[0006] In another illustrative example, another apparatus for synchronizing a frame rate of perspective content from a camera with a display refresh rate is provided. In some examples, the apparatus may include means for obtaining a first frame and a second frame captured by an image capture device, the first frame being captured before the second frame, wherein the display refresh rate of the means for displaying is greater than the camera frame rate of the image capture device; means for determining motion between the first frame and the second frame; and means for predicting a third frame corresponding to a time after the second frame based on the second frame and the motion between the first frame and the second frame; wherein the means for displaying is configured to display content including the predicted third frame and virtual content rendered for the apparatus.

[0007] In some aspects, an apparatus may be a mobile device (e.g., a mobile phone or so-called "smart phone" or other mobile device), a camera (e.g., a digital camera, an Internet Protocol (IP) camera, a camera of a mobile device), a smart wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a multi-camera system, a vehicle or a computing device or system of a vehicle, or other device, or may be a portion thereof. In some aspects, the apparatus includes one or more cameras for capturing one or more images. In some aspects, the apparatus also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus described above may include one or more sensors.

[0008] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0009] The foregoing and other features and embodiments will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Exemplary embodiments of the present application are described in detail below with reference to the following drawings:

[0011] Figure 1 is a simplified block diagram illustrating an example extended reality system according to some examples of the present disclosure;

[0012] Figure 2 is a diagram illustrating an example flow for extrapolating perspective camera frames to synchronize the frame rate of the perspective camera frames with a display refresh rate according to some examples of the present disclosure;

[0013] Figure 3A is a diagram illustrating an example flow for extrapolating perspective camera frames to synchronize the frame rate of the perspective camera frames with a display refresh rate according to some examples of the present disclosure;

[0014] Figure 3B is a diagram illustrating another example flow for extrapolating perspective camera frames to synchronize the frame rate of the perspective camera frames with a display refresh rate according to some examples of the present disclosure;

[0015] Figure 4A is a diagram illustrating an example of motion estimation for extrapolating camera frames according to some examples of the present disclosure;

[0016] Figure 4Bis a diagram illustrating another example motion estimation for extrapolating camera frames according to some examples of the present disclosure;

[0017] Figure 5 is a diagram illustrating example calculations of motion vector magnitudes for extrapolating a camera frame from a previous frame according to some examples of the present disclosure;

[0018] Figure 6 is a flow chart illustrating an example process for synchronizing a frame rate of see-through content from a camera sensor with a display refresh rate of a display according to some examples of the present disclosure; and

[0019] Figure 7 is a diagram illustrating an example system architecture for implementing certain aspects described herein. DETAILED DESCRIPTION

[0020] Some aspects and embodiments of the present disclosure are provided below. Some of these aspects and embodiments can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes in order to provide a thorough understanding of each embodiment of the application. However, it will be apparent that each embodiment can be put into practice without these specific details. Each drawing and description are not intended to be restrictive.

[0021] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the subsequent description of the exemplary embodiments will provide an enabling description for implementing the exemplary embodiments to those skilled in the art. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the scope of the present application as set forth in the appended claims.

[0022] As previously explained, extended reality (XR) devices may allow users to experience or interact with immersive virtual environments or content. To provide a realistic XR experience, XR devices may integrate virtual content with the physical world and / or a video feed depicting a scene in the physical world. In some cases, XR devices may match the relative pose and movement of objects and devices in the physical world. For example, an XR device may use tracking information to calculate the relative pose of features of a device, an object, and / or a real-world environment in order to match the relative positioning and movement of the device, object, and / or the real-world environment. In some cases, an XR device may use the pose and movement of one or more devices, objects, and / or the real-world environment to render content relative to the real-world environment in a convincing manner. Relative pose information may be used to match virtual content to the user's perceived motion and the spatiotemporal state of the device, object, and the real-world environment.

[0023] An illustrative example of an XR device is a video see-through or mixed reality device. A video see-through or mixed reality device combines content rendered by the device with a depiction of a scene in the real world. In some examples, the XR device may capture a camera image of a scene and combine the image of the scene with the rendered content. For example, the device may combine a video feed captured by the XR device with the rendered content during a time warp operation (e.g., asynchronous reprojection). The video feed (e.g., a see-through frame) may depict the scene (e.g., a real-world environment) surrounding the user of the XR device, and may therefore allow the user to view virtual reality content rendered by the XR device and a view of the scene as depicted in the video feed. Even if the XR device blocks or covers the user's eyes and the user's visibility of the scene, the user may still obtain a view of the scene through the video feed captured by the XR device.

[0024] In some cases, if the camera frame rate (e.g., the rate at which the device's camera captures frames) does not match the display refresh rate (e.g., the rate at which the device's display presents new images), the image from the camera and the new image rendered by the display may become misaligned. Such misalignment may cause artifacts in the displayed content and / or an unrealistic experience for the user of the device. For example, sometimes the display refresh rate of the XR device does not match the camera frame rate of the camera of the XR device, which may cause a timing misalignment between the perspective content from the camera and the content rendered by the XR device based on the display refresh rate. In many cases, the camera of the XR device may make a compromise between resolution and frame rate, such as trading a higher resolution for a reduced frame rate, which may also cause a timing misalignment between the perspective content from the camera and the content rendered by the XR device based on the display refresh rate. However, the mismatch between the display refresh rate (e.g., and / or the frame rate of the frames rendered by the display of the XR device) and the camera perspective frame rate (e.g., the camera frame rate) may produce undesirable artifacts.

[0025] Described herein are systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively, "systems and techniques") for synchronizing the frame rate of perspective content from a camera with a display refresh rate. In some aspects, the systems and techniques described herein can improve perspective smoothness and reduce perspective latency even if the camera frame rate of the XR device is different from (e.g., less than) the display refresh rate of the display of the XR device. In some examples, the systems and techniques described herein can perform frame extrapolation based on motion estimation to predict future perspective camera frames for presentation at the XR device and ensure that the rate of the perspective camera frames matches (or significantly matches) the display refresh rate. For example, if the camera frame rate is less than the display refresh rate, the camera of the XR device may not be able to generate perspective frames at the same rate as the display refresh rate (e.g., at the same rate as the XR device renders new frames based on the display refresh rate). In order to increase the rate of perspective frames generated for display in a mixed reality or perspective experience, the XR device may predict future camera frames based on previously captured camera frames. In some examples, the speed at which the XR device predicts future camera frames may be faster than the speed at which the camera of the XR device captures new frames. Therefore, by predicting future camera frames for perspective, the XR device can increase the rate of perspective frames consistent with the display refresh rate.

[0026] In some aspects, the XR device may combine camera frames captured by the camera with predicted future frames to increase the rate of perspective frames provided to the display for rendering in combination with virtual content. In this way, some of the perspective frames displayed by the XR device may include frames captured by the camera of the XR device and frames predicted based on previous frames captured by the camera of the XR device. In some cases, the XR device may estimate the optical flow between two or more previous frames captured by the camera, and use the optical flow to predict future frames for use as perspective frames. The XR device may perform frame extrapolation based on the optical flow to predict future frames. In some cases, the XR device may use the current camera frame and the previous camera frame combined with the head pose information to estimate the motion between frames. In addition, the XR device may use the estimated motion vector field to extrapolate future perspective camera frames from previously captured frames (e.g., the last frame captured by the camera of the XR device). In some examples, the XR device may use motion estimation to perform time-warp-based extrapolation during frame synthesis. In some cases, the XR device may process motion estimation and frame extrapolation information on one or more buffers (e.g., one or more off-screen buffers).

[0027] In some cases, when predicting a future frame, the XR device may determine the magnitude of motion between previous frames, and scale the motion used to predict the future frame based on the magnitude of motion between previous frames. For example, the amount of motion estimated between camera frame 1 and camera frame 3 may be higher than the amount of motion expected for future camera frame 4. Therefore, the XR device may determine the magnitude of motion between camera frame 1 and camera frame 3, and use the magnitude of motion to scale the estimated motion between camera frame 3 and future camera frame 4. In some examples, the XR device may scale the estimated motion between camera frame 3 and future camera frame 4 to match the ratio of motion between camera frame 1 and camera frame 3. For example, the XR device may calculate the ratio between the amount of time that elapsed between camera frame 1 and camera frame 3 and the amount of time between future camera frame 4 and the display time of future camera frame 4, and use the ratio of those times to intelligently adapt the magnitude of motion applied to future camera frame 4.

[0028] In some aspects, the XR device may calculate a base point for motion vectors for camera frame extrapolation. For example, if the XR device uses standard optical flow to estimate motion vectors for two input frames with motion from point A to point B, then the motion vector may indicate that the XR device should apply motion from point A to point B. However, for frame extrapolation, the XR device may instead need to move point B forward. In other words, in the case of standard optical flow, the motion vector may cause the XR device to move the frame area behind the target depicted in the frame, rather than moving the target depicted in the frame as expected. To avoid such problems, the XR device may calculate the motion vector backwards and invert the calculated motion vector to obtain a base point for the forward-projected motion vector. For illustration, using the previous example of a frame with motion from point A to point B, the XR device may estimate the optical flow from point B to point A, and invert the corresponding motion vector to obtain a motion vector with a base point and projecting motion forward from point B.

[0029] In some cases, the XR device may use time warping (e.g., asynchronous reprojection) to correct rotational motion between frames, and use optical flow to correct translational motion between frames. For example, time warping can generally correct rotational motion efficiently and accurately. Therefore, the XR device may use time warping to correct rotational motion in an extrapolated camera frame. The XR device may reproject an extrapolated camera frame with translational motion but no rotational motion. For example, the XR device may subtract the rotational motion between frames from the estimated motion, and use the resulting translational motion to estimate future camera frames. The XR device may use time warping to correct rotational motion in future camera frames. In some examples, the XR device may determine the pose of the XR device when it captures a previous camera frame for extrapolating future camera frames. Before determining and / or applying optical flow for future camera frames, the XR device may use pose information to reproject previous camera frames without rotational motion. The XR device may then use the reprojected previous camera frame to estimate the optical flow for future camera frames, and use time warping to correct rotational motion in future camera frames.

[0030] In some cases, the XR device may correct and / or project different types of motion. For example, the XR device may correct and / or project rotational head movement (e.g., three degrees of freedom or "3DOF"). The rotational movement may include rotations around each axis. The XR device may use time warping to correct for rotational movement by applying a rotational transform to the camera frame. In addition, the XR device may correct and / or project translational head movement (e.g., six degrees of freedom or "6DOF"). The translational movement may include translation along each axis. The XR device may also correct and / or project object movement in the scene. Object movement may correspond to an object that is in motion in the scene and has changed position between camera frames. In some examples, the motion estimation described herein may cover rotational movement (e.g., 3DOF), translational movement (e.g., 6DOF), and object movement.

[0031] In some examples, the XR device may use time warping to correct and / or estimate simple types of movement, such as rotational movement. In some cases, the motion estimation and frame extrapolation described herein may be more accurate if those types of movement are removed prior to motion estimation. For example, if the time warp is configured to estimate 3DOF rotational movement, the XR device may reproject the input frame without 3DOF movement (e.g., so that the 3DOF movement has been removed) before applying motion estimation and frame extrapolation as described herein. Motion estimation and frame extrapolation focus on 6DOF movement and object movement. 3DOF correction may be applied by time warping after the frame has been extrapolated. In some examples, in the case of 3DOF reprojection, the XR device may use a matrix transformation to align the input frame. In the case of 3DOF plus 6DOF reprojection, the XR device may use depth information to be applied to mesh rendering of the mesh geometry to achieve 3DOF plus 6DOF transformation.

[0032] In some cases, for frame extrapolation, the XR device may map a previous camera frame texture to a mesh of geometric shapes. The XR device may move the positioning of vertices in the mesh based on the motion of the area of ​​the camera frame determined during motion estimation. This may shift and manipulate the area of ​​the camera frame according to the historical motion of the area of ​​the camera frame in order to move an object into the position of the object predicted by the previously described inverse backward motion estimation.

[0033] In some cases, the XR device may perform extrapolation by moving geometry or by moving UV texture coordinates. For example, the XR device may perform an extrapolated rendering process by moving the geometry of a mesh or manipulating UV texture coordinates to change how a source camera frame is mapped to the geometry. In some cases, the XR device may distort a mesh by moving the geometry of the mesh. In some examples, when moving geometry, the XR device may move each vertex during the vertex shader process based on the magnitude and direction of the motion vector at the corresponding point. In some examples, when moving texture coordinates, the XR device may determine the inverse movement and use the inverse to adjust the texture coordinates.

[0034] Examples of systems and techniques for processing data described herein are Figures 1 to 7 exemplified in and described below.

[0035] Figure 1is a diagram of an example extended reality (XR) system 100 illustrating some examples according to the present disclosure. The XR system 100 may implement the systems and techniques disclosed herein. The XR system 100 may perform various tasks and operations, such as, for example, extended reality tasks and operations (e.g., tracking, mapping, localization, content rendering, pose estimation, object detection / recognition, video see-through, etc.), image / video processing and / or post-processing, data processing and / or post-processing, computer graphics, machine vision, object modeling and registration, multimedia rendering and / or synthesis, time warping, frame extrapolation, synchronization of display refresh rate with video see-through frame rate, and / or any other data processing tasks, effects, and / or computations.

[0036] In some examples, the XR system 100 may synchronize the frame rate of the perspective content from the camera sensor with the display refresh rate. Even if the camera frame rate of the XR device is different from (e.g., less than) the display refresh rate of the display of the XR device, the XR system 100 may still improve the smoothness of the perspective frames and reduce the perspective latency. In some examples, the XR system 100 may perform frame extrapolation based on motion estimation to predict future perspective camera frames for presentation at the XR device and ensure that the rate of perspective camera frames (including extrapolated camera frames) matches (or significantly matches) the display refresh rate. For example, if the camera frame rate is less than the display refresh rate, then the camera sensor (e.g., camera sensor 102) of the XR system 100 may not be able to generate perspective frames at the same rate as the display refresh rate (e.g., at the same rate as the XR system 100 renders new frames based on the display refresh rate). In order to improve the rate of perspective frames generated for display in a mixed reality or perspective experience, the XR system 100 may predict future camera frames based on previously captured camera frames. In some examples, XR system 100 may predict future camera frames faster than new frames can be captured by the camera sensor of XR system 100. Thus, by predicting future camera frames for perspective, XR system 100 may increase the rate of perspective frames consistent with the display refresh rate.

[0037] In some examples, XR system 100 may perform tracking and positioning; pose estimation, mapping of the physical world / environment (e.g., scene) around XR system 100 (e.g., where XR system 100 is located); and positioning and rendering of virtual content on a screen, display, and / or visual plane / area as part of an XR experience. For example, XR system 100 may generate a map (e.g., a three-dimensional (3D) map) of the scene in the physical world, track the pose (e.g., position and orientation) of XR system 100 relative to the scene (e.g., relative to a 3D map of the scene), position and / or anchor virtual content in a particular location on the map of the scene, and render the virtual content on a display (e.g., display 106) so that the virtual content appears as if it is at a physical location in the scene that corresponds to the particular location on the map of the scene where the virtual content is positioned and / or anchored.

[0038] exist Figure 1 In the example shown, XR system 100 includes camera sensor 102, one or more other sensors 104, display 106, storage device 108, computing component 110, XR engine 120, image processing engine 122, and rendering engine 124. It should be noted that Figure 1 The components 102 to 126 shown are non-limiting examples provided for purposes of illustration and explanation, and other examples may include Figure 1 More, fewer, and / or different components than those shown may be included. For example, in some cases, XR system 100 may include Figure 1 One or more other processing engines, one or more receivers (e.g., global positioning system, global navigation satellite system, etc.), one or more communication devices (e.g., radio frequency (RF) interface and / or any other wireless / wired communication receiver / transmitter), one or more other hardware components, and / or one or more other software and / or hardware components not shown in the figure. Figure 7 Further described are example architecture and example hardware components that may be implemented by XR system 100 .

[0039] The XR system 100 may be part of or implemented by a single computing device or multiple computing devices. In some examples, the XR system 100 may be part of an electronic device (or multiple electronic devices), such as a camera system (e.g., a digital camera, an IP camera, a video camera, a security camera, etc.), a phone system (e.g., a smart phone, a cellular phone, a conferencing system, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a smart TV, a display device, a game console, a video streaming device, an IoT (Internet of Things) device, a smart wearable device (e.g., a head-mounted display (HMD), smart glasses, etc.), or any other suitable electronic device.

[0040] In some implementations, the camera sensor 102, one or more other sensors 104, the display 106, the storage device 108, the computing component 110, the XR engine 120, the image processing engine 122, and the rendering engine 124 may be part of the same computing device. For example, in some cases, the camera sensor 102, one or more other sensors 104, the display 106, the storage device 108, the computing component 110, the XR engine 120, the image processing engine 122, and the rendering engine 124 may be integrated into a smartphone, a laptop computer, a tablet computer, a smart wearable device, a gaming system, and / or any other computing device. In other implementations, the camera sensor 102, one or more other sensors 104, the display 106, the storage device 108, the computing component 110, the XR engine 120, the image processing engine 122, and the rendering engine 124 may be part of two or more separate computing devices. For example, in some cases, some of the components 102 to 124 may be part of or implemented by one computing device, and the remaining components may be part of or implemented by one or more other computing devices.

[0041] The camera sensor 102 may include any image and / or video sensor or capture device, such as a digital camera sensor, a video camera sensor, a smartphone camera sensor, an image / video capture device on an electronic device (such as a television or computer), a camera, etc. In some cases, the camera sensor 102 may be part of a camera or computing device (such as an XR device (e.g., HMD, smart glasses, etc.), a digital camera, a smartphone, a smart TV, a gaming system, etc.). In some examples, the camera sensor 102 may be part of a multi-camera assembly (such as a dual-camera assembly). The camera sensor 102 may capture image and / or video content (e.g., raw image and / or video data), which may then be processed by the computing component 110, the XR engine 120, the image processing engine 122, and / or the rendering engine 124, as described herein.

[0042] In some examples, camera sensor 102 may capture image data and generate frames based on the image data and / or provide the image data or frames to XR engine 120, image processing engine 122, and / or rendering engine 124 for processing. A frame may include a video frame or a static image in a video sequence. A frame may include an array of pixels representing a scene. For example, a frame may be: a red-green-blue (RGB) frame with red, green, and blue color components per pixel; a luminance, redness, blueness (YCbCr) frame with a luminance component and two chrominance (color) components (redness and blueness) per pixel; or any other suitable type of color or monochrome picture.

[0043] For the sake of brevity and explanation, Figure 1 A single camera sensor on XR system 100 is illustrated. However, in other examples, XR system 100 may include multiple camera sensors. Furthermore, reference to any of the components of XR system 100 in the singular or plural should not be construed as limiting the number of such components implemented by XR system 100 to one or more than one. For example, reference to a processor in the singular should not be construed as limiting the number of processors implemented by XR system 100 to one. One of ordinary skill in the art will recognize that reference to any of the components of XR system 100 in the singular or plural should not be construed as limiting the number of such components implemented by XR system 100 to one. Figure 1 For any of the components shown, XR system 100 may include only one of such components or more than one of such components.

[0044] The one or more other sensors 104 may include one or more sensors of any type. For example, in some cases, the one or more other sensors 104 may include one or more inertial measurement units (IMUs), one or more radio detection and ranging (RADAR) sensors, one or more light detection and ranging (LIDAR) sensors, one or more acoustic / sound sensors, one or more infrared (IR) sensors, one or more magnetometers, one or more touch sensors, one or more laser rangefinders, one or more light sensors, one or more proximity sensors, one or more motion sensors, one or more active pixel sensors, one or more machine vision sensors, one or more ultrasonic sensors, smart scene sensors, and / or any other sensor or combination of sensors.

[0045] In some cases, one or more other sensors 104 may measure the pose of XR system 100, the motion of XR system 100, and / or one or more characteristics of the environment surrounding XR system 100. In some examples, one or more other sensors 104 may detect specific forces and angular rates of XR system 100. In some cases, one or more other sensors 104 may detect the orientation of XR system 100. One or more other sensors 104 may generate linear acceleration measurements, rotation rate measurements, and / or heading measurements. In some examples, one or more other sensors 104 may be used to measure the pitch, roll, and yaw of XR system 100. In some cases, XR engine 120 may use data and / or measurements from camera sensor 102 and / or one or more other sensors 104 to track the pose of XR system 100.

[0046] Display 106 may include any display device for displaying content, such as virtual reality content, video see-through content, and / or any type of content. In some examples, display 106 may include a screen configured to output digital content (e.g., images, videos, graphics, and / or any type of visual content). Display 106 is capable of displaying content at one or more display resolutions and / or one or more display refresh rates.

[0047] Storage 108 may be any storage device for storing data. In addition, storage 108 may store data from any of the components of XR system 100. For example, storage 108 may store data from camera sensor 102 (e.g., image or video data), data from one or more other sensors 104 (e.g., measurements), data from computing component 110 (e.g., processing parameters, preferences, virtual content, rendered content, scene maps, tracking and positioning data, object detection data, video frames, configurations, XR application data, recognition data, outputs, computational results, etc.), data from XR engine 120, data from image processing engine 122, and / or data from rendering engine 124 (e.g., output frames). In some examples, storage 108 may include a buffer for storing frames and / or other data for processing by computing component 110.

[0048] The one or more computing components 110 may include a central processing unit (CPU) 112, a graphics processing unit (GPU) 114, a digital signal processor (DSP) 116, and / or an image signal processor (ISP) 118. The computing component 110 may perform various operations such as image enhancement, computer vision, graphics rendering, extended reality (e.g., tracking, localization, pose estimation, mapping, content anchoring, content rendering, video see-through, etc.), image / video processing, sensor processing, recognition (e.g., text recognition, facial recognition, object recognition, feature recognition, tracking or pattern recognition, scene recognition, occlusion detection, etc.), machine learning, filtering, object detection, frame extrapolation, time warping, synchronization of display refresh rate with video see-through frame rate, and / or any of the various operations described herein. Figure 1 In the example shown, computing component 110 may implement XR engine 120, image processing engine 122, and rendering engine 124. In other examples, computing component 110 may also implement one or more other processing engines.

[0049] Operations for XR engine 120, image processing engine 122, and rendering engine 124 (as well as any other processing engines) may be implemented by any of the computing components in computing components 110. In one illustrative example, operations of rendering engine 124 may be performed by GPU 114, and operations of XR engine 120 and image processing engine 122 may be performed by CPU 112, DSP 116, and / or ISP 118. In some cases, computing component 110 may include other electronic circuitry or hardware, computer software, firmware, or any combination thereof to perform any of the various operations described herein.

[0050] In some examples, XR engine 120 may perform XR operations based on data from camera sensor 102, one or more other sensors 104, and / or storage device 108. In some examples, XR engine 120 may perform tracking, positioning, pose estimation, mapping, content anchoring, motion estimation, and / or any other XR operation / functionality. In some examples, XR engine 120 may coordinate with image processing engine 122 and / or rendering engine 124 to render user interfaces, control objects, virtual content, perspective camera frames, mixed reality content, and / or any other visual content.

[0051] Image processing engine 122 may perform one or more image processing operations. In some examples, image processing engine 122 may perform image processing operations based on data from camera sensor 102, one or more other sensors 104, storage 108, and / or one or more other devices or components. In some examples, image processing engine 122 may also use data from other devices or applications, such as data from or reported by other user devices and / or data provided by a user of XR system 100.

[0052] The image processing engine 122 may perform image processing operations such as, for example, filtering, demosaicing, scaling, color correction, color conversion, segmentation, noise reduction filtering, spatial filtering, artifact correction, etc. The rendering engine 124 may obtain image data generated and / or processed by the computing component 110, the camera sensor 102, the XR engine 120, and / or the image processing engine 122, and render content (e.g., virtual content, video, images, etc.) for presentation on the display 106.

[0053] In some examples, image processing engine 122 may compute the pose of XR system 100, compute motion in frames captured by camera sensor 102, perform time warping operations, extrapolate future frames based on previous frames, synchronize a display refresh rate with a video perspective frame rate, and / or perform any other operations as described herein. In some examples, image processing engine 122 may use data from camera sensor 102, one or more other sensors 104, storage device 108, and / or any other components to detect objects in a scene (e.g., edges, surfaces, items on surfaces, windows, doors, walls, tables, books, equipment, chairs, people, hands, animals, etc.), identify characteristics of a scene and / or characteristics of objects in the scene, identify occlusions in the scene, and / or the like.

[0054] In some examples, the XR engine 120, the image processing engine 122, and / or the rendering engine 124 may perform various operations to provide an XR experience to a user of the XR system 100. The XR experience may include using the XR system 100 to present XR content (e.g., virtual reality content, augmented reality content, mixed reality content, etc.) to a user associated with the XR system 100 (e.g., a user wearing the XR system 100 and / or otherwise using the XR system 100 for an XR experience). In some examples, the XR content and experience may be provided by the XR system 100 through an XR application that provides a specific XR experience, such as, for example, an XR gaming experience, an XR classroom experience, an XR shopping experience, an XR entertainment experience, an XR activity (e.g., an operation, a troubleshooting activity, etc.), etc. During the XR experience, the user may use the XR system 100 to view and / or interact with the virtual content. In some cases, a user may view and / or interact with virtual content while also being able to view and / or interact with the physical environment depicted in frames captured by camera sensor 102, thereby allowing the user to have an immersive experience between the physical environment and the virtual content (blended or integrated with the frames depicting the physical environment).

[0055] Although XR system 100 is shown as including certain components, one of ordinary skill will appreciate that XR system 100 may include more than Figure 1 For example, in some instances, XR system 100 may also include one or more memory devices (e.g., RAM, ROM, and / or cache, etc.), one or more network interfaces (e.g., wired and / or wireless communication interfaces, etc.), and / or Figure 1 Other hardware or processing devices not shown in the figure. Figure 7 Illustrative examples of computing devices and hardware components that may be implemented with XR system 100 are described.

[0056] Figure 2is a diagram illustrating an example flow 200 for extrapolating a perspective camera frame to synchronize the frame rate of the perspective camera frame with the display refresh rate. Flow 200 may combine camera frames obtained from camera sensor 102 with virtual content rendered by XR system 100 to provide a mixed reality experience to a user wearing XR system 100. Even if the user cannot see the scene (e.g., a scene in the physical world) around the user while wearing XR system 100 (e.g., because XR system 100 blocks the user's eyes and / or visibility of the scene), the camera frame from camera sensor 102 may still capture the scene and may be used as a perspective camera frame to allow the user to see the scene (e.g., through the perspective camera frame). XR system 100 may render virtual content as well as the perspective camera frame to provide a mixed reality experience to the user, wherein the user may see virtual content rendered within a depiction of the scene from the perspective camera frame.

[0057] In this example flow 200, at time t1, the time warp process 210 receives a first camera frame 202 captured by the camera sensor 102 and virtual content from the eye buffer 220, and generates a rendered frame for presentation (e.g., via the display 106) to a user of the XR system 100. The rendered frame may be a mixed reality frame generated from the first camera frame 202 captured by the camera sensor 102 and the virtual content from the eye buffer 220. For example, to generate the rendered frame, the time warp process 210 may combine the first camera frame 202 with the virtual content from the eye buffer 220. The first camera frame 202 may depict a scene around the user wearing the XR system 100, and the virtual content may include rendered content generated by the XR system 100 (e.g., via an application of the XR system 100, such as an XR application). Thus, the rendered frame may depict the virtual content from the eye buffer 220 within a view of the scene depicted by the first camera frame 202.

[0058] When generating a rendered frame, the time warp process 210 may warp or reproject the rendered frame before transmitting the rendered frame to the display 106 for presentation to the user. The time warp process 210 may warp or reproject the rendered frame to compensate for movement of the XR system 100 (e.g., and thus compensate for head movement of the user). In some examples, the time warp process 210 may warp or reproject the rendered frame to account for movement that occurs after the rendered frame is generated and before the rendered frame is displayed (e.g., at the time the rendered frame is transmitted to the display 106 and / or at the time the display 106 is refreshed to update or refresh content displayed by the display 106 (e.g., at a display refresh time)). In some cases, the time warp process 210 may warp or reproject the rendered frame to also account for movement that occurs between the time the first camera frame 202 is captured by the camera sensor 102 and the time the rendered frame is displayed at the display 106 (and / or at the time the rendered frame is transmitted to the display 106 for presentation to the user).

[0059] In some examples, the time warp process 210 may warp the rendered frame by geometrically distorting or reprojecting the rendered frame to account for movement since the rendered frame was generated and / or since the first camera frame 202 was captured by the camera sensor 102. In some cases, the time warp process 210 may warp the rendered frame to compensate for rotational movement (e.g., three degrees of freedom (3DOF) movement) of the XR system 100 (and / or rotational movement of a head of a user wearing the XR system 100). For example, the time warp process 210 may geometrically warp the rendered frame in the direction in which the XR system 100 is rotated (and / or the direction in which a user wearing the XR system 100 rotates the user's head) between the time the rendered frame is generated (and / or the time the first camera frame 202 is captured by the camera sensor 102) and the time the rendered frame is transmitted to the display 106 and / or displayed at the display 106. In some examples, the time warp process 210 may rotate the rendered frame about each axis of rotation. In some cases, the time warp process 210 may compensate for rotational movement by applying a rotational transform to a rendered frame to be displayed to a user of the XR system 100 .

[0060] In some cases, the warping or reprojection performed by the time warp process 210 may also compensate for translational movement (e.g., six degrees of freedom (6DOF) movement) of the XR system 100 (and / or the head of the user wearing the XR system 100). For example, the time warp process 210 may geometrically warp the rendered frame in the direction of the translational movement of the XR system 100 between the time the rendered frame is generated (and / or the time the first camera frame 202 is captured by the camera sensor 102) and the time the rendered frame is transmitted to the display 106 and / or displayed at the display 106. In some examples, the time warp process 210 may apply a translation to the rendered frame about each translation axis. In some cases, the time warp process 210 may compensate for the translational movement by using depth information about the rendered frame and a rough estimate of how a region of the rendered frame will move due to parallax.

[0061] At time t2, the time warp process 212 may obtain virtual content from the eye buffer 222 to generate a new rendered frame for presentation by the display 106 at the next display refresh time of the display 106. Since the display refresh rate of the display 106 is less than the camera frame rate of the camera sensor 102, the time warp process 212 cannot obtain a new camera frame from the camera sensor 102 at time t2. Therefore, the time warp process 212 may generate a new rendered frame based on the camera frame 202 obtained at time t1 and the virtual content from the eye buffer 222. In some examples, when generating the new rendered frame, the time warp process 212 may warp or reproject the new rendered frame to compensate for movement, as previously explained. In some examples, the time warp process 212 may compensate for movement since the virtual content from the eye buffer 222 was rendered and / or the new rendered frame was generated, as well as movement since the time the new rendered frame was transmitted to the display 106 and / or the time the new rendered frame was displayed at the display 106. In some cases, the time warp process 212 may warp or reproject the camera frame 202 to compensate for movement since the camera frame 202 was captured by the camera sensor 102 (or the time the camera frame 202 was warped by the time warp process 210) and since the time the newly rendered frame was transmitted to the display 106 and / or the time the newly rendered frame was displayed at the display 106.

[0062] At time t3, camera sensor 102 may have second camera frame 204 ready for presentation by XR system 100. For example, the camera frame rate of camera sensor 102 may be such that camera sensor 102 may generate and provide a new camera frame every other time interval (e.g., at time t1 prior to time t3), where each time interval is based on or controlled by a display refresh rate of display 106. Thus, at time t3, time warp process 214 may obtain second camera frame 204 from camera sensor 102 and additional virtual content from eye buffer 224 to generate a new rendered frame for presentation by display 106 at the next display refresh time of display 106.

[0063] The new rendered frame may be a mixed reality frame generated from the second camera frame 204 and the additional virtual content from the eye buffer 224. For example, to generate the new rendered frame at time t3, the time warp process 214 may combine the second camera frame 204 with the additional virtual content from the eye buffer 224. The second camera frame 204 may depict a scene around a user wearing the XR system 100, and the additional virtual content may include rendered content generated by the XR system 100 (e.g., via an application of the XR system 100, such as an XR application). Thus, the new rendered frame may depict the virtual content from the eye buffer 224 within a view of the scene depicted by the second camera frame 204.

[0064] When generating a new rendered frame, the time warp process 214 may warp or reproject the new rendered frame before transmitting the new rendered frame to the display 106 for presentation to the user. The time warp process 214 may warp or reproject the new rendered frame to compensate for movement of the XR system 100 (e.g., and thus compensate for head movement of the user). In some examples, the time warp process 214 may warp or reproject the new rendered frame to account for movement that occurs after the new rendered frame is generated and before the new rendered frame is displayed (e.g., at the time the new rendered frame is transmitted to the display 106 and / or at the time the display 106 is refreshed to update or refresh content displayed by the display 106 (e.g., at a display refresh time)). In some cases, the time warp process 214 may warp or reproject the new rendered frame to also account for movement that occurs between the time the second camera frame 204 is captured by the camera sensor 102 and the time the new rendered frame is displayed at the display 106 (and / or at the time the new rendered frame is transmitted to the display 106 for presentation to the user).

[0065] In some examples, the time warp process 214 may warp the rendered frame by geometrically distorting or reprojecting the new rendered frame to account for movement since the new rendered frame was generated and / or since the second camera frame 204 was captured by the camera sensor 102. In some cases, the time warp process 214 may warp the new rendered frame to compensate for rotational movement of the XR system 100 (and / or rotational movement of the head of the user wearing the XR system 100). In some cases, the warping or reprojection performed by the time warp process 214 may also compensate for translational movement of the XR system 100 (and / or the head of the user wearing the XR system 100). For example, the time warp process 210 may geometrically warp the new rendered frame in the direction of translational movement of the XR system 100 between the time the new rendered frame is generated (and / or the time the second camera frame 204 is captured by the camera sensor 102) and the time the new rendered frame is transmitted to the display 106 and / or displayed at the display 106.

[0066] At time t4, the time warp process 216 may obtain the additional virtual content from the eye buffer 226 to generate a new rendered frame 228 for presentation by the display 106 at the next display refresh time of the display 106. Since the display refresh rate of the display 106 is less than the camera frame rate of the camera sensor 102, the time warp process 216 cannot obtain a new camera frame from the camera sensor 102 at time t4. However, in order to synchronize the display refresh rate with the camera frame rate of the camera sensor 102, the XR system 100 (e.g., via the time warp process 216 or an application on the XR system 100) may extrapolate a new camera frame based on the optical flow between the first camera frame 202 and the second camera frame 204. For example, the XR system 100 may calculate the motion between the first camera frame 202 and the second camera frame 204, and use the calculated motion to generate the extrapolated camera frame 206. The extrapolated camera frame 206 may represent a predicted future frame.

[0067] XR system 100 may generate extrapolated camera frame 206 by adjusting second camera frame 204 based on the optical flow between first camera frame 202 and second camera frame 204. In some examples, XR system 100 may adjust the magnitude of the optical flow used to generate extrapolated camera frame 206 based on the time interval between first camera frame 202 and second camera frame 204 and the time interval between second camera frame 204 and extrapolated camera frame 206. For example, in Figure 2 In the illustrative example shown, the motion between the first camera frame 202 and the second camera frame 204 spans two time intervals (e.g., from time t1 to time t2 and from time t2 to time t3), while the motion between the second camera frame 204 and the extrapolated camera frame 206 spans from time t3 to time t4. Therefore, instead of adjusting the second camera frame 204 based on the entire motion between the first camera frame 202 and the second camera frame 204, the XR system 100 may scale the motion calculated between the first camera frame 202 and the second camera frame 204 according to the time interval between the second camera frame 204 and the extrapolated camera frame 206.

[0068] For example, XR system 100 may determine motion between first camera frame 202 and second camera frame 204. XR system 100 may then determine a magnitude of motion at each time interval. In some examples, to determine the magnitude of motion, XR system 100 may divide the motion between first camera frame 202 and second camera frame 204 by the number of time intervals after first camera frame 202 until second camera frame 204 (e.g., the number of time intervals spanned by the motion). Figure 2In the illustrative example shown, the motion between the first camera frame 202 and the second camera frame 204 spans a first time interval between time t1 and time t2 of the first camera frame 202 and a second time interval between time t2 and time t3 of the second camera frame 204. Therefore, to determine the magnitude of the motion between the first camera frame 202 and the second camera frame 204, the XR system 100 may divide the motion between the first camera frame 202 and the second camera frame 204 by 2 (e.g., this takes into account the time interval between the first camera frame 202 and the second camera frame 204). The XR system 100 may then use the calculated magnitude of the motion to generate an extrapolated camera frame 206. The magnitude of the motion may reflect the estimated amount of motion from time t3 to time t4.

[0069] In other words, XR system 100 may scale the motion between second camera frame 204 and extrapolated camera frame 206 to match the ratio of motion between first camera frame 202 and second camera frame 204. To illustrate, XR system 100 may calculate a ratio between the amount of time that elapsed between first camera frame 202 and second camera frame 204 and the amount of time between second camera frame 204 and extrapolated camera frame 206, and use the ratio of those times to intelligently adapt the magnitude of motion used to generate extrapolated camera frame 206.

[0070] In some examples, XR system 100 may generate extrapolated camera frame 206 by using the estimated motion magnitude to adjust second camera frame 204. For example, XR system 100 may adjust second camera frame 204 by projecting the motion magnitude from time t3 associated with second camera frame 204 forward (e.g., into the future) to time t4 associated with extrapolated camera frame 206.

[0071] In some cases, the XR system 100 may calculate a base point for motion vectors for camera frame extrapolation. For example, if the XR system 100 estimates motion vectors for the first camera frame 202 and the second camera frame 204 (e.g., from time t1 to time t3) using standard optical flow, the motion vectors may indicate that the XR system 100 should apply motion from time t1 to time t3. However, for frame extrapolation, the XR system 100 may instead need to apply motion from time t3 (associated with the second camera frame 204) forward. In other words, in the case of standard optical flow, the motion vectors may cause the XR system 100 to move a region of the frame behind an object depicted in the frame, rather than moving the object depicted in the frame as expected. To avoid such problems, the XR system 100 may calculate motion vectors backwards and invert the calculated motion vectors to obtain a base point for the forward-projected motion vector. For illustration, the XR system 100 may estimate optical flow from time t1 to time t3, and invert the corresponding motion vectors to obtain a motion vector having a base point and projecting motion forward from time t3.

[0072] In some cases, the time warp process 216 may correct for rotational motion between the second camera frame 204 and the extrapolated camera frame 206, and use optical flow to correct for translational motion between the second camera frame 204 and the extrapolated camera frame 206. For example, time warping may generally correct for rotational motion efficiently and accurately. Therefore, the XR system 100 may use the time warp process 216 to correct for rotational motion in the extrapolated camera frame 206. The XR system 100 may reproject the extrapolated camera frame 206 with translational motion but without rotational motion. For example, the XR system 100 may subtract the rotational motion between the second camera frame 204 and the extrapolated camera frame 206 from the estimated motion, and use the resulting translational motion to estimate the extrapolated camera frame 206. The XR system 100 may use the time warp process 216 to correct for rotational motion in the extrapolated camera frame 206. In some examples, the XR system 100 may determine the pose of the XR system 100 when it captured the second camera frame 204 used to generate the extrapolated camera frame 206. Before determining and / or applying the optical flow for the extrapolated camera frame 206, the XR system 100 may use the pose information to reproject the second camera frame 204 without rotational motion. The XR system 100 may then use the reprojected second camera frame 204 to estimate the optical flow for the extrapolated camera frame 206 and use the time warping process 216 to correct the rotational motion in the extrapolated camera frame 206.

[0073] In other examples, the time warp process 216 may warp the extrapolated camera frame 206 to compensate for translational and rotational motion of the XR system 100, and use optical flow to compensate for translational motion of objects in the scene depicted by the extrapolated camera frame 206. The XR system 100 may correct and / or project different types of motion, as previously explained. For example, the XR system 100 may correct and / or project rotational movement (e.g., 3DOF movement) of the XR system 100. The time warp process 216 may correct for rotational movement by applying a rotational transform to the extrapolated camera frame 206. In addition, the XR system 100 may correct and / or project translational movement (e.g., 6DOF movement) of the XR system 100. The XR system 100 may also correct and / or project object movement in the scene. Object movement may correspond to objects that are in motion in the scene and have changed position between camera frames. In some examples, motion estimation and compensation may cover rotational movement (e.g., 3DOF), translational movement (e.g., 6DOF), and object movement.

[0074] In some examples, the XR system 100 may use the time warp process 216 to correct and / or estimate simple types of movement, such as rotational movement. In some cases, if those types of movement are removed prior to motion estimation, the motion estimation and frame extrapolation may be more accurate. For example, if the time warp process 216 is configured to estimate 3DOF rotational movement, the XR system 100 may reproject the extrapolated camera frame 206 without 3DOF movement (e.g., such that the 3DOF movement has been removed) before applying motion estimation and frame extrapolation as described herein. Motion estimation and frame extrapolation focus on 6DOF movement and object movement. 3DOF correction may be applied by the time warp process 216 after the extrapolated camera frame 206 has been generated. In some examples, in the case of 3DOF reprojection, the XR system 100 may use a matrix transformation to align the extrapolated camera frame 206. In the case of 3DOF plus 6DOF reprojection, the XR system 100 may implement a 3DOF plus 6DOF transformation using mesh rendering where depth information is applied to the mesh geometry.

[0075] In some cases, for frame extrapolation, the XR system 100 may texture map the second camera frame 204 to a mesh of geometric shapes. The XR system 100 may move the positioning of vertices in the mesh based on the motion of the area of ​​the second camera frame 204 determined during motion estimation. This may shift and manipulate the area of ​​the second camera frame 204 according to its historical motion in order to move the object into the position of the object predicted by the previously described inverse backward motion estimation.

[0076] In some cases, the XR system 100 may perform camera frame extrapolation by moving geometry or by moving UV texture coordinates. For example, the XR system 100 may perform an extrapolated rendering process by moving the geometry of the mesh or manipulating the UV texture coordinates to change how the second camera frame 204 is mapped to the geometry. In some cases, the XR system 100 may distort the mesh by moving the geometry of the mesh. In some examples, when moving geometry, the XR system 100 may move each vertex during the vertex shader process based on the magnitude and direction of the motion vector at the corresponding point. In some examples, when moving texture coordinates, the XR system 100 may determine an inverse movement and use the inverse to adjust the texture coordinates.

[0077] Once the timewarp process 216 has completed generating / processing the extrapolated camera frame 206, the timewarp process 216 may generate a new rendered frame 228 by combining the extrapolated camera frame 206 with the virtual content from the eye buffer 226. In some examples, the timewarp process 216 may then warp or reproject the new rendered frame 228 as previously described. Once the timewarp process 216 has completed generating and / or adjusting the new rendered frame 228, the timewarp process 216 may provide the new rendered frame 228 to the display 106 for presentation to the user.

[0078] Figure 3A is a diagram illustrating an example flow 300 for extrapolating a perspective camera frame using a time warp process. Flow 300 may combine camera frames obtained from camera sensor 102 with virtual content rendered by XR system 100 to provide a mixed reality experience to a user wearing XR system 100. Even if the user cannot see the scene around the user while wearing XR system 100 (e.g., because XR system 100 blocks the user's eyes and / or visibility of the scene), camera frames from camera sensor 102 may still capture the scene and may be used as perspective camera frames to allow the user to see the scene (e.g., through the perspective camera frame). XR system 100 may render virtual content as well as the perspective camera frame to provide a mixed reality experience to the user, wherein the user may see virtual content rendered within a depiction of the scene from the perspective camera frame.

[0079] In this example flow 300, at time t1, the time warp process 320 receives a first camera frame 310 captured by the camera sensor 102 and virtual content from the eye buffer 330, and generates a rendered frame for presentation (e.g., via the display 106) to a user of the XR system 100. The rendered frame may be a mixed reality frame generated from the first camera frame 310 captured by the camera sensor 102 and the virtual content from the eye buffer 330. For example, to generate the rendered frame, the time warp process 320 may combine the first camera frame 310 with the virtual content from the eye buffer 330. The first camera frame 310 may depict a scene around the user wearing the XR system 100, and the virtual content may include rendered content generated by the XR system 100 (e.g., via an application of the XR system 100, such as an XR application). Thus, the rendered frame may depict the virtual content from the eye buffer 330 within a view of the scene depicted by the first camera frame 310.

[0080] When generating a rendered frame, the time warp process 320 may warp or reproject the rendered frame before transmitting the rendered frame to the display 106 for presentation to the user. The time warp process 320 may warp or reproject the rendered frame to compensate for movement of the XR system 100 (e.g., and thus compensate for head movement of the user). In some examples, the time warp process 320 may optionally obtain a pose 302 of the XR system 100 at the time when the first camera frame 310 was generated, and use the pose 302 of the XR system 100 to determine motion associated with the XR system 100 and / or compensate for movement of the XR system 100. In some cases, the time warp process 320 may use the pose 302 and motion information associated with the first camera frame 310 and / or the XR system 100 to estimate a future pose of the XR system 100 at the time when the rendered frame is expected to be presented at the display 106, and warp or reproject the rendered frame based at least in part on the future pose.

[0081] For example, time warp process 320 may use pose 302 and motion information associated with first camera frame 310 and / or XR system 100 to estimate a future pose of XR system 100 when presenting a rendered frame at display 106 at a future time. Time warp process 320 may warp or reproject the rendered frame to account for any changes between pose 302 and the estimated future pose. In some cases, time warp process 320 may warp or reproject the rendered pose to account for changes in the pose and / or to appear as if it were rendered from the perspective of the future pose.

[0082] In some examples, the time warp process 320 may warp or reproject the rendered frame to account for movement that occurs after the rendered frame is generated and before the rendered frame is displayed (e.g., at the time the rendered frame is transmitted to the display 106 and / or the time the display 106 is refreshed to update or refresh the content displayed by the display 106 (e.g., at a display refresh time). In some cases, the time warp process 320 may warp or reproject the rendered frame to also account for movement that occurs between the time when the first camera frame 310 is captured by the camera sensor 102 and the time when the rendered frame is displayed at the display 106 (and / or the time when the rendered frame is transmitted to the display 106 for presentation to the user).

[0083] In some examples, the time warp process 320 may warp the rendered frame by geometrically distorting or reprojecting the rendered frame to account for movement since the rendered frame was generated and / or since the first camera frame 310 was captured by the camera sensor 102. In some cases, the time warp process 320 may warp the rendered frame to compensate for rotational movement (e.g., 3DOF movement) of the XR system 100 (e.g., rotational movement of the XR system 100 relative to the pose 302 of the XR system 100). For example, the time warp process 320 may geometrically warp the rendered frame in the direction in which the XR system 100 is rotated (and / or the direction in which a user wearing the XR system 100 rotates the user's head) between the time the rendered frame is generated (and / or the time the first camera frame 310 is captured by the camera sensor 102) and the time the rendered frame is transmitted to the display 106 and / or displayed at the display 106. In some examples, the time warp process 320 may rotate the rendered frame about any axis of rotation. In some cases, the time warp process 320 may compensate for rotational movement by applying a rotational transform to a rendered frame to be displayed to a user of the XR system 100 .

[0084] In some cases, the warping or reprojection performed by the time warp process 320 may also compensate for translational movement (e.g., 6DOF movement) of the XR system 100 (e.g., pose 302 of the XR system 100). For example, the time warp process 320 may geometrically warp the rendered frame in the direction of the translational movement of the XR system 100 between the time the rendered frame is generated (and / or the time the first camera frame 310 is captured by the camera sensor 102) and the time the rendered frame is transmitted to the display 106 and / or displayed at the display 106. In some examples, the time warp process 320 may apply a translation to the rendered frame about any translation axis. In some cases, the time warp process 320 may compensate for the translational movement by using depth information about the rendered frame and a rough estimate of how a region of the rendered frame will move due to parallax.

[0085] At time t2, the time warp process 322 may obtain the virtual content from the eye buffer 332 to generate a new rendered frame for presentation by the display 106 at the next display refresh time of the display 106. Since the display refresh rate of the display 106 is less than the camera frame rate of the camera sensor 102, the time warp process 322 cannot obtain a new camera frame from the camera sensor 102 at time t2. Therefore, the time warp process 322 may generate a new rendered frame based on the camera frame 310 obtained at time t1 and the virtual content from the eye buffer 332. In some examples, when generating the new rendered frame, the time warp process 322 may warp or reproject the new rendered frame to compensate for movement, as previously explained.

[0086] In some examples, the time warp process 322 may compensate for movement since the virtual content from the eye buffer 332 was rendered and / or the new rendered frame was generated, as well as movement since the time the new rendered frame was transmitted to the display 106 and / or the time the new rendered frame was displayed at the display 106. In some cases, the time warp process 322 may warp or reproject the camera frame 310 to compensate for movement since the camera frame 310 was captured by the camera sensor 102 (or the time the camera frame 310 was warped by the time warp process 320), as well as movement since the time the new rendered frame was transmitted to the display 106 and / or the time the new rendered frame was displayed at the display 106.

[0087] At time t3, camera sensor 102 may have second camera frame 312 ready for presentation by XR system 100. For example, the camera frame rate of camera sensor 102 may be such that camera sensor 102 may generate and provide a new camera frame every other time interval (e.g., at time t1 prior to time t3), where each time interval is based on or controlled by a display refresh rate of display 106. Thus, at time t3, time warp process 324 may obtain second camera frame 312 from camera sensor 102 and additional virtual content from eye buffer 334 to generate a new rendered frame for presentation by display 106 at the next display refresh time of display 106.

[0088] The new rendered frame may be a mixed reality frame generated from the second camera frame 312 and the additional virtual content from the eye buffer 334. For example, to generate the new rendered frame at time t3, the time warp process 324 may combine the second camera frame 312 with the additional virtual content from the eye buffer 334. The second camera frame 312 may depict a scene around a user wearing the XR system 100, and the additional virtual content may include rendered content generated by the XR system 100 (e.g., via an application of the XR system 100, such as an XR application). Thus, the new rendered frame may depict the virtual content from the eye buffer 334 within a view of the scene depicted by the second camera frame 312.

[0089] When generating a new rendered frame, the time warp process 324 may warp or reproject the new rendered frame before transmitting the new rendered frame to the display 106 for presentation to the user. The time warp process 324 may warp or reproject the new rendered frame to compensate for movement of the XR system 100 (e.g., and therefore compensate for head movement of the user).

[0090] In some cases, time warp processing 324 may optionally obtain pose 304 of XR system 100 at the time when second camera frame 312 is generated, and use pose 304 of XR system 100 to determine motion associated with XR system 100 and / or compensate for movement of XR system 100. In some cases, time warp processing 324 may use pose 304 and motion information associated with second camera frame 312 and / or XR system 100 to estimate a future pose of XR system 100 at a time when a rendered frame is expected to be presented at display 106, and warp or reproject the rendered frame based at least in part on the future pose. In some cases, XR system 100 may estimate the future pose of XR system 100 and / or any movement associated with a new rendered frame (and / or XR system 100) based on pose 302 and pose 304. For example, XR system 100 may estimate a future pose of XR system 100 and / or any movement associated with a newly rendered frame (and / or XR system 100 ) based on changes in pose 302 and pose 304 and / or estimated movement between pose 302 and pose 304 .

[0091] In some examples, XR system 100 may determine a magnitude of a change between pose 302 and pose 304, and use the magnitude of the change between pose 302 and pose 304 to estimate a future pose of XR system 100 at a time when a new rendered frame will be presented at display 106. The future pose of XR system 100 may reflect a change in pose 304 at a future time when a new rendered frame will be presented at display 106. In some examples, such a change in pose 304 may be projected from pose 304 to a future time based on the magnitude of the change between pose and pose 304 and an estimated amount of time between the time of pose 304 and the future time when a new rendered frame will be presented at display 106.

[0092] In some examples, the time warp process 324 may warp or reproject the new rendered frame to account for movement that occurs after the new rendered frame is generated and before the new rendered frame is displayed (e.g., at the time the new rendered frame is transmitted to the display 106 and / or the time the display 106 is refreshed to update or refresh the content displayed by the display 106 (e.g., at a display refresh time). In some cases, the time warp process 324 may warp or reproject the new rendered frame to also account for movement that occurs between the time the second camera frame 312 is captured by the camera sensor 102 and the time the new rendered frame is displayed at the display 106 (and / or the time the new rendered frame is transmitted to the display 106 for presentation to the user).

[0093] In some examples, the time warp process 324 may warp the rendered frame by geometrically distorting or reprojecting the new rendered frame to account for movement since the new rendered frame was generated and / or since the second camera frame 312 was captured by the camera sensor 102. In some cases, the time warp process 324 may warp the new rendered frame to compensate for rotational movement of the XR system 100 (and / or rotational movement of the head of the user wearing the XR system 100). In some cases, the warping or reprojection performed by the time warp process 324 may also compensate for translational movement of the XR system 100 (and / or the head of the user wearing the XR system 100). For example, the time warp process 320 may geometrically warp the new rendered frame in the direction of translational movement of the XR system 100 between the time the new rendered frame is generated (and / or the time the second camera frame 312 is captured by the camera sensor 102) and the time the new rendered frame is transmitted to the display 106 and / or displayed at the display 106.

[0094] At time t4, the time warp process 326 may obtain additional virtual content from the eye buffer 336 to generate a new rendered frame 338 for presentation by the display 106 at the next display refresh time of the display 106. Since the display refresh rate of the display 106 is less than the camera frame rate of the camera sensor 102, the time warp process 326 cannot obtain a new camera frame from the camera sensor 102 at time t4. In order to synchronize the display refresh rate with the camera frame rate of the camera sensor 102, the time warp process 326 may extrapolate a new camera frame based on the optical flow between the first camera frame 310 and the second camera frame 312. For example, the time warp process 326 may calculate the motion 314 between the first camera frame 310 and the second camera frame 312, and use the motion 314 to generate an extrapolated camera frame. The extrapolated camera frame may represent a predicted future frame. Then, the time warp process 326 may generate a new rendered frame 338 based on the extrapolated camera frame and the virtual content from the eye buffer 336.

[0095] Time warp process 326 may determine motion 314 by estimating optical flow between first camera frame 310 and second camera frame 312. In some cases, time warp process 326 may additionally calculate motion 314 using pose 302, pose 304, and pose 306 of XR system 100 at time t4 for determining an extrapolated camera frame. In some examples, time warp process 326 may calculate the extrapolated camera frame from the perspective of XR system 100 at pose 306.

[0096] In some examples, the time warping process 326 may generate an extrapolated camera frame by adjusting the second camera frame 312 based on the optical flow between the first camera frame 310 and the second camera frame 312. In some cases, the time warping process 326 may adjust the magnitude of the optical flow used to generate the extrapolated camera frame based on the time interval between the first camera frame 310 and the second camera frame 312 and the time interval between the second camera frame 312 and the extrapolated camera frame. Figure 3A In the illustrative example shown, the motion between the first camera frame 310 and the second camera frame 312 spans two time intervals (e.g., from time t1 to time t2 and from time t2 to time t3), while the motion between the second camera frame 312 and the extrapolated camera frame spans from time t3 to time t4. Therefore, rather than adjusting the second camera frame 312 based on the entire motion between the first camera frame 310 and the second camera frame 312, the time warping process 326 may scale the motion calculated between the first camera frame 310 and the second camera frame 312 according to the time interval between the second camera frame 312 and the extrapolated camera frame.

[0097] For example, time warp process 326 may determine motion between first camera frame 310 and second camera frame 312. Time warp process 326 may then determine the magnitude of the motion at each time interval. In some examples, to determine the magnitude of the motion, time warp process 326 may divide the motion between first camera frame 310 and second camera frame 312 by the number of time intervals after first camera frame 310 until second camera frame 312 (e.g., the number of time intervals spanned by the motion). Figure 3A In the illustrative example shown, the motion between the first camera frame 310 and the second camera frame 312 spans a first time interval between time t1 and time t2 and a second time interval between time t2 and time t3. Therefore, to determine the magnitude of the motion between the first camera frame 310 and the second camera frame 312, the time warping process 326 may divide the motion between the first camera frame 310 and the second camera frame 312 by 2 (e.g., this takes into account the time interval between the first camera frame 310 and the second camera frame 312). The time warping process 326 may then use the calculated motion magnitude to generate an extrapolated camera frame. The motion magnitude may reflect the estimated amount of motion from time t3 to time t4.

[0098] In other words, the time warp process 326 may scale the motion between the second camera frame 312 and the extrapolated camera frame to match the ratio of motion between the first camera frame 310 and the second camera frame 312. To illustrate, the time warp process 326 may calculate a ratio between the amount of time that elapsed between the first camera frame 310 and the second camera frame 312 and the amount of time between the second camera frame 312 and the extrapolated camera frame, and use those ratios of time to intelligently adapt the magnitude of motion used to generate the extrapolated camera frame.

[0099] In some examples, time warp process 326 may generate an extrapolated camera frame by using the estimated motion magnitude to adjust second camera frame 312. For example, time warp process 326 may adjust second camera frame 312 by projecting the motion magnitude from time t3 associated with second camera frame 312 forward (e.g., into the future) to time t4 associated with the extrapolated camera frame.

[0100] In some cases, the time warp process 326 may calculate a base point for motion vectors for camera frame extrapolation. For example, if the time warp process 326 estimates motion vectors for the first camera frame 310 and the second camera frame 312 (e.g., from time t1 to time t3) using standard optical flow, the motion vectors may indicate that the time warp process 326 should apply motion from time t1 to time t3. However, for frame extrapolation, the time warp process 326 may instead need to apply motion from time t3 (associated with the second camera frame 312) forward. In other words, in the case of standard optical flow, the motion vectors may cause the time warp process 326 to move a region of the frame behind an object depicted in the frame, rather than moving the object depicted in the frame as expected. To avoid such problems, the time warp process 326 may calculate motion vectors backwards and invert the calculated motion vectors to obtain a base point for the forward-projected motion vector. For example, the time warp process 326 may estimate optical flow from time t1 to time t3, and invert the corresponding motion vectors to obtain a motion vector having a base point and projecting motion forward from time t3.

[0101] In some cases, the time warp process 326 may correct for rotational motion between the second camera frame 312 and the extrapolated camera frame, and use optical flow to correct for translational motion between the second camera frame 312 and the extrapolated camera frame. For example, the time warp process 326 may correct for rotational motion in the extrapolated camera frame. The time warp process 326 may reproject the extrapolated camera frame with translational motion but without rotational motion. For example, the time warp process 326 may subtract the rotational motion between the second camera frame 312 and the extrapolated camera frame from the estimated motion, and use the resulting translational motion to estimate the extrapolated camera frame. The time warp process 326 may correct for rotational motion in the extrapolated camera frame. In some examples, the time warp process 326 may use the pose 302, the pose 304, and / or the pose 306 to reproject the second camera frame 312 without rotational motion. Then, the time warp process 326 may use the reprojected second camera frame 312 to estimate the optical flow for the extrapolated camera frame, and may correct for rotational motion in the extrapolated camera frame.

[0102] In other examples, the time warp process 326 may warp the extrapolated camera frames to compensate for translational and rotational motion of the XR system 100, and use optical flow to compensate for translational motion of objects in the scene depicted by the extrapolated camera frames. The time warp process 326 may correct and / or project different types of motion, as previously explained. For example, the time warp process 326 may correct and / or project rotational movement (e.g., 3DOF movement) of the XR system 100. The time warp process 326 may correct for rotational movement by applying a rotational transformation to the extrapolated camera frames. In addition, the time warp process 326 may correct and / or project translational movement (e.g., 6DOF movement) of the XR system 100. The time warp process 326 may also correct and / or project object movement in the scene. Object movement may correspond to objects that are in motion in the scene and have changed position between camera frames. In some examples, motion estimation and compensation may cover rotational movement (e.g., 3DOF), translational movement (e.g., 6DOF), and object movement.

[0103] In some examples, the time warp process 326 may correct and / or estimate simple types of movement, such as rotational movement. In some cases, if those types of movement are removed prior to motion estimation, the motion estimation and frame extrapolation may be more accurate. For example, if the time warp process 326 is configured to estimate 3DOF rotational movement, then before applying motion estimation and frame extrapolation as described herein, the time warp process 326 may reproject the extrapolated camera frame without 3DOF movement (e.g., so that the 3DOF movement has been removed). Motion estimation and frame extrapolation focus on 6DOF movement and object movement. 3DOF correction may be applied by the time warp process 326 after the extrapolated camera frame has been generated. In some examples, in the case of 3DOF reprojection, the time warp process 326 may use a matrix transformation to align the extrapolated camera frame. In the case of 3DOF plus 6DOF reprojection, the time warp process 326 may use depth information to be applied to mesh rendering of mesh geometry to achieve 3DOF plus 6DOF transformation.

[0104] In some cases, for frame extrapolation, the time warping process 326 may texture map the second camera frame 312 to a mesh of geometric shapes. The time warping process 326 may move the positioning of vertices in the mesh based on the motion of the area of ​​the second camera frame 312 determined during motion estimation. This may shift and manipulate the area of ​​the second camera frame 312 according to its historical motion in order to move the object into the position of the object predicted by the previously described inverse backward motion estimation.

[0105] In some cases, the time warp process 326 may perform camera frame extrapolation by moving geometry or by moving UV texture coordinates. For example, the time warp process 326 may perform an extrapolated rendering process by moving the geometry of the mesh or manipulating the UV texture coordinates to change how the second camera frame 312 is mapped to the geometry. In some cases, the time warp process 326 may distort the mesh by moving the geometry of the mesh. In some examples, when moving geometry, the time warp process 326 may move each vertex during the vertex shader process based on the magnitude and direction of the motion vector at the corresponding point. In some examples, when moving texture coordinates, the time warp process 326 may determine an inverse of the movement and use the inverse to adjust the texture coordinates.

[0106] Once the time warp process 326 has completed generating / processing the extrapolated camera frame, the time warp process 326 may generate a new rendered frame 338 by combining the extrapolated camera frame with the virtual content from the eye buffer 336. In some cases, the time warp process 326 may also warp or reproject the new rendered frame 338 as previously described. Once the new rendered frame 338 is complete, the time warp process 326 may provide the new rendered frame 338 to the display 106 for presentation to the user.

[0107] Figure 3B 3 is a diagram illustrating an example flow 340 for extrapolating perspective camera frames to synchronize the frame rate of the perspective camera frames with the display refresh rate. In this example, the extrapolated camera frames may be generated by an application on XR system 100. The application may be separate from the time warp process (e.g., separate from time warp process 320, time warp process 322, time warp process 324, or time warp process 326). In some examples, the application on XR system 100 may include an XR application, a machine learning model, or any application on XR system 100.

[0108] Stream 340 may combine camera frames obtained from camera sensor 102 with virtual content rendered by XR system 100 to provide a mixed reality experience to a user wearing XR system 100. As previously explained, even if the user cannot see the scene around the user while wearing XR system 100 (e.g., because XR system 100 blocks the user's eyes and / or visibility of the scene), camera frames from camera sensor 102 may still capture the scene and may be used as a perspective camera frame to allow the user to see the scene (e.g., through a perspective camera frame). XR system 100 may render virtual content as well as the perspective camera frame to provide a mixed reality experience to the user, wherein the user may see virtual content rendered within a depiction of the scene from the perspective camera frame.

[0109] At time t1, the time warp process 320 receives the first camera frame 310 captured by the camera sensor 102 and the virtual content from the eye buffer 330, and generates a rendered frame for presentation (e.g., via the display 106) to the user of the XR system 100. The rendered frame may be a mixed reality frame generated from the first camera frame 310 captured by the camera sensor 102 and the virtual content from the eye buffer 330. To generate the rendered frame, the time warp process 320 may combine the first camera frame 310 with the virtual content from the eye buffer 330. The first camera frame 310 may depict a scene around the user wearing the XR system 100, and the virtual content may include rendered content generated by the XR system 100 (e.g., via an application of the XR system 100, such as an XR application). Thus, the rendered frame may depict the virtual content from the eye buffer 330 within a view of the scene depicted by the first camera frame 310.

[0110] When generating a rendered frame, the time warp process 320 may warp or reproject the rendered frame before transmitting the rendered frame to the display 106 for presentation to the user. The time warp process 320 may warp or reproject the rendered frame, as previously described with respect to Figure 3A described.

[0111] At time t2, the time warp process 322 may obtain the virtual content from the eye buffer 332 to generate a new rendered frame for presentation by the display 106 at the next display refresh time of the display 106. Since the display refresh rate of the display 106 is less than the camera frame rate of the camera sensor 102, the time warp process 322 cannot obtain a new camera frame from the camera sensor 102 at time t2. Therefore, the time warp process 322 may generate a new rendered frame based on the camera frame 310 obtained at time t1 and the virtual content from the eye buffer 332. In some examples, when generating the new rendered frame, the time warp process 322 may warp or reproject the new rendered frame to compensate for movement, as previously explained.

[0112] At time t3, camera sensor 102 may have second camera frame 312 ready for presentation by XR system 100. For example, the camera frame rate of camera sensor 102 may be such that camera sensor 102 may generate and provide a new camera frame every other time interval (e.g., at time t1 prior to time t3), where each time interval is based on or controlled by a display refresh rate of display 106. Thus, at time t3, time warp process 324 may obtain second camera frame 312 from camera sensor 102 and additional virtual content from eye buffer 334 to generate a new rendered frame for presentation by display 106 at the next display refresh time of display 106.

[0113] When generating a new rendered frame, the time warp process 324 may warp or reproject the new rendered frame before transmitting the new rendered frame to the display 106 for presentation to the user. The time warp process 324 may warp or reproject the new rendered frame, as previously described with respect to Figure 3A Explained.

[0114] At time t4, the time warp process 326 may obtain the additional virtual content from the eye buffer 336 to generate a new rendered frame 338 for presentation by the display 106 at the next display refresh time of the display 106. Since the display refresh rate of the display 106 is less than the camera frame rate of the camera sensor 102, the time warp process 326 cannot obtain a new camera frame from the camera sensor 102 at time t4. In order to synchronize the display refresh rate with the camera frame rate of the camera sensor 102, the application on the XR system 100 may extrapolate a new camera frame based on the optical flow between the first camera frame 310 and the second camera frame 312. For example, the application on the XR system 100 may calculate the motion 314 between the first camera frame 310 and the second camera frame 312, and use the motion 314 to generate the extrapolated camera frame 342. An application on the XR system 100 may provide the extrapolated camera frame 342 to the time warp process 326 , which may use the extrapolated camera frame 342 to generate a new rendered frame 338 based on the extrapolated camera frame 342 and virtual content from the eye buffer 336 .

[0115] The application on the XR system 100 may determine the motion 314 as previously described with respect to Figure 3A In some examples, the application on XR system 100 may generate an extrapolated camera frame 342 by adjusting the second camera frame 312 based on the optical flow between the first camera frame 310 and the second camera frame 312. In some cases, the application on XR system 100 may adjust the amplitude of the optical flow used to generate the extrapolated camera frame 342 based on the time interval between the first camera frame 310 and the second camera frame 312 and the time interval between the second camera frame 312 and the extrapolated camera frame 342, as previously described.

[0116] In some examples, the application on XR system 100 may generate extrapolated camera frame 342 by using the estimated motion magnitude to adjust second camera frame 312. For example, the application on XR system 100 may adjust second camera frame 312 by projecting the motion magnitude from time t3 associated with second camera frame 312 forward (e.g., into the future) to time t4 associated with the extrapolated camera frame.

[0117] In some cases, an application on XR system 100 may calculate a base point for motion vectors for camera frame extrapolation. For example, if an application on XR system 100 uses standard optical flow to estimate motion vectors for first camera frame 310 and second camera frame 312 (e.g., from time t1 to time t3), the motion vectors may indicate that the application of XR system 100 should apply motion from time t1 to time t3. However, for frame extrapolation, the application on XR system 100 may instead need to apply motion from time t3 (associated with second camera frame 312) forward. As previously noted, in the case of standard optical flow, the motion vectors may cause the application of XR system 100 to move a region of the frame behind an object depicted in the frame, rather than moving the object depicted in the frame as expected. To avoid such issues, the application of XR system 100 may calculate motion vectors backwards and invert the calculated motion vectors to obtain a base point for the forward-projected motion vectors. For example, an application of XR system 100 may estimate optical flow from time t1 to time t3 and invert the corresponding motion vector to obtain a motion vector having a base point and projecting motion forward from time t3.

[0118] In some cases, the application of XR system 100 may correct for translational motion between second camera frame 312 and extrapolated camera frame 342, and the time warp process 326 may correct for rotational motion between second camera frame 312 and extrapolated camera frame 342. In some examples, the application of XR system 100 may use optical flow to correct for translational motion between second camera frame 312 and extrapolated camera frame 342, and the time warp process 326 may perform warping or reprojection to correct for rotational motion. In other cases, the time warp process 326 may correct for translational motion between second camera frame 312 and extrapolated camera frame 342 and for rotational motion between second camera frame 312 and extrapolated camera frame 342.

[0119] In some examples, the application on XR system 100 may generate extrapolated camera frames 342 with translational motion but no rotational motion. In such examples, time warp processing 326 may correct for rotational motion in extrapolated camera frames 342. In other examples, time warp processing 326 may warp the extrapolated camera frames to compensate for the translational motion and rotational motion of XR system 100, and the application on XR system 100 may use optical flow to compensate for the translational motion of objects in the scene depicted by extrapolated camera frames 342.

[0120] Once the application on the XR system 100 has finished generating / processing the extrapolated camera frame 342, the timewarp process 326 may generate a new rendered frame 338 by combining the extrapolated camera frame 342 with the virtual content from the eye buffer 336. In some cases, the timewarp process 326 may also warp or reproject the new rendered frame 338 as previously described. Once the new rendered frame 338 is complete, the timewarp process 326 may provide the new rendered frame 338 to the display 106 for presentation to the user.

[0121] Figure 4A 404. FIG. 4 is a diagram illustrating an example of motion estimation 400 for extrapolating camera frames. In this example, motion is estimated from frame 402 to frame 404. However, motion estimation 400 shows that motion vectors reflecting motion from frame 402 to frame 404 are used directly (e.g., without adjustment, as described below with respect to FIG. 404). Figure 4B 404 ). As a result, the base point of the motion vector will be misaligned with respect to the positioning of the object depicted in frame 404 and / or the scene depicted in frame 404 .

[0122] For example, if motion estimation 400 uses standard optical flow to estimate motion vectors for motion from frame 402 to frame 404, the motion vectors may indicate that the motion of the extrapolated camera frame should be applied from frame 402 to frame 404, which may cause the frame region of the extrapolated camera frame to be moved behind the objects depicted in frames 402 and 404, rather than moving forward from frame 404 to the objects depicted in frames 402 and 404 as expected.

[0123] To avoid such problems, the motion vectors associated with frame 402 and frame 404 may be calculated backward and inverted to obtain a base point for the forward projected motion vector. For example, the optical flow may be calculated from frame 404 to frame 402 and inverted to obtain a motion vector having a base point for the forward projected motion from frame 404.

[0124] Figure 4B 4 is a diagram illustrating another example motion estimation 420 for extrapolating camera frames. In this example, the motion vectors are properly aligned for forward extrapolation from frame 404. For example, optical flow is calculated from frame 404 to frame 402 and inverted to obtain a motion vector with a base point for projecting motion forward from frame 404.

[0125] Therefore, in order to generate a motion vector with an appropriate base point and magnitude without requiring any additional post-processing steps, the motion vector may be calculated as the inverse of the motion vector calculated based on the motion from frame 404 to frame 402. Figure 4B As shown, the inverse backward motion vector is properly aligned for forward extrapolation from frame 404. Extrapolation of camera frames may then be performed from frame 404 forward based on the inverse backward motion vector.

[0126] Motion estimation 420 may cover any type of movement, such as, for example, 3DOF movement, 6DOF movement, movement of objects in the scene depicted in frames 402 and 404, etc. As previously explained, in some cases, the time warp process may cover 3DOF rotational movement. In some examples, before applying motion estimation and frame extrapolation, the input frames may be reprojected so that 3DOF movement of the XR system 100 is removed. In some cases, motion estimation and frame extrapolation may focus on 6DOF movement and object movement in the scene. In some examples, the time warp process may apply 3DOF motion correction after the camera frames have been extrapolated.

[0127] In some example cases of 3DOF reprojection, a matrix transformation may be used to align frame 402 and frame 404. In some example cases of 3DOF and 6DOF reprojection, XR system 100 may implement the transformation using mesh rendering where depth information is applied to the mesh geometry of frame 404.

[0128] In some examples, extrapolation of the camera frame may be performed using distorted mesh rendering. Additionally, the frame 404 may be texture mapped to a regular geometric structure grid. The positioning of the vertices may be moved based on the motion of the region of the frame 404 determined during motion estimation. This may shift and manipulate the region of the camera frame 404 according to its historical motion in order to move an object into the position of the object predicted by inverse backward motion estimation.

[0129] Figure 5 is a diagram illustrating an example calculation of motion vector magnitude for extrapolating a camera frame from a previous frame. In this example, vertical synchronization (Vsync) 510 is shown with respect to camera frame 502, Vsync 512 and Vsync 514 are shown with respect to camera frame 504, and Vsync 516 is shown with respect to extrapolated camera frame 506. Each Vsync (e.g., Vsync 510, Vsync 512, Vsync 514, Vsync 516) depicts synchronization of the frame rate of virtual content with the display refresh rate.

[0130] The frame extrapolation motion vector magnitude may be adjusted based on a time ratio 520 between camera frame 502 and camera frame 504 to determine a scaled vector magnitude 522 for generating an extrapolated camera frame 506 from camera frame 504. Scaled vector magnitude 522 may represent the motion vector and the time between a previous camera frame (e.g., camera frame 504) and an extrapolation target frame (e.g., extrapolated camera frame 506).

[0131] For example, when predicting extrapolated camera frame 506, XR system 100 may determine a magnitude of motion between camera frame 502 and camera frame 504 based on the motion between camera frame 502 and camera frame 504 and a time ratio 520 between camera frame 502 and camera frame 504. The magnitude of motion may be used to determine a scaled vector magnitude 522 from camera frame 504 and extrapolated camera frame 506. XR system 100 may use scaled vector magnitude 522 to predict extrapolated camera frame 506 based on camera frame 504.

[0132] Figure 6 6 is a flow chart illustrating an example process 600 for synchronizing the frame rate of perspective content from a camera sensor with the display refresh rate of a display. In some cases, process 600 can be performed by a mobile device such as a mobile device such as a Figure 1 At block 602, process 600 may include obtaining a first frame and a second frame captured by an image capture device (e.g., camera sensor 102). The first frame may be a frame captured before the second frame. In some cases, the first frame and the second frame may depict a scene associated with the computing device (e.g., XR system). The display refresh rate of the display is greater than the camera frame rate of the image capture device. In some cases, the image capture device is part of the computing device, such as an XR system (e.g., XR system 100). In some cases, the display is part of the computing device (e.g., display 106 of XR system 100).

[0133] In some aspects, process 600 may include determining that a display refresh rate of the display (e.g., display 106) is greater than the camera frame rate of the image capture device. For example, the computing device (e.g., XR system 100) may determine the camera frame rate of the image capture device and the display refresh rate of the display, and determine that the display refresh rate is greater than the camera frame rate.

[0134] At block 604, process 600 may include determining motion between the first frame and the second frame. In some cases, process 600 may include determining the motion between the first frame and the second frame based on the display refresh rate being greater than the camera frame rate. In some examples, determining the motion between the first frame and the second frame may include determining an optical flow between the first frame and the second frame. In some cases, determining the motion between the first frame and the second frame may further include determining a first pose of the computing device (e.g., XR system 100) at a first time when the first frame was captured and a second pose of the computing device (e.g., XR system 100) at a second time when the second frame was captured. In some examples, the motion between the first frame and the second frame may be based on the optical flow between the first frame and the second frame, the first pose, and the second pose.

[0135] At block 606, process 600 may include predicting a third frame corresponding to a time after the second frame based on the second frame and the motion between the first frame and the second frame. For example, the computing device (e.g., XR system 100) may adjust the second frame to reflect the motion between the first frame and the second frame. In some examples, predicting the third frame may include extrapolating the third frame from the second frame. In some cases, the third frame is extrapolated for a future time corresponding to the time after the second frame.

[0136] At block 608, process 600 may include displaying, at the display, content including the predicted third frame and virtual content rendered for the computing device (e.g., by XR system 100 or by at least one server in a split rendering architecture for XR system 100, where the server sends content to XR system 100 and XR system 100 receives the content from the server). The content may depict a scene associated with the computing device (e.g., XR system 100) (e.g., based on the predicted third frame) combined with the virtual content rendered for the computing device (e.g., XR system 100). In one illustrative example, the content is mixed reality content.

[0137] In some examples, determining the motion between the first frame and the second frame may include determining one or more motion vectors representing the motion from the second frame to the first frame, and determining one or more inverse motion vectors representing the inverse of the motion from the second frame to the first frame. In some cases, predicting the third frame may include projecting the movement of one or more regions of the second frame to the time after the second frame based on the one or more inverse motion vectors. In some examples, the time after the second frame includes a future time after a different time when the second frame was captured.

[0138] In some examples, the motion between the first frame and the second frame may include rotational motion and translational motion. In some aspects, process 600 may include subtracting the rotational motion from the motion between the first frame and the second frame, predicting the third frame based on the translational motion, and time warping the third frame based on the rotational motion.

[0139] In some cases, predicting the third frame may include determining a first amount of time between the first frame and the second frame, determining a second amount of time between the prediction of the third frame and an estimated display time for the third frame, determining a ratio of the first amount of time to the second amount of time, adjusting the amount of motion between the first frame and the second frame based on the ratio of the first amount of time to the second amount of time, and predicting the third frame based on the adjusted amount of motion between the first frame and the second frame. In some cases, predicting the third frame includes projecting the adjusted amount of motion from the second frame forward to the predicted third frame.

[0140] In some aspects, in order to display the content including the predicted third frame and the virtual content, process 600 may combine the predicted third frame with the virtual content into a single frame. For example, in some cases, process 600 may include combining the virtual content with the predicted third frame, and displaying the combined virtual content and the predicted third frame at the display. In some examples, the single frame including the combined predicted third frame and virtual content may include the content (e.g., mixed reality content). In some examples, displaying the content (e.g., mixed reality content) including the single frame (including the combined predicted third frame and virtual content) may include synchronizing the display of the content (e.g., mixed reality content) or the single frame (including the combined predicted third frame and virtual content) with the display refresh rate of the display. In some cases, synchronizing the display of the content (e.g., mixed reality content) or the single frame (including the combined predicted third frame and virtual content) with the display refresh rate of the display may include transmitting the content (e.g., mixed reality content) or the single frame (including the combined predicted third frame and virtual content) to the display at a specific time associated with the display refresh rate.

[0141] In some aspects, determining the motion between the first frame and the second frame may include determining a translational motion of one or more objects in the scene depicted by the first frame and the second frame, determining a translational motion of the computing device (e.g., XR system 100), and determining a rotational motion of the computing device (e.g., XR system 100). In some aspects, process 600 may include predicting the third frame based on the translational motion of the one or more objects in the scene depicted by the first frame and the second frame, and time-warping the third frame based on the translational motion of the computing device (e.g., XR system 100) and the rotational motion of the computing device (e.g., XR system 100).

[0142] In some examples, the processes described herein (e.g., process 600 and / or any other process described herein) can be performed by a computing device or apparatus. In one example, process 600 can be performed by Figure 1 In another example, process 600 may be performed by an XR system 100 having Figure 7 The computing system 700 shown in the figure is implemented by a computing system having a computing device architecture. Figure 7 The computing device of the computing device architecture of the computing system 700 shown may implement Figure 6 The operation and / or this article about Figures 1 to 6 The components and / or operations described in any of the Figures.

[0143] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, an AR glasses, a networked watch or smart watch, or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a laptop, a smart TV, a camera, and / or any other computing device having resource capabilities to perform the processes described herein (including process 600 and / or other processes described herein). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0144] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0145] Process 600 is illustrated as a logic flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0146] In addition, process 600 and / or other processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, through hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0147] Figure 7 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Figure 7 An example of a computing system 700 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using a connection 705. The connection 705 may be a physical connection using a bus, or a direct connection into the processor 710, such as in a chipset architecture. The connection 705 may also be a virtual connection, a networked connection, or a logical connection.

[0148] In some embodiments, computing system 700 is a distributed system, where the functionality described in the present disclosure may be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a number of such components that each perform some or all of the functionality for which the component is described. In some embodiments, a component may be a physical device or a virtual device.

[0149] The example system 700 includes at least one processing unit (CPU or processor) 710 and connections 705 that couple various system components including system memory 715, such as read only memory (ROM) 720 and random access memory (RAM) 725, to the processor 710. The computing system 700 may include a cache 712 of high-speed memory directly connected to the processor 710, in close proximity to the processor, or integrated as part of the processor.

[0150] Processor 710 may include any general purpose processor and hardware services or software services stored in storage device 730 that are configured to control processor 710 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 710 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0151] To enable user interaction, the computing system 700 includes an input device 745 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 700 may also include an output device 735 that can be one or more of a plurality of output mechanisms. In some instances, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 700. The computing system 700 may include a communication interface 740, which may generally govern and manage user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using a wired and / or wireless transceiver, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Wireless signal transfer, Low Energy (BLE) wireless signal transfer, Wireless signal transfer, radio frequency identification (RFID) wireless signal transfer, near field communication (NFC) wireless signal transfer, dedicated short range communication (DSRC) wireless signal transfer, 702.11 Wi-Fi wireless signal transfer, wireless local area network (WLAN) signal transfer, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transfer, public switched telephone network (PSTN) signal transfer, integrated services digital network (ISDN) signal transfer, 3G / 4G / 5G / LTE cellular data network wireless signal transfer, ad hoc network signal transfer, radio wave signal transfer, microwave signal transfer, infrared signal transfer, visible light signal transfer, ultraviolet light signal transfer, wireless signal transfer along the electromagnetic spectrum, or some combination thereof. The communication interface 740 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 700 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the base features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0152] The storage device 730 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a magnetic cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory ... card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0153] Storage device 730 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 710, causes the system to perform functions. In some embodiments, hardware services that perform specific functions may include software components for performing functions stored in a computer-readable medium connected to necessary hardware components such as processor 710, connection 705, output device 735, etc. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer readable media may store thereon code and / or machine executable instructions, which may represent a procedure, function, subprogram, program, routine, subroutine, module, software package, category, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0154] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals including bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0155] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be appreciated by those skilled in the art that embodiments may be practiced without these specific details. For clarity of explanation, in some instances, the present technology may be presented as including a separate functional block, which includes a device, device assembly, step or routine in a method embodied in a combination of software or hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid obscuring these embodiments in unnecessary details. In other cases, known circuits, processes, algorithms, structures and techniques may be shown without necessary details to avoid confusing each embodiment.

[0156] Individual embodiments may be described above as processes or methods depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe an operation as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or main function.

[0157] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from computer readable media. Such instructions may include, for example, instructions and data that configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or function group. Parts of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary, intermediate format instructions (such as assembly language), firmware, source code. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memories, USB devices with non-volatile memory, networked storage devices, etc.

[0158] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0159] The instructions, the media for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0160] In the foregoing description, various aspects of the present application are described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary embodiments of the present application have been described in detail herein, it is to be understood that the inventive concept can be embodied and adopted in various other ways, and the appended claims are intended to be interpreted as including such variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, without departing from the broader essence and scope of this specification, the embodiments can be used in any number of environments and applications beyond the environments and applications described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that in an alternative embodiment, each method can be performed in a different order than described.

[0161] It should be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0162] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0163] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0164] Claim language or other language in this disclosure that states "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language that states "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language that states "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language that states "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0165] Various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0166] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices, mobile phones, and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a computer-readable data storage medium including a program code, which includes instructions for executing one or more of the methods, algorithms, and / or operations described above when executed. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0167] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0168] Exemplary embodiments of the present disclosure include:

[0169] Aspect 1. A device for rendering content, the device comprising: a memory; and one or more processors, the one or more processors coupled to the memory, the one or more processors configured to: obtain a first frame and a second frame captured by an image capture device, the first frame being captured before the second frame, wherein a display refresh rate of a display is greater than a camera frame rate of the image capture device; determine a motion between the first frame and the second frame; predict a third frame corresponding to a time after the second frame based on the second frame and the motion between the first frame and the second frame; and display, at the display, content including the predicted third frame and virtual content rendered for the device.

[0170] Aspect 2. An apparatus according to Aspect 1, wherein, in order to determine the motion between the first frame and the second frame, the one or more processors are configured to: determine one or more motion vectors representing the motion from the second frame to the first frame; and determine one or more inverse motion vectors representing the inverse of the motion from the second frame to the first frame.

[0171] Aspect 3. An apparatus according to Aspect 2, wherein, in order to predict the third frame, the one or more processors are configured to: project the movement of one or more regions of the second frame to the time after the second frame based on the one or more inverse motion vectors, wherein the time after the second frame includes future times after different times when the second frame is captured.

[0172] Aspect 4. The apparatus according to any one of aspects 1 to 3, wherein the motion between the first frame and the second frame comprises rotational motion and translational motion.

[0173] Aspect 5. An apparatus according to Aspect 4, wherein the one or more processors are configured to: subtract the rotational motion from the motion between the first frame and the second frame; predict the third frame based on the translational motion; and time-warp the third frame based on the rotational motion.

[0174] Aspect 6. An apparatus according to any one of Aspects 1 to 5, wherein, in order to predict the third frame, the one or more processors are configured to: determine a first amount of time between the first frame and the second frame; determine a second amount of time between the prediction of the third frame and an estimated display time for the third frame; determine a ratio of the first amount of time to the second amount of time; adjust the amount of motion between the first frame and the second frame based on the ratio of the first amount of time to the second amount of time; and predict the third frame based on the adjusted amount of motion between the first frame and the second frame.

[0175] Aspect 7. The apparatus according to aspect 6, wherein, to predict the third frame, the one or more processors are configured to forward project the adjusted motion from the second frame to the predicted third frame.

[0176] Aspect 8. An apparatus according to any one of Aspects 1 to 7, wherein, in order to display the content including the predicted third frame and the virtual content, the one or more processors are configured to: combine the predicted third frame and the virtual content into a single frame.

[0177] Aspect 9. The apparatus according to aspect 8, wherein the one or more processors are configured to: synchronize the display of the content with the display refresh rate of the display.

[0178] Aspect 10. An apparatus according to Aspect 9, wherein, in order to synchronize the display of the content with the display refresh rate of the display, the one or more processors are configured to transmit the content to the display at a specific time associated with the display refresh rate.

[0179] Aspect 11. An apparatus according to any one of Aspects 1 to 10, wherein, in order to determine the motion between the first frame and the second frame, the one or more processors are configured to: determine the translational motion of one or more objects in the scene depicted by the first frame and the second frame; determine the translational motion of the apparatus; and determine the rotational motion of the apparatus.

[0180] Aspect 12. An apparatus according to Aspect 11, wherein the one or more processors are configured to: predict the third frame based on the translational motion of the one or more objects in the scene depicted by the first frame and the second frame; and time-warp the third frame based on the translational motion of the apparatus and the rotational motion of the apparatus.

[0181] Aspect 13. An apparatus according to any one of Aspects 1 to 12, wherein the one or more processors are configured to: determine that the display refresh rate of the display is greater than the camera frame rate of the image capture device.

[0182] Aspect 14. An apparatus according to any one of Aspects 1 to 13, wherein the content comprises mixed reality content.

[0183] Aspect 15. An apparatus according to any one of aspects 1 to 14, wherein the apparatus comprises a mobile device comprising the image capture device and the display.

[0184] Aspect 16. An apparatus according to any one of Aspects 1 to 15, wherein the mobile device comprises an extended reality device.

[0185] Aspect 17. The apparatus according to any one of aspects 1 to 16, wherein the one or more processors are configured to: render the virtual content.

[0186] Aspect 18. The apparatus according to any one of aspects 1 to 16, wherein the one or more processors are configured to: receive the virtual content rendered by a server.

[0187] Aspect 19. A method for rendering content, the method comprising: obtaining a first frame and a second frame captured by an image capture device of an electronic device, the first frame being captured before the second frame, wherein a display refresh rate of a display is greater than a camera frame rate of the image capture device; determining a motion between the first frame and the second frame; predicting a third frame corresponding to a time after the second frame based on the second frame and the motion between the first frame and the second frame; and displaying content including the predicted third frame and virtual content rendered for the electronic device at the display of the electronic device.

[0188] Aspect 20. A method according to Aspect 19, wherein determining the motion between the first frame and the second frame includes: determining one or more motion vectors representing the motion from the second frame to the first frame; and determining one or more inverse motion vectors representing the inverse of the motion from the second frame to the first frame.

[0189] Aspect 21. A method according to Aspect 20, wherein predicting the third frame includes: projecting the movement of one or more regions of the second frame to the time after the second frame based on the one or more inverse motion vectors, wherein the time after the second frame includes a future time after a different time when the second frame was captured.

[0190] Aspect 22. The method according to any one of aspects 19 to 21, wherein the motion between the first frame and the second frame comprises rotational motion and translational motion.

[0191] Aspect 23. The method according to aspect 22, the method further comprising: subtracting the rotational motion from the motion between the first frame and the second frame; predicting the third frame based on the translational motion; and time-warping the third frame based on the rotational motion.

[0192] Aspect 24. A method according to any one of Aspects 19 to 23, wherein predicting the third frame includes: determining a first amount of time between the first frame and the second frame; determining a second amount of time between the prediction of the third frame and an estimated display time for the third frame; determining a ratio of the first amount of time to the second amount of time; adjusting the amount of motion between the first frame and the second frame based on the ratio of the first amount of time to the second amount of time; and predicting the third frame based on the adjusted amount of motion between the first frame and the second frame.

[0193] Clause 25. The method of clause 24, wherein predicting the third frame comprises forward projecting the adjusted amount of motion from the second frame to the predicted third frame.

[0194] Aspect 26. The method according to any one of aspects 19 to 25, wherein displaying the content including the predicted third frame and the virtual content includes combining the predicted third frame and the virtual content into a single frame.

[0195] Aspect 27. The method according to aspect 26, further comprising: synchronizing the display of the content with the display refresh rate of the display.

[0196] Aspect 28. The method of aspect 27, wherein synchronizing the display of the content with the display refresh rate of the display comprises transmitting the content to the display at a specific time associated with the display refresh rate.

[0197] Aspect 29. A method according to any one of Aspects 19 to 28, wherein determining the motion between the first frame and the second frame includes: determining the translational motion of one or more objects in the scene depicted by the first frame and the second frame; determining the translational motion of the electronic device; and determining the rotational motion of the electronic device.

[0198] Aspect 30. The method according to Aspect 29 further includes: predicting the third frame based on the translational motion of the one or more objects in the scene depicted by the first frame and the second frame; and time-warping the third frame based on the translational motion of the electronic device and the rotational motion of the electronic device.

[0199] Aspect 31. The method according to any one of aspects 19 to 30, the method further comprising: determining that the display refresh rate of the display is greater than the camera frame rate of the image capture device.

[0200] Aspect 32. A method according to any one of Aspects 19 to 31, wherein the content includes mixed reality content.

[0201] Aspect 33. A method according to any one of Aspects 19 to 32, wherein the electronic device comprises a mobile device, the mobile device comprising the image capture device and the display.

[0202] Aspect 34. The method according to aspect 33, wherein the mobile device comprises an extended reality device.

[0203] Aspect 35. The method according to any one of aspects 19 to 34, further comprising: rendering the virtual content.

[0204] Aspect 36. The method according to any one of Aspects 19 to 34, further comprising: receiving the virtual content rendered by a server.

[0205] Aspect 37. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform a method according to any one of aspects 19 to 36.

[0206] Aspect 38. An apparatus comprising means for performing the method according to any one of aspects 19 to 36.

Claims

1. A device for rendering content, the device comprising: Memory; and one or more processors, the one or more processors coupled to the memory, the one or more processors configured to: Obtaining a first frame and a second frame captured by an image capture device, the first frame being captured before the second frame, wherein a display refresh rate of a display is greater than a camera frame rate of the image capture device; determining motion between the first frame and the second frame; predicting a third frame corresponding to a time after the second frame based on the second frame and the motion between the first frame and the second frame; as well as Content including the predicted third frame and virtual content rendered for the device is displayed at the display.

2. The device according to claim 1, wherein: To determine the motion between the first frame and the second frame, the one or more processors are configured to: determining one or more motion vectors representing motion from the second frame to the first frame; and One or more inverse motion vectors representing the inverse of the motion from the second frame to the first frame are determined.

3. The device according to claim 2, wherein: To predict the third frame, the one or more processors are configured to: The movement of one or more regions of the second frame is projected to the time after the second frame based on the one or more inverse motion vectors, wherein the time after the second frame includes a future time after a different time when the second frame was captured. The apparatus of claim 1 , wherein the motion between the first frame and the second frame comprises rotational motion and translational motion.

5. The apparatus of claim 4, wherein the one or more processors are configured to: subtracting the rotational motion from the motion between the first frame and the second frame; predicting the third frame based on the translational motion; and The third frame is time warped based on the rotational motion.

6. The device according to claim 1, wherein: To predict the third frame, the one or more processors are configured to: determining a first amount of time between the first frame and the second frame; determining a second amount of time between the prediction of the third frame and an estimated display time for the third frame; determining a ratio of the first amount of time to the second amount of time; adjusting the amount of motion between the first frame and the second frame based on the ratio of the first amount of time to the second amount of time; as well as The third frame is predicted based on the adjusted amount of motion between the first frame and the second frame.

7. The device according to claim 6, wherein: To predict the third frame, the one or more processors are configured to forward project the adjusted amount of motion from the second frame to the predicted third frame.

8. The device according to claim 1, wherein: To display the content including the predicted third frame and the virtual content, the one or more processors are further configured to: The predicted third frame is combined with the virtual content into a single frame.

9. The apparatus of claim 8, wherein the one or more processors are configured to: Display of the content is synchronized with the display refresh rate of the display.

10. The device according to claim 9, wherein: To synchronize the display of the content with the display refresh rate of the display, the one or more processors are configured to: The content is transferred to the display at specific times associated with the display refresh rate.

11. The device according to claim 1, wherein: To determine motion between the first frame and the second frame, the one or more processors are configured to: determining translational motion of one or more objects in a scene depicted by the first frame and the second frame; determining translational motion of the device; and The rotational movement of the device is determined.

12. The apparatus of claim 11, wherein the one or more processors are configured to: predicting the third frame based on the translational motion of the one or more objects in the scene depicted by the first frame and the second frame; and The third frame is time warped based on the translational motion of the device and the rotational motion of the device.

13. The apparatus of claim 1, wherein the one or more processors are configured to: The display refresh rate of the display is determined to be greater than the camera frame rate of the image capture device.

14. The apparatus of claim 1, wherein the content comprises mixed reality content.

15. The apparatus of claim 1, wherein the apparatus comprises a mobile device comprising the image capture device and the display. The apparatus of claim 15 , wherein the mobile device comprises an extended reality device.

17. The apparatus of claim 1, wherein the one or more processors are configured to: The virtual content is rendered.

18. The apparatus of claim 1, wherein the one or more processors are configured to: The virtual content rendered by the server is received.

19. A method for rendering content, the method comprising: Obtaining a first frame and a second frame captured by an image capture device of an electronic device, the first frame being captured before the second frame, wherein a display refresh rate of a display is greater than a camera frame rate of the image capture device; determining motion between the first frame and the second frame; predicting a third frame corresponding to a time after the second frame based on the second frame and the motion between the first frame and the second frame; as well as Content including the predicted third frame and virtual content rendered for the electronic device is displayed at the display of the electronic device.

20. The method of claim 19, wherein determining the motion between the first frame and the second frame comprises: determining one or more motion vectors representing motion from the second frame to the first frame; as well as One or more inverse motion vectors representing the inverse of the motion from the second frame to the first frame are determined.

21. The method of claim 20, wherein predicting the third frame comprises: The movement of one or more regions of the second frame is projected to the time after the second frame based on the one or more inverse motion vectors, wherein the time after the second frame includes a future time after a different time when the second frame was captured.

22. The method of claim 19, wherein the motion between the first frame and the second frame comprises rotational motion and translational motion.

23. The method according to claim 22, further comprising: subtracting the rotational motion from the motion between the first frame and the second frame; predicting the third frame based on the translational motion; as well as The third frame is time warped based on the rotational motion.

24. The method of claim 19, wherein predicting the third frame comprises: determining a first amount of time between the first frame and the second frame; determining a second amount of time between the prediction of the third frame and an estimated display time for the third frame; determining a ratio of the first amount of time to the second amount of time; adjusting the amount of motion between the first frame and the second frame based on the ratio of the first amount of time to the second amount of time; as well as The third frame is predicted based on the adjusted amount of motion between the first frame and the second frame.

25. The method of claim 24, wherein predicting the third frame comprises forward projecting the adjusted amount of motion from the second frame to the predicted third frame. 26 . The method of claim 25 , wherein displaying the content including the predicted third frame and the virtual content includes combining the predicted third frame with the virtual content into a single frame.

27. The method according to claim 26, further comprising: Display of the content is synchronized with the display refresh rate of the display.

28. The method of claim 27, wherein synchronizing the display of the content with the display refresh rate of the display comprises transmitting the content to the display at a specific time associated with the display refresh rate.

29. The method of claim 19, wherein determining motion between the first frame and the second frame comprises: determining translational motion of one or more objects in a scene depicted by the first frame and the second frame; determining a translational motion of the electronic device; as well as Rotational movement of the electronic device is determined.

30. The method according to claim 29, further comprising: predicting the third frame based on the translational motion of the one or more objects in the scene depicted by the first frame and the second frame; as well as The third frame is time-warped based on the translational motion of the electronic device and the rotational motion of the electronic device.

Images