Method and apparatus for vertex pose adjustment with passthrough and time-warp transformations for video see-through (VST) extended reality (XR)

Abstract

A method includes determining at an extended reality (XR) device, a first set of vertex adjustment values of a distortion mesh and receiving image frame data of a scene captured at a first time and at a first head pose using a see-through camera of the XR device. The method further includes applying the first set of vertex adjustment values of the distortion mesh to the image frame data to obtain intermediate image data, and predicting a second head pose at a second time subsequent to the first time. The method also includes generating, based on the predicted second head pose, a second set of vertex adjustment values of the distortion mesh, applying the second set of vertex adjustment values of the distortion mesh to the intermediate image data to generate a rendered virtual frame and displaying the rendered virtual frame by the XR device at the second time.

Description

METHOD AND APPARATUS FOR VERTEX POSE ADJUSTMENT WITH PASSTHROUGH AND TIME-WARP TRANSFORMATIONS FOR VIDEO SEE-THROUGH (VST) EXTENDED REALITY (XR)

[0001] This disclosure relates generally to extended reality (XR) systems and processes. More specifically, this disclosure relates to vertex pose adjustment with passthrough and time-warp transformations in video see-through (VST) XR.

[0002] Extended reality (XR) systems are becoming more and more popular over time, and numerous applications have been and are being developed for XR systems. Some XR systems (such as augmented reality or "AR" systems and mixed reality or "MR" systems) can enhance a user's view of his or her current environment by overlaying digital content (such as information or virtual objects) over the user's view of the current environment. For example, some XR systems can often seamlessly blend virtual objects generated by computer graphics with real-world scenes.

[0003] This disclosure relates to vertex pose adjustment with passthrough and time-warp transformations in video see-through (VST) XR.

[0004] In a first embodiment, a method includes determining, using at least one processing device of an extended reality (XR) device, a first set of vertex adjustment values of a distortion mesh and receiving, using the at least one processing device, image frame data of a scene captured at a first time and at a first head pose using a see-through camera of the XR device. The method further includes applying, using the at least one processing device, the first set of vertex adjustment values of the distortion mesh to the image frame data to obtain intermediate image data, and predicting, using the at least one processing device, a second head pose at a second time subsequent to the first time. The method also includes generating, using the at least one processing device, based on the predicted second head pose, a second set of vertex adjustment values of the distortion mesh, applying, using the at least one processing device, the second set of vertex adjustment values of the distortion mesh to the intermediate image data to generate a rendered virtual frame and displaying the rendered virtual frame by the XR device at the second time, the rendered virtual frame comprising a corrected view of the scene.

[0005] According to embodiments in the present disclosure, image data from a see-through camera lens can be rendered to match image data from a normal lens situated at a user's eye. Correction of the see-through camera image data can be implemented in a way that the corrected image is projected from a viewpoint that corresponds to an XR device wearer's understanding of their viewpoint at the time.

[0006] For a more complete understanding of this disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, in which:

[0007] FIGURE 1 illustrates an exemplary network configuration including an electronic device in accordance with this disclosure;

[0008] FIGURE 2 illustrates an example of applying vertex adjustment values to a distortion map to correct image data;

[0009] FIGURE 3 illustrates an exemplary pipeline for vertex pose adjustment with passthrough and time-warp transformations in video see-through (VST) XR in accordance with this disclosure;

[0010] FIGURE 4 illustrates an example of vertex pose adjustment and time warp transformation in accordance with this disclosure;

[0011] FIGURE 5 illustrates an exemplary pipeline for vertex pose adjustment with passthrough and time-warp transformations in video see-through (VST) XR in accordance with this disclosure;

[0012] FIGURE 6 illustrates an exemplary pipeline for vertex pose adjustment with passthrough and time-warp transformations in video see-through (VST) XR in accordance with this disclosure; and

[0013] FIGURE 7 illustrates an exemplary XR device for vertex pose adjustment with passthrough and time-warp transformations in VST XR in accordance with this disclosure.

[0014] FIGURE 8 illustrates an exemplary method for vertex pose adjustment with passthrough and time-warp transformations in VST XR in accordance with this disclosure.

[0015] In a first embodiment, a method includes determining, using at least one processing device of an extended reality (XR) device, a first set of vertex adjustment values of a distortion mesh and receiving, using the at least one processing device, image frame data of a scene captured at a first time and at a first head pose using a see-through camera of the XR device. The method further includes applying, using the at least one processing device, the first set of vertex adjustment values of the distortion mesh to the image frame data to obtain intermediate image data, and predicting, using the at least one processing device, a second head pose at a second time subsequent to the first time. The method also includes generating, using the at least one processing device, based on the predicted second head pose, a second set of vertex adjustment values of the distortion mesh, applying, using the at least one processing device, the second set of vertex adjustment values of the distortion mesh to the intermediate image data to generate a rendered virtual frame and displaying the rendered virtual frame by the XR device at the second time, the rendered virtual frame comprising a corrected view of the scene.

[0016] In a second embodiment, an XR device includes at least one display, a see-through camera and at least one processing device. The at least one processing device is configured to determine a first set of vertex adjustment values of a distortion mesh, receive image frame data of a scene captured at a first time and at a first head pose by the see-through camera and apply the first set of vertex adjustment values of the distortion mesh to the image frame data to obtain intermediate image data. The at least one processing device is further configured to predict, a second head pose at a second time subsequent to the first time, generate, based on the predicted second head pose, a second set of vertex adjustment values of the distortion mesh, apply the second set of vertex adjustment values of the distortion mesh to the intermediate image data to generate a rendered virtual frame, and display at the at least one display, the rendered virtual frame by the XR device at the second time, the rendered virtual frame comprising a corrected view of the scene.

[0017] In a third embodiment, a non-transitory machine readable medium contains instructions that when executed cause at least one processor to determine a first set of vertex adjustment values of a distortion mesh, receive, from a see-through camera of an XR device, image frame data of a scene captured at a first time and at a first head pose by the see-through camera, and apply the first set of vertex adjustment values of the distortion mesh to the image frame data to obtain intermediate image data. When executed, the instructions further cause the at least one processing device to predict, a second head pose at a second time subsequent to the first time generate, based on the predicted second head pose, a second set of vertex adjustment values of the distortion mesh, apply the second set of vertex adjustment values of the distortion mesh to the intermediate image data to generate a rendered virtual frame, and display, at least one display of the XR device, the rendered virtual frame by the XR device at the second time, the rendered virtual frame comprising a corrected view of the scene.

[0018] Other technical features may be readily apparent to one skilled in the art from the following figures, descriptions, and claims.

[0019] Before undertaking the present disclosure below, it may be advantageous to set forth definitions of certain words and phrases used throughout this patent document. The terms "transmit," "receive," and "communicate," as well as derivatives thereof, encompass both direct and indirect communication. The terms "include" and "comprise," as well as derivatives thereof, mean inclusion without limitation. The term "or" is inclusive, meaning and / or. The phrase "associated with," as well as derivatives thereof, means to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, have a relationship to or with, or the like.

[0020] Moreover, various functions described below can be implemented or supported by one or more computer programs, each of which is formed from computer readable program code and embodied in a computer readable medium. The terms "application" and "program" refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, related data, or a portion thereof adapted for implementation in a suitable computer readable program code. The phrase "computer readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer readable medium" includes any type of medium capable of being accessed by a computer, such as read only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory. A "non-transitory" computer readable medium excludes wired, wireless, optical, or other communication links that transport transitory electrical or other signals. A non-transitory computer readable medium includes media where data can be permanently stored and media where data can be stored and later overwritten, such as a rewritable optical disc or an erasable memory device.

[0021] As used here, terms and phrases such as "have," "may have," "include," or "may include" a feature (like a number, function, operation, or component such as a part) indicate the existence of the feature and do not exclude the existence of other features. Also, as used here, the phrases "A or B," "at least one of A and / or B," or "one or more of A and / or B" may include all possible combinations of A and B. For example, "A or B," "at least one of A and B," and "at least one of A or B" may indicate all of (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B. Further, as used here, the terms "first" and "second" may modify various components regardless of importance and do not limit the components. These terms are only used to distinguish one component from another. For example, a first user device and a second user device may indicate different user devices from each other, regardless of the order or importance of the devices. A first component may be denoted a second component and vice versa without departing from the scope of this disclosure.

[0022] It will be understood that, when an element (such as a first element) is referred to as being (operatively or communicatively) "coupled with / to" or "connected with / to" another element (such as a second element), it can be coupled or connected with / to the other element directly or via a third element. In contrast, it will be understood that, when an element (such as a first element) is referred to as being "directly coupled with / to" or "directly connected with / to" another element (such as a second element), no other element (such as a third element) intervenes between the element and the other element.

[0023] As used here, the phrase "configured (or set) to" may be interchangeably used with the phrases "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of" depending on the circumstances. The phrase "configured (or set) to" does not essentially mean "specifically designed in hardware to." Rather, the phrase "configured to" may mean that a device can perform an operation together with another device or parts. For example, the phrase "processor configured (or set) to perform A, B, and C" may mean a generic-purpose processor (such as a CPU or application processor) that may perform the operations by executing one or more software programs stored in a memory device or a dedicated processor (such as an embedded processor) for performing the operations.

[0024] The terms and phrases as used here are provided merely to describe some embodiments of this disclosure but not to limit the scope of other embodiments of this disclosure. It is to be understood that the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. All terms and phrases, including technical and scientific terms and phrases, used here have the same meanings as commonly understood by one of ordinary skill in the art to which the embodiments of this disclosure belong. It will be further understood that terms and phrases, such as those defined in commonly-used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined here. In some cases, the terms and phrases defined here may be interpreted to exclude embodiments of this disclosure.

[0025] Examples of an "electronic device" according to embodiments of this disclosure may include at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device (such as smart glasses, a head-mounted device (HMD), electronic clothes, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, a smart mirror, or a smart watch). Other examples of an electronic device include a smart home appliance. Examples of the smart home appliance may include at least one of a television, a digital video disc (DVD) player, an audio player, a refrigerator, an air conditioner, a cleaner, an oven, a microwave oven, a washer, a dryer, an air cleaner, a set-top box, a home automation control panel, a security control panel, a TV box, a smart speaker or speaker with an integrated digital assistant, a gaming console, an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame. Still other examples of an electronic device include at least one of various medical devices (such as diverse portable medical measuring devices (like a blood sugar measuring device, a heartbeat measuring device, or a body temperature measuring device), a magnetic resource angiography (MRA) device, a magnetic resource imaging (MRI) device, a computed tomography (CT) device, an imaging device, or an ultrasonic device), a navigation device, a global positioning system (GPS) receiver, an event data recorder (EDR), a flight data recorder (FDR), an automotive infotainment device, a sailing electronic device (such as a sailing navigation device or a gyro compass), avionics, security devices, vehicular head units, industrial or home robots, automatic teller machines (ATMs), point of sales (POS) devices, or Internet of Things (IoT) devices (such as a bulb, various sensors, electric or gas meter, sprinkler, fire alarm, thermostat, street light, toaster, fitness equipment, hot water tank, heater, or boiler). Other examples of an electronic device include at least one part of a piece of furniture or building / structure, an electronic board, an electronic signature receiving device, a projector, or various measurement devices (such as devices for measuring water, electricity, gas, or electromagnetic waves). Note that, according to various embodiments of this disclosure, an electronic device may be one or a combination of the above-listed devices. According to some embodiments of this disclosure, the electronic device may be a flexible electronic device. The electronic device disclosed here is not limited to the above-listed devices and may include any other electronic devices now known or later developed.

[0026] In the following description, electronic devices are described with reference to the accompanying drawings, according to various embodiments of this disclosure. As used here, the term "user" may denote a human or another device (such as an artificial intelligent electronic device) using the electronic device.

[0027] Definitions for other certain words and phrases may be provided throughout this patent document. Those of ordinary skill in the art should understand that in many if not most instances, such definitions apply to prior as well as future uses of such defined words and phrases.

[0028] None of the description in this application should be read as implying that any particular element, step, or function is an essential element that must be included in the claim scope. The scope of patented subject matter is defined only by the claims. Use of any other term, including without limitation "mechanism," "module," "device," "unit," "component," "element," "member," "apparatus," "machine," "system," "processor," or "controller," within a claim is understood by the Applicant to refer to structures known to those skilled in the relevant art.

[0029] FIGURES 1 through 7, discussed below, and the various embodiments of this disclosure are described with reference to the accompanying drawings. However, it should be appreciated that this disclosure is not limited to these embodiments, and all changes and / or equivalents or replacements thereto also belong to the scope of this disclosure. The same or similar reference denotations may be used to refer to the same or similar elements throughout the specification and the drawings.

[0030] As noted above, extended reality (XR) systems are becoming more and more popular over time, and numerous applications have been and are being developed for XR systems. Some XR systems (such as augmented reality or "AR" systems and mixed reality or "MR" systems) can enhance a user's view of his or her current environment by overlaying digital content (such as information or virtual objects) over the user's view of the current environment. For example, some XR systems can often seamlessly blend virtual objects generated by computer graphics with real-world scenes.

[0031] Optical see-through (OST) XR systems refer to XR systems in which users directly view real-world scenes through head-mounted devices (HMDs). Unfortunately, OST XR systems face many challenges that can limit their adoption. Some of these challenges include limited fields of view, limited usage spaces (such as indoor-only usage), failure to display fully-opaque black objects, and usage of complicated optical pipelines that may require projectors, waveguides, and other optical elements. In contrast to OST XR systems, video see-through (VST) XR systems (also called "passthrough" XR systems) present users with generated video sequences of real-world scenes. VST XR systems can be built using virtual reality (VR) technologies and can have various advantages over OST XR systems. For example, VST XR systems can provide wider fields of view and can provide improved contextual augmented reality.

[0032] Viewpoint correction, also known as viewpoint matching, is often a useful or important operation in VST XR pipelines. Viewpoint matching typically refers to a process for creating video frames that are presented at a user's eye viewpoint locations using video frames captured at see-through camera viewpoint locations, which allows the user to feel as if the see-through cameras are positioned at the user's eye viewpoint locations, rather. Among other things, viewpoint matching can involve depth-based reprojection in which objects within a scene are reprojected into virtual views based on the objects' depths within the scene. However, depth-based reprojection may require large amounts of computational resources (such as processing and memory resources) in order to reconstruct depths and perform depth reprojection, which can become particularly problematic at higher video resolutions (such as 4K resolutions and above). Moreover, depth-based reprojection may create latencies in VST XR pipelines, consume limited battery resources, generate excess heat at a head-worn device, or can cause noticeable delays or other issues (for example, motion sickness) for users.

[0033] In many cases, viewpoint matching is one of a plurality of static corrections which need to be implemented by a rendering pipeline in order to provide a satisfactory VST XR experience. As used in this disclosure, the expression "static corrections" encompasses corrections of differences between how a scene appears through a see-through camera, and a ground truth view (for example, through a human eye or a normal lens) of the same scene which are not dependent on dynamic factors (for example, the color mix of the objects in the video frame of the scene, or pose changes of the viewer). Further examples of static corrections which typically need to be performed on image data obtained from a see-through camera before display to a user include corrections for lens distortions. Many XR devices use see-through cameras with fisheye lenses, which have the benefit of capturing data across wide fields of view, but at the cost of significant distortion in the initially obtained video frames, which, left uncorrected, would diminish a viewer's XR viewing experience.

[0034] In addition to performing static corrections, providing a satisfactory XR viewing experience typically requires performing dynamic corrections of the video frame. As used in this disclosure, the expression "dynamic correction" encompasses corrections for factors specific to one or more conditions obtained over the interval between capturing a video frame by a see-through camera and displaying a XR video frame based on the captured frame. When combined with excessive latency in rendering XR video frames, motion effects (for example, a user turning her head) can create a disparity between the perspective of the images presented through the pass-through XR display and the perspective expected by the user based on the user's own sense of proprioception. Typically, the greater the mismatch between the perspective of the pass-through view of a scene presented in an XR display and the user's native understanding of the user's viewing perspective, the worse the XR viewing experience. Further, for many users, perceptible mismatches between the perspective of the XR display and their perceived current perspective can induce motion sickness in the user, which is particularly undesirable.

[0035] This disclosure provides examples of apparatus, method and computer-executable program code for vertex pose adjustment with passthrough and time-warp transformations for VST XR. As described in more detail below, locations within an image frame, including image frames captured by a see-through camera, a camera with a normal lens, or a rendered XR frame can be mapped to positions in an isometric grid, wherein the position comprises a point of intersection (also known as a vertex) between projections along a fixed value of a coordinate system. For example, in a Cartesian coordinate system, the vertices comprise corners of a grid paralleling the x and y axes. Similarly, in a polar coordinate system, the vertices comprise the intersections between rays emanating from the origin of the system and circles of specified radii. The same object in a scene can, due to persistent, or static factors, as well as context-dependent dynamic factors, occupy different coordinate values in image data obtained by a see-through camera lens and a normal lens projection (i.e., a camera projection generally corresponding to the field of view of a human eye). Thus, in order for image data from a see-through camera lens to be rendered to match image data from a normal lens situated at a user's eye, the vertices of the coordinate system of see-through camera lens need to be corrected to match those of the coordinate system of the normal lens view. Additionally, because correcting the coordinate system can be computationally intensive and is typically performed over human-perceptible processing intervals, it is beneficial that correction of the see-through camera image data be implemented in a way that the corrected image is projected from a viewpoint that corresponds to an XR device wearer's understanding of their viewpoint at the time.

[0036] Several embodiments according to the present disclosure reduce the computational load and processing time associated with performing static and dynamic correction of the vertices of a coordinate system of a see-through camera and reducing discrepancies between the viewpoint perspective of the presented XR display, and the viewpoint perspective expected by the user based on their own capacity for proprioception.

[0037] FIGURE 1 illustrates an example network configuration 100 including an electronic device in accordance with this disclosure. The embodiment of the network configuration 100 shown in FIGURE 1 is for illustration only. Other embodiments of the network configuration 100 could be used without departing from the scope of this disclosure.

[0038] According to embodiments of this disclosure, an electronic device 101 is included in the network configuration 100. The electronic device 101 can include at least one of a bus 110, a processor 120, a memory 130, an input / output (I / O) interface 150, a display 160, a communication interface 170, and a sensor 180. In some embodiments, the electronic device 101 may exclude at least one of these components or may add at least one other component. The bus 110 includes a circuit for connecting the components 120-180 with one another and for transferring communications (such as control messages and / or data) between the components.

[0039] The processor 120 includes one or more processing devices, such as one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). In some embodiments, the processor 120 includes one or more of a central processing unit (CPU), an application processor (AP), a communication processor (CP), a graphics processor unit (GPU), or a neural processing unit (NPU). The processor 120 is able to perform control on at least one of the other components of the electronic device 101 and / or perform an operation or data processing relating to communication or other functions. As described below, the processor 120 may perform one or more functions related to vertex pose adjustment with passthrough and time-warp transformations for VST XR.

[0040] The memory 130 can include a volatile and / or non-volatile memory. For example, the memory 130 can store commands or data related to at least one other component of the electronic device 101. According to embodiments of this disclosure, the memory 130 can store software and / or a program 140. The program 140 includes, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and / or an application program (or "application") 147. At least a portion of the kernel 141, middleware 143, or API 145 may be denoted an operating system (OS).

[0041] The kernel 141 can control or manage system resources (such as the bus 110, processor 120, or memory 130) used to perform operations or functions implemented in other programs (such as the middleware 143, API 145, or application 147). The kernel 141 provides an interface that allows the middleware 143, the API 145, or the application 147 to access the individual components of the electronic device 101 to control or manage the system resources. The application 147 may include one or more applications that, among other things, perform vertex pose adjustment with passthrough and time-warp transformations for VST XR. These functions can be performed by a single application or by multiple applications that each carries out one or more of these functions. The middleware 143 can function as a relay to allow the API 145 or the application 147 to communicate data with the kernel 141, for instance. A plurality of applications 147 can be provided. The middleware 143 is able to control work requests received from the applications 147, such as by allocating the priority of using the system resources of the electronic device 101 (like the bus 110, the processor 120, or the memory 130) to at least one of the plurality of applications 147. The API 145 is an interface allowing the application 147 to control functions provided from the kernel 141 or the middleware 143. For example, the API 145 includes at least one interface or function (such as a command) for filing control, window control, image processing, or text control.

[0042] The I / O interface 150 serves as an interface that can, for example, transfer commands or data input from a user or other external devices to other component(s) of the electronic device 101. The I / O interface 150 can also output commands or data received from other component(s) of the electronic device 101 to the user or the other external device.

[0043] The display 160 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum-dot light emitting diode (QLED) display, a microelectromechanical systems (MEMS) display, or an electronic paper display. The display 160 can also be a depth-aware display, such as a multi-focal display. The display 160 is able to display, for example, various contents (such as text, images, videos, icons, or symbols) to the user. The display 160 can include a touchscreen and may receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a body portion of the user.

[0044] The communication interface 170, for example, is able to set up communication between the electronic device 101 and an external electronic device (such as a first electronic device 102, a second electronic device 104, or a server 106). For example, the communication interface 170 can be connected with a network 162 or 164 through wireless or wired communication to communicate with the external electronic device. The communication interface 170 can be a wired or wireless transceiver or any other component for transmitting and receiving signals.

[0045] The wireless communication is able to use at least one of, for example, WiFi, long term evolution (LTE), long term evolution-advanced (LTE-A), 5th generation wireless system (5G), millimeter-wave or 60 GHz wireless communication, Wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunication system (UMTS), wireless broadband (WiBro), or global system for mobile communication (GSM), as a communication protocol. The wired connection can include, for example, at least one of a universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232), or plain old telephone service (POTS). The network 162 or 164 includes at least one communication network, such as a computer network (like a local area network (LAN) or wide area network (WAN)), Internet, or a telephone network.

[0046] The electronic device 101 further includes one or more sensors 180 that can meter a physical quantity or detect an activation state of the electronic device 101 and convert metered or detected information into an electrical signal. For example, the sensor(s) 180 include cameras or other imaging sensors, which may be used to capture images of scenes. The sensor(s) 180 can also include one or more buttons for touch input, one or more microphones, a depth sensor, a gesture sensor, a gyroscope or gyro sensor, an air pressure sensor, a magnetic sensor or magnetometer, an acceleration sensor or accelerometer, a grip sensor, a proximity sensor, a color sensor (such as a red green blue (RGB) sensor), a bio-physical sensor, a temperature sensor, a humidity sensor, an illumination sensor, an ultraviolet (UV) sensor, an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an infrared (IR) sensor, an ultrasound sensor, an iris sensor, or a fingerprint sensor. Moreover, the sensor(s) 180 can include one or more position sensors, such as an inertial measurement unit that can include one or more accelerometers, gyroscopes, and other components. In addition, the sensor(s) 180 can include a control circuit for controlling at least one of the sensors included here. Any of these sensor(s) 180 can be located within the electronic device 101.

[0047] In some embodiments, the electronic device 101 can be a wearable device or an electronic device-mountable wearable device (such as an HMD). For example, the electronic device 101 may represent an XR wearable device, such as a headset or smart eyeglasses. In other embodiments, the first external electronic device 102 or the second external electronic device 104 can be a wearable device or an electronic device-mountable wearable device (such as an HMD). In those other embodiments, when the electronic device 101 is mounted in the electronic device 102 (such as the HMD), the electronic device 101 can communicate with the electronic device 102 through the communication interface 170. The electronic device 101 can be directly connected with the electronic device 102 to communicate with the electronic device 102 without involving with a separate network.

[0048] The first and second external electronic devices 102, 104 and the server 106 each can be a device of the same or a different type from the electronic device 101. According to an embodiment of this disclosure, the server 106 may include a group of one or more servers. Also, according to an embodiment of this disclosure, all or some of the operations executed on the electronic device 101 may be executed on another or multiple other electronic devices (such as the electronic devices 102, 104 or server 106). Further, according to an embodiment of this disclosure, when the electronic device 101 should perform some function or service automatically or at a request, the electronic device 101, instead of executing the function or service on its own or additionally, may request another device (such as electronic devices 102, 104 or server 106) to perform at least some functions associated therewith. The other electronic device (such as electronic devices 102, 104 or server 106) is able to execute the requested functions or additional functions and transfer a result of the execution to the electronic device 101. The electronic device 101 can provide a requested function or service by processing the received result as it is or additionally. To that end, a cloud computing, distributed computing, or client-server computing technique may be used, for example. While FIGURE 1 shows that the electronic device 101 includes the communication interface 170 to communicate with the external electronic device 104 or server 106 via the network 162 or 164, the electronic device 101 may be independently operated without a separate communication function according to some embodiments of this disclosure.

[0049] The server 106 can include the same or similar components as the electronic device 101 (or a suitable subset thereof). The server 106 can support to drive the electronic device 101 by performing at least one of operations (or functions) implemented on the electronic device 101. For example, the server 106 can include a processing module or processor that may support the processor 120 implemented in the electronic device 101. As described below, the server 106 may perform one or more functions related to depth-varying reprojection passthrough in VST XR.

[0050] Although FIGURE 1 illustrates one example of a network configuration 100 including an electronic device 101, various changes may be made to FIGURE 1. For example, the network configuration 100 could include any number of each component in any suitable arrangement. In general, computing and communication systems come in a wide variety of configurations, and FIGURE 1 does not limit the scope of this disclosure to any particular configuration. Also, while FIGURE 1 illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.

[0051] FIGURE 2 illustrates an example of corrections applied to a see-through camera image frame based on applying vertex adjustment values to correct a distortion mesh. For ease of explanation, the distortion meshes and corrections described with reference to FIGURE 2 are described as being performed by or implemented using the electronic device 101 in the network configuration 100 of FIGURE 1. However, the operations described with reference to FIGURE 2 may be implemented using any other suitable device(s) and in any other suitable system(s), such as the server 106.

[0052] As shown in FIGURE 2, the see-through camera of an XR device can capture image frame data of a scene, wherein the intrinsics of the camera, such as the lens shape and other focal properties of the see-through camera create distortions in the image frame data of the scene. For example, the wide field of view provided by fisheye lenses comes at the cost of the raw image data of a scene appearing significantly distorted relative to how the same scene would appear through the lens of the human eye. Specifically, image frame data appears "curvy" relative to a human eye view of the same scene, with parallel lines in the scene becoming increasingly curved at points away from the center of the image, and points at the center of the image appearing excessively close relative to a human eye view.

[0053] As shown in FIG. 2, the distortions to the image frame data created by the intrinsics of the see-through camera, and other static and dynamic effects cause points in the image frame data to appear distorted - as if mapped to a non-rectilinear coordinate grid. This distortion can be readily observed in images of checkerboards, test cards and other rectilinear grids obtained by cameras fisheye or wide-angle lenses. Referring to the illustrative example of FIG. 2, locations within the image frame data can be described as being mapped to coordinates (x, y) in an initial mesh 201. The difference between the coordinate values of locations in initial mesh 201 and a distortion mesh 250 which accounts for the distortions produced by the intrinsics of the lens and static and dynamic effects can be given by:

[0054]

[0055] Where are grid point adjustments to offset the shift of a location in the image frame data from its location in a regular, rectilinear mesh and its location in initial mesh 201. For example, a first set of adjustment values accounts for the coordinate shift to perform a camera undistortion and rectification 205 to offset the distortions induced by the shaple of the camera lens. A second set of adjustment values accounts for the coordinate shift to perform a static passthrough transformation 210 to offset distortions or projection effects from static causes (for example, the positioning of a see-through camera at a point removed from a viewer's eyeball. A third set of adjustment values accounts for the coordinate shift to perform a dynamic passthrough transformation 215 to account for coordinate shifts associated with dynamic factors (for example, head pose change compensation). Other adjustment values account for the coordinate shift to perform display correction including corrections of geometric distortions and chromatic aberrations.

[0056] As discussed herein, for sources of distortion which are static (i.e., sources of distortion where the values of do not change over time or in response to present values of any variable), vertex adjustment values for reprojecting the image data according to a corrected mesh can be determined in advance, and do not need to be subsequently recalculated. Examples of static sources of distortion include focal distortions (for example, barrel or pincushion distortions) of the lens of the see-through camera from which image data is obtained. Other examples of static sources of distortion include parallax, or viewpoint effects arising from differences in the location of the see-through camera (for example, on the exterior perimeter of a wearable XR device) relative to the expected location of a viewer's eye. In this way, embodiments according to the present disclosure reduce the computational load associated with generating a VST XR display.

[0057] Although FIGURE 2 illustrates one example of distortion meshes and corrected meshes, various changes may be made to the example described with reference to FIGURE 2. For example, depending on the character of the lens distortion, initial and distortion meshes 201, 250 may comprise vertices in a polar coordinate system.

[0058] FIGURE 3 illustrates an exemplary pipeline 300 for vertex pose adjustment with passthrough and time-warp transformations in VST XR in accordance with this disclosure. For ease of explanation, the pipeline 300 of FIGURE 3 is described as a set of processes that can be implemented using the electronic device 101 in the network configuration 100 of FIGURE 1. However, the pipeline 300 may be implemented using any other suitable device(s) and in any other suitable system(s), such as the server 106, and the pipeline 300 may perform any other suitable process(es).

[0059] Referring to the illustrative example of FIGURE 3, pipeline 300 comprises a processing architecture by which a frame 301 from a see-through camera is rendered as a virtual frame, and the displayed rendered virtual frame 399 is presented at a time such that the pose-dependent viewpoint of displayed rendered virtual frame 399 corresponds to the predicted pose-dependent viewpoint of a user at the time of display. In this way, embodiments implementing pipeline 300 can eliminate or significantly reduce both rendering latency and discrepancies between the viewpoint of a VST XR display, and the viewpoint a user expects based on their body's sense of proprioception.

[0060] At block 305, vertex position adjustments of the static components of a distortion mesh may be computed. Block 305 may be performed prior to capturing see-through camera frame 301, as part of a calibration process performed during startup of the XR device, or, in some embodiments, can be performed as part of an initial configuration during manufacture of the XR device. Vertex adjustments for one or more static components of the distortion mesh may be obtained by capturing image data of a test pattern, checkerboard, or other subject in which the distortions due to static factors can be identified or quantified from the image data.

[0061] For example, the vertex adjustments associated with lens distortions of a see-through camera may be determined by first creating a regular grid Gd(m) for defining a distortion mesh, as shown below:

[0062] grid = Gd(m, n), (m=0, 1, ..., M ; n=0, 1, ..., N) (1)

[0063] Here, M is the grid width, is the grid height. The values (m, n) can be normalized to the range [0, 1] to create an identity mesh.

[0064] A distortion mesh Md(x, y) for distortion transformation and rendering may be defined as follows:

[0065] mesh = Md(x, y), (x∈[-1, 1]; y∈[-1, 1]) (2)

[0066] Here, Md(x, y) is also normalized to the range [-1, 1].

[0067] Camera lens distortion Dcmay be defined as follows:

[0068] Dc= C(xc, yc), (xc∈[0, 1]; yc∈[0, 1]) (3)

[0069] Here, (xc, yc) is normalized to the range [0, 1] for camera lens distortion. The lens distortion may be computed by a lens distortion model from camera calibration, such as calibration based on image data of a test image.

[0070] From the above, vertex adjustments for camera lens distortion may be computed as follows:

[0071] dc= C(xc, yc) - Gd(m, n)

[0072] Here, (xc, yc) and (m, n) are normalized to the range [0, 1].

[0073] In addition to computing vertex adjustment values for fixed distortions due to the shape of the lens of a see-through camera, vertex adjustment values to match the viewpoint of image data obtained by the see-through camera to that of the user's eye (to offset the fact that the see-through camera cannot occupy the same physical location as the user's eye) may also be performed at block 305. Similar to camera lens distortion Dc(described with reference to Equation (3) above), the distortion Dm due to the difference in location between a see-through camera lens and a user's eye may be represented by a transformation, such as shown below:

[0074] Dm = Vp(xv, yv), (xv∈[-1, 1]; yv∈[-1, 1]) (5)

[0075] From this vertex adjustment values for viewpoint matching may be computed, such as shown below:

[0076] dv= Dm- Dc(6)

[0077] Chromatic aberrations, such as the blue-yellow or red-green fringes seen along edges between areas of high contrast between light and dark can be particularly pronounced in image data obtained from cameras with wide-angle or fisheye lenses, and present a further example of static distortion for which vertex adjustment values may be obtained at block 305. In some embodiments, vertex adjustment values for chromatic aberrations are generated along similar lines due to the overall lens shape and vertex adjustment values may be necessarily calculated. For example, distortion for each channel of the color space used in the image data (for example, RGB or CYMK) can be modeled as follows:

[0078]

[0079] (7)

[0080]

[0081] Here, , , are lens distortion models in the individual color channels of the color space, which in this explanatory example, is the red-green-blue ("RGB") color space.

[0082] From the color-channel specific lens distortion models, distortion differences due to chromatic aberrations can be computed as follows:

[0083]

[0084] (8)

[0085] Here, is the previously-calculated display lens geometric distortion and the differences may be due to chromatic aberrations.

[0086] By pre-calculating vertex position adjustments for static sources of distortion at block 305 significant processing time savings between capturing see-through camera image frames and displaying a rendered XR frame may be realized. Table 1 below lists observed processing times associated with calculating vertex point adjustments for certain sources of static distortion.

[0087] OperationMeasured Processing Time (ms)Generating Distortion Mesh5Determining Vertex Adjustment Values for See-Through Camera Lens Distortion371.842Determining Vertex Adjustment Values for Viewpoint Matching0.136Determining Vertex Adjustment Values for Lens Distortion130.045Display Lens Chromatic Aberration260.090

[0088] As shown above, the five computing operations shown above collectively require approximately 0.75 seconds to perform, meaning that if performed on a rolling, per frame basis, would introduce significant latency between image capture and display at an XR headset. Pre-computing vertex adjustment values and distortion meshes for these static sources of distortion at block 305 may significantly reduce the latency associated with providing a VST XR display. Even with the time savings associated with the operations performed at block 305, rendering and displaying a VST XR frame generally entails processing performed over human-perceptible time intervals (for example, hundredths of a second), and, by implication, human perceptible latency between the time see-through camera frame 301 is captured and the rendered virtual frame 399 is displayed. In applications where human-perceptible latency between image frame capture and VST XR frame display is present, changes in a user's head pose (rotational, translational or both) during this interval can result in human-perceptible mismatches between the pose-dependent perspective of the displayed rendered virtual frame and the user's proprioception-based understanding of the user's pose at the time of display. As noted in this disclosure, even small mismatches between the pose-dependent perspective of rendered virtual frame 399 and the user's native understanding of the user's pose-dependent perspective can cause nausea, which significantly degrades a user's XR experience.

[0089] Referring to the illustrative example of FIGURE 3, in cases where the processing associated with rendering virtual frame 399 requires a human-perceptible time interval, the XR experience can be significantly improved by rendering virtual frame 399 from an expected pose-dependent viewpoint of the viewer at the time of display, rather than from the viewpoint of the user's head at the time see-through frame 301 is captured. Accordingly at block 307, the processor (whether in the XR headset itself or an accessory device) may estimate the correction interval associated with the operations between capturing see-through camera frame 301 and displaying rendered virtual frame 399. As used in this disclosure, the expression "correction interval" encompasses the period of time (or latency) between image frame capture and XR frame display during which the processing platform corrects the raw image obtained from the see-through camera obtained at a first time for display at a second, subsequent time.

[0090] Depending on embodiments, at block 307, the correction interval may be estimated programmatically, for example, by starting with a minimum correction interval, and incrementing the correction interval by predetermined intervals upwards based on one or more rules-based criteria (for example, the number of other applications executing at the processor, available memory, etc.). In some embodiments, the correction interval may be calculated dynamically, as a weighted numerical function of one or more present values (for example, available memory, quantity of virtual content to be rendered as part of the XR) describing the available processing resources and expected size of the instant processing task.

[0091] Once a correction interval has been estimated, at block 307, the processor may also estimate a change in the user's head pose during the correction interval, and from this, predict the user's pose-dependent viewpoint at the end of the correction interval. Based on the user's predicted pose-dependent viewpoint, a second set of vertex position adjustments to further correct the image data of see-through camera frame 301 to account for the change in the user's pose-dependent viewpoint over the correction interval.

[0092] As described in greater detail with reference to FIGURE 4 of this disclosure, depending on the data regarding the scene and the user's head available to the processing device(s) implementing the XR display, the user's predicted pose-dependent viewpoint at the end of the correction interval can be computed according to a variety of methods. In some embodiments, the user's predicted pose-dependent viewpoint may be based upon a time-warp projection accounting only for rotational movement during the correction interval. In some embodiments, including embodiments where data capturing the user's translational movement (for example, depth data, or motion sensor data) is available, the predicted pose-dependent viewpoint may also account for translational movement.

[0093] At block 309, the processing platform integrates or "puts together" the first set of vertex position adjustment values obtained at block 305 and the second set of vertex position values obtained at block 307 to create a distortion mesh for correcting both the static sources of distortion, as well as the dynamic sources of differences between see-through camera frame 301 and rendered virtual frame 399 (for example, the user's predicted pose-dependent viewpoint). Depending on the extent of the static distortions corrected at block 305, and the extent and nature of it can be computationally advantageous to apply the first set of vertex adjustment values to obtain intermediate image data, wherein in the intermediate image data is corrected for static sources of distortion, prior to applying the second set of vertex adjustment values to account for dynamic changes, such as predicted changes in the user's pose-dependent viewpoint. For example, when performed on heavily distorted image data (such as obtained from a fisheye lens) certain perspective corrections, such as time-warp reprojection to account for changes in perspective, are harder to perform correctly on uncorrected image data.

[0094] In addition to applying the first and second sets of vertex adjustment values of the distortion mesh, at block 309, the processing device may render a final virtual frame. Rendering a final virtual frame can include, without limitation, rendering and positioning items of virtual content within the final virtual frame. As used in this disclosure, the expression "item of visual content" encompasses objects appearing in an XR display, and which are not present in the physical world captured by the see-through camera. Examples of items of visual content include, without limitation, text or characters appearing to sit on surfaces of real-world objects, and avatar reprojections (for example, reprojections of recognized faces in the scene as cartoon characters) of real-world people and objects.

[0095] Once rendered, virtual frame 399 may be displayed at the display of the XR device at a second time corresponding to the end of the correction interval.

[0096] Although FIGURE 3 illustrates one example of a pipeline 300 for vertex pose adjustment with passthrough and time-warp transformations in VST XR, various changes may be made to FIGURE 3. For example, various components or functions in FIGURE 3 may be combined, further subdivided, replicated, omitted, or rearranged and additional components or functions may be added according to particular needs. As a particular example, in some embodiments, generating intermediate image data by separately applying the first set of vertex position adjustments prior to applying the second set of vertex position adjustments may be omitted.

[0097] FIGURE 4 illustrates aspects of generating vertex position adjustments for a pose-dependent viewpoint according to various embodiments of this disclosure. Specifically, FIGURE 4 describes an example of computing vertex adjustment values to compensate for head movements over a correction interval (for example, as performed at block 307 in FIGURE 3).

[0098] Referring to the explanatory example of FIGURE 4, at a fundamental level, compensating for head movements over a correction interval requires creating a new image frame at a new pose from an existing frame at an existing pose , wherein the existing frame is based on image data obtained from a see-through camera (for example, see-through camera frame 301) at a first time, and the new image frame is a reprojection based on the user's predicted pose at a second time at the end of the correction interval.

[0099] As shown in FIGURE 4, first frame is captured from a first perspective Ci and the new image frame comprises a reprojection of at the second time, wherein the user's pose is predicted to have changed based on a rotation R over the correction interval, such that new image frame is displayed from the predicted new perspective Co. The reprojection of as new image frame can be performed by a transform of the form:

[0100] (9)

[0101] Here, is a projection matrix, is a predicted head pose, and is a head pose while capturing frame .

[0102] Depending on the data available to the processing device implementing vertex position adjustment for pass-through VST, can be obtained in one of a plurality of ways, using simultaneous location and mapping ("SLAM") techniques for predicting the user's pose at the end of the correction interval. For example, where depth information of the scene captured by the see-through image data is not available, the user's predicted pose can be predicted based on a time-warp transformation based on data as to the magnitude and direction of the user's head rotation. Data on the magnitude and direction of a user's head rotation can be obtained in multiple ways, including, without limitation, from sensor data (i.e., data from a 3 DOF or 6 DOF accelerometer), or based on frame-over-frame changes in frames obtained prior to frame

[0103] In some embodiments, such as where depth information is not available, or performing calculations using depth information may introduce unwanted latency, can be determined only based on a predicted rotation of a user's head during the compensation interval, with translational changes expressed through depth information removed from the pose prediction. In such cases, Equation 9 can be simplified as:

[0104] (10)

[0105] Here, is a depth-independent time-warp transformation accounting for only rotational changes in the user's pose.

[0106] can be further expressed as:

[0107] (11)

[0108] Here, So and Si are expressions of head pose which only capture the rotational position of the user's head at the start and end of the correction interval.

[0109] From the time-warp transformation Tw, vertex adjustment values can be computed to account for the shift in viewpoint over the correction interval, as shown below:

[0110] (12)

[0111] As noted previously, vertex position adjustments can be applied in either multiple passes (for example, by applying corrections for static sources of distortion in a first pass, and then further correcting to account for pose changes during a compensation interval in a second pass), or in a single pass.

[0112] Regardless of whether the vertex position adjustments are applied in a single or multiple passes, each vertex of a final distortion mesh can be expressed as a value (x,y) of an initial distortion mesh plus the sum of vertex adjustment values for each of the compensated static and dynamic sources of distortion, as shown below:

[0113] (13)

[0114] Although FIGURE 3 illustrates one example of predicting a head pose at the end of a correction interval and generating vertex position adjustments for the predicted head pose at the end of a correction interval as part of a rendering pipeline of a VST AR display, various changes may be made to FIGURE 3. For example, various components or functions in FIGURE 3 may be combined, further subdivided, replicated, omitted, or rearranged and additional components or functions may be added according to particular needs. As a particular example, where depth information is available, the time-warp reprojection to determine the vertex adjustments associated with the change in the user's pose-dependent viewpoint can be expanded to account for both rotational and translational movement of the user's head. Specifically, the frame-over-frame change over the correction interval can be represented as:

[0115] (14)

[0116] Here, the existing frame and the new frame express coordinates in a three-dimensional space and poses and include head rotation and translation.

[0117] Subsequent to implementing vertex corrections for the predicted pose change over the correction interval, a depth-based reprojection of the objects in the see-through camera image frame is computed, for example, by a vertex shader or fragment shader, to account for occlusion and changes in the relative size of objects in the image frame..

[0118] In a further example of additional ways in which some embodiments according to this disclosure can account for the changes in pose during a compensation interval, is that, in embodiments in which depth information is available, depth based reprojection and VST pass-through correction can be handled in a single pass by one or more processing modules responsible for depth-based reprojection of image data, such as by a vertex shader or a fragment shader.

[0119] FIGURE 5 illustrates an exemplary pipeline 500 for vertex pose adjustment with passthrough and time-warp transformations in VST XR in accordance with this disclosure. For ease of explanation, the pipeline 500 of FIGURE 5 is described as a set of processes that can be implemented using the electronic device 101 in the network configuration 100 of FIGURE 1. However, the pipeline 500 may be implemented using any other suitable device(s) and in any other suitable system(s), such as the server 106, and the pipeline 500 may perform any other suitable processes.

[0120] Referring to the illustrative example of FIGURE 5, blocks 501-515 comprise the constituent processes of generating a passthrough transformation. Specifically, blocks 501-515 comprise generating a distortion mesh and determining a first set of vertex adjustment values to compensate for static sources of distortion in image frame data obtained from a see-through camera of an XR device. In some embodiments, the operations described with reference to blocks 501-515 can be performed before starting to obtain see-through camera image frame data. For example, blocks 501-515 can be performed by a processing device as part of an initialization routine of an XR device.

[0121] At block 501, a distortion model for the see-through camera may be determined. In some embodiments, determining the distortion model comprises obtaining image data from the see-through camera of a ground truth image (such as a test pattern or checkerboard shown in FIGURE 2) from which the geometric distortions (for example, barrel or pincushion distortions) inherent to the shape of the lens of the see-through camera can be determined and expressed in a quantitative model, such as a radial distortion model.

[0122] At block 503, a model for the geometric relationship between the location see-through camera of the XR device providing the image frame data for the VST XR display and the expected location of the viewer's eye may be determined. Given that a see-through camera cannot occupy the same location as the viewer's eye, and that there are analytical benefits (for example, improved parallax-based depth estimation) to spacing the one or more see-through cameras further apart than the natural spread of human eyes, the spatial relationship between the perspective of the see-through camera(s) and the viewer's eyes needs to be modeled and compensated for.

[0123] At block 505, a model for the lens characteristics other than the intrinsic distortions due to the lens shapes may be generated. For example, the differences in refraction across light wavelengths which give rise to chromatic aberrations and other color-specific distortions may be quantified and modeled at block 505.

[0124] At block 507, a transform of the model for the intrinsic distortion due to the shape of the see-through camera's lens may be determined. According to some embodiments, the model underlying the transform may be of the same general form as the distortion model described with reference to Equation (3) of this disclosure.

[0125] According to some embodiments, at block 509, a transform of the model for viewpoint matching between the see-through camera and the user's eye may be determined. The transform for viewpoint correction and parallax issues arising from differences in lens spread between two or more see-through cameras and the spread of a user's eyes can be of the same general form as the transformation described with reference to Equation (5) of this disclosure.

[0126] At block 511, a transform to account for further, color-specific distortions (such as chromatic aberrations) may be determined based on the model generated at block 505. The transforms for color fringing-type chromatic aberration may be of the same general form as those described with reference to Equations (7)-(8) of this disclosure.

[0127] At block 513, a distortion mesh (for example, distortion mesh 203) embodying the distortions modeled at blocks 501, 503, and 505 may be created. At block 515, vertex adjustment values for each of the transformations performed at blocks 507-511 may be generated and applied to the distortion mesh generated at block 513. In some embodiments, the operations performed at block 515 may parallel the corrections described with reference to FIGURE 2 to generate corrected mesh 205 from distortion mesh 203.

[0128] To reduce latency in generating VST XR frames from see-through camera image frames, corrections for static sources of distortion in the passthrough image data may be corrected in advance, and vertex adjustment values pre-calculated for rapid application to image frames from a see-through camera. As discussed with reference to Table 1 of this disclosure, by performing blocks 501-515 in advance, rather on a rolling frame-by-frame basis, the latency associated with providing an XR display from see-through camera image data may be reduced by up to ¾ of a second.

[0129] At block 517, the processing device implementing pipeline 500 may perform camera pose tracking. Depending on the sensors provided at the XR device worn by the user, camera pose tracking may be implemented in a variety of ways. For example, where the XR device includes motion sensors (for example, 3 DOF or 6 DOF accelerometers) and / or depth sensors (for example, time of flight ("TOF") sensors), the camera pose tracking performed at block 517 may comprise an implementation of a full-featured SLAM pipeline. Additionally, or alternatively, where, for example, the XR device does not include motion and / or depth sensors, or conserving processing resources is a priority, camera pose tracking at block 517 may be performed by performing a frame-over-frame analysis to estimate rate and direction of changes in a user's pose.

[0130] At block 519, the processing platform providing the VST XR display may obtain one or more frames of image data from one or more see-through cameras of the XR device, wherein the one or more frames of image data are obtained at a first time. Depending on the configuration of the XR device and the see-through camera(s) provided thereon, block 517 may comprise receiving the direct (or straight-out-of-camera ("SOOC")) output of a CMOS sensor or the like. In some embodiments, capturing a see-through frame may comprise capturing a plurality of frames obtained at the same time by a see-through camera, wherein each of the plurality of frames corresponds to a channel of a color space of the see-through camera. In the illustrative example of FIGURE 5, the see-through camera provides an output in the red-green-blue color space, and at block 519, the processing device capture an image frame in the red channel, a frame in the green channel and an image frame in the blue channel at a first time. Additionally, in embodiments in which an XR device is provided with multiple see-through cameras (for example, to widen a field of view or as a source of stereoscopic depth data), the data obtained at block 517 may be pre-processed (for example, to fuse the outputs of two cameras).

[0131] As noted elsewhere herein, the technical benefits provided by certain embodiments according to this disclosure include improved synchronization between the perspective of the VST XR display and the user's native understanding of the user's pose-dependent perspective. Synchronizing the perspective of a future XR display with a user's future pose-dependent perspective generally requires that the processing device know the future time which the XR display is to be presented. For many applications, the primary sources of at least some unavoidable latency between frame capture at block 519 and display of a rendered XR frame at block 551 may include latency capturing the image (for example, due to the time associated with exposing the sensor, and buffering and outputting the data from the sensor as an image frame), latency in rendering the frame (for example, latency associated with depth-based reprojection and generating and positioning items of virtual content within an XR frame) and latency in displaying the XR frame (for example, where the XR frame is one of a plurality of items of content to be placed on the display).

[0132] Accordingly, at block 521, the processing platform may estimate the latency associated with the capture of a see-through image frame at block 519. Depending on embodiments, estimating the latency for capturing an image frame may be done programmatically (for example, by applying tabulated values for present constraints, such as the selected resolution of the image frame) or calculated dynamically.

[0133] Similarly, at block 547, the latency associated with rendering an XR frame for display may be estimated. As with estimating the latency for capturing an image frame, depending on embodiments and available resources, the latency associated with rendering the XR frame for display may be estimated either programmatically or dynamically calculated.

[0134] Likewise, at block 549, the latency associated with displaying a rendered XR frame at an XR device may be estimated. This, the latency associated with displaying the rendered XR frame at the XR device may be estimated programmatically or analytically, depending on the design goals and processing resources of the specific implementation.

[0135] In the explanatory example of FIGURE 5, the outputs of blocks 521, 547, and 549 may be summed to generate an estimated correction interval (i.e., an estimate of the total interval between capture of a see-through frame at block 519 and display of an XR frame at block 551). From the correction interval, the processing device implementing pipeline 500 may have a target time for performing a head pose prediction. At block 523, the processing device implementing pipeline 500 may apply the camera pose tracking information obtained at block 517 to generate a prediction of the XR device wearer's head pose at a second time at the end of the correction interval. Predicting the user's head pose at block 523 may be performed according to one or more techniques known in the art, such as extrapolation from accelerometer data obtained previous to, and / or simultaneously with capturing a see-through image frame at block 519. Additionally, or alternatively, predicting the user's head pose at the second time can be performed using machine learning or artificial intelligence-based approaches, which recognize movement patterns in the camera pose tracking data obtained at block 517.

[0136] Referring to the explanatory example of FIGURE 5, at block 525, the processing device may perform vertex position adjustment for the static sources of distortion, as well as a transformation of the see through frame captured at block 519 to pose-dependent user viewpoint predicted at block 523 to obtain a corrected mesh (for example, corrected mesh 205) which can be used in rendering the XR frame for display.

[0137] It will be understood that that integrating the passthrough corrections performed at blocks 501-515 with further corrections to compensate for the movement of the XR device wearer's head during the compensation interval can be performed according to a diverse plurality of operations. Put differently, there are multiple ways of incorporating the corrections for static distortions and predictions about the change in the user's pose over the compensation interval. Blocks 527-543 of FIGURE 5 are intended to be illustrative, rather than limitative of how vertex position adjustment with a pose transformation can be performed to achieve the result of block 525.

[0138] At block 537, the processing device may apply vertex position adjustments for both the static sources of distortion, and the change in pose-dependent viewpoint over the correction interval (shown in FIGURE 5 as blocks 527-533 and 539-541). It will be understood that, depending on the implementation, the corrections performed at blocks 527-533 and 539-541 may be performed in different sequences, and can, where appropriate, be batched. For example, in some embodiments, it can be advantageous to apply vertex position adjustments to correct for static sources of distortion in a first batch, thereby obtaining intermediate image data, from which the lens and viewpoint disparities between the user's eye and the see-through camera have been corrected, but which has not yet been corrected to account for changes in pose-dependent viewpoint over the correction interval.

[0139] At block 527, the processing device may apply the vertex adjustment values to correct for the intrinsic distortion due the shape of the see-through lens and (obtained via blocks 501, 507, and 515) to each channel of the frame data obtained at block 519. In this example, because the image data of the see-through frame is provided as data in each of the channels of an RGB color space, vertex adjustment values for color-specific sources of distortion (due to wavelength-dependent differences in refraction through the lens of the see-through camera) may be applied separately based on one or more predetermined models (for example, the model(s) determined at block 505). Thus, at block 529, vertex position adjustment values for color-specific distortions in the red channel may be applied to the red channel image data. Similarly, at block 531, vertex position adjustment values for color-specific distortions in the green channel may be applied to the green channel image data. At block 533, vertex position adjustment values for color-specific distortions in the blue channel may be applied to the blue channel image data.

[0140] At block 541, vertex position adjustment values based on the model determined at block 503 for viewpoint correction may be applied to the image data. In contrast to the vertex position adjustment values applied at blocks 529-533, the vertex position adjustment values applied at block 541 may be color-independent and applied identically across each of the color channels.

[0141] At block 541, the processing device may apply vertex position adjustment values (for example, values calculated based on the model generated at block 503) to correct for the viewpoint disparity between an XR device user's eye(s) and the see-through camera(s). Here again, the vertex position adjustment values are color independent, and can be applied either separately or batched to each channel of the image frame data.

[0142] At block 539, the processing device may apply a transformation to account, based on the head pose prediction obtained at block 523, for the user's pose-dependent viewpoint at the end of the correction interval, and may apply vertex adjustment values based on the transformation, such as described with respect to Equations (9)-(12) of this disclosure. In the illustrative example of FIGURE 5, the transformation underlying the vertex adjustment values is a time-warp to compensate only for head rotation during the compensation interval.

[0143] At block 543, the vertex correction values applied at block 537 may be consolidated and re-expressed as a corrected mesh (for example, corrected mesh 205 in FIGURE 2) to be used during rendering of the XR frame for display.

[0144] At block 545, a VST XR frame based on the image data obtained at block 519 may be rendered by the processing device, such as a GPU. Rendering the VST XR frame may include reprojecting the image data obtained at block 519 according to the corrected mesh generated at block 543. Depending on the availability of depth information, at block 545, a depth-based reprojection of the image data, to account for resizing of objects as a result of the user's change in pose-dependent viewpoint may also be performed. Additionally, at block 545, one or more items of virtual content to be included in the XR display may be rendered, scaled, and positioned in the VST XR frame. At block 551, the rendered VST XR frame may be displayed to the user at the XR device at a second time estimated by the correction interval.

[0145] Although FIGURE 5 illustrates an exemplary pipeline 500 for vertex pose adjustment with passthrough and time-warp transformations in VST XR, other embodiments or variations are possible and within the contemplated scope of this disclosure. For instance, FIGURE 6 describes another exemplary embodiment of such a pipeline. Additionally, in some embodiments, the processing device implementing pipeline 500 displays the VST XR frame at a second time scheduled by the correction interval, that is, the estimated correction interval schedules the display of the VST XR frame, and if the frame is not ready by the specified time, it can be dropped. In some embodiments, the correction interval is used only as an estimation tool, and the VST XR frame is displayed when ready. Depending on the application, variations in frame rate may detract less from the XR experience than dropped frames, or vice versa.

[0146] FIGURE 6 illustrates an exemplary pipeline 600 for vertex pose adjustment with passthrough and time-warp transformations in VST XR in accordance with this disclosure. For ease of explanation, the pipeline 600 of FIGURE 6 is described as a set of processes that can be implemented using the electronic device 101 in the network configuration 100 of FIGURE 1. However, the pipeline 600 may be implemented using any other suitable device(s) and in any other suitable system(s), and the pipeline 600 may perform any other suitable processes.

[0147] As noted elsewhere in this disclosure, this disclosure contemplates multiple processing architectures for generating vertex adjustment values to conform image data obtained from a see-through camera at a first time and mapped to a distortion mesh to a corrected mesh associated with a predicted pose-dependent viewpoint at a second time. Exemplary pipeline 600 is for illustration and should not be construed as limitative of this disclosure or the claims.

[0148] Referring to the explanatory exemplary of FIGURE 6, at block 601, the processing device (for example, a CPU or GPU provided at an XR device) obtains at a first time, a frame of image data from one or more see-through cameras of the XR device presenting a VST XR display. In this example, the see-through camera frame comprises image data in each of the three channels of the red-green-blue color space. It will be understood that other implementations with different numbers of channels and different color spaces, such as cyan-magenta-yellow-key ("CMYK") are possible and within the contemplated scope of this disclosure.

[0149] In the illustrative example of FIGURE 6, vertex adjustment values may be applied in parallel, across three initial distortion meshes (one for each channel of the color space) to obtain three corrected meshes for rendering a VST XR frame. In addition to parallel application of vertex adjustment values, correction for static sources of distortion and pose changes over a predicted correction interval, pipeline 600 may split the application of vertex corrections into three discrete stages.

[0150] At a first stage 610, the processing device implementing pipeline 600 may apply three sets of across-the-board vertex adjustment values for static sources of distortion. In this example, because three separate distortion meshes are created for each of the three color channels of the image data, corrective vertex adjustment values may be applied separately to each of the distortion meshes.

[0151] For example, at block 611, the processing device may apply vertex adjustment values to correct for distortions (for example, barrel, fisheye or moustache distortion) inherent to shape of the lens of the see-through camera from which the image frame data was obtained. As the distortions due to lens shape (excluding wavelength-dependent variations in refraction), are the same across all color channels, a single set of vertex adjustment values may be computed and applied across to each channel of the color space of the see-through camera image data. According to some embodiments, the corrections performed at block 611 may be performed based on the model generated at block 501 of FIGURE 5.

[0152] Referring to the illustrative example of FIGURE 6, at block 613, the processing device applies to each of the three distortion meshes, vertex adjustment values to correct for the viewpoint difference between the viewpoint of the see-through camera and the viewpoint of an XR device user's eye(s) may be applied to each of the color channel distortion meshes. As with vertex adjustment values for distortions due to lens shape, the vertex adjustments for correcting the viewpoint mismatch between the see-through camera and the viewer's eye(s) may be color-independent and may be computed once and a single result applied across all of the color channels. In some embodiments, the viewpoint adjustment values may be determined based on the model determined at block 503 of FIGURE 5.

[0153] At block 615, vertex adjustment values to correct for chromatic aberrations (for example, multi-color fringes around high-contrast edges may be applied to the distortion meshes of each of the constituent color channels of the image data. The aforementioned vertex adjustment values may be determined based on the distortion model of block 505 of FIGURE 5.

[0154] As shown in FIGURE 6, subsequent to applying vertex correction values to the red, green, and blue distortion meshes to account for static sources of distortion, a second batch of vertex adjustments for color-specific sources of distortion may be performed at block 620. For example, the color-specific distortion of each color channel can be computed from the lens distortion models in Equation (7). Further, and as described with reference to in Equation (8), distortion differences between channels of the color space (for example, red and green channels and blue and green channels) can be used to determine vertex adjustment values for chromatic aberration corrections. Because the refractive properties of the lens of a see-through camera can vary due to the wavelength of light passing through the lens, further, color-specific refinements to the across-the-board corrections of block 611 may be required. For example, all other factors being equal, the wider the angle of the lens, the more pronounced color-specific distortions can become.

[0155] Thus, at block 621, the processing device may apply vertex adjustment values to account for wavelength specific distortion to the distortion mesh for the red channel. At block 623, the processing device may apply vertex adjustment values to correct for wavelength specific diction to the distortion mesh for the green channel. Similarly, at block 625, the processing device may apply color-specific corrections to the distortion mesh for the blue channel.

[0156] To further illustrate the variety in which the correction operations for removing static sources of distortion from image data obtained from a see-through camera and updating the pose-dependent viewpoint of an image frame being batched, sequenced and performed, at a third stage 630, time warp transformations to obtain vertex adjustment values for updating the pose-dependent perspective to correspond to a user's predicted pose-dependent perspective at the end of a correction interval, may be performed in parallel on each color channel's distortion mesh.

[0157] Thus, in the illustrative example of FIGURE 6, at block 631, vertex adjustment values of a time warp transformation (for example, the time warp performed at block 539 of FIGURE 5) may be applied to the distortion mesh of the red channel of the see-through camera image frame. From this, a final, corrected mesh for rendering the red channel of an VST XR frame may be obtained.

[0158] Similarly, at block 633, vertex adjustment values of the same time-warp transformations may be applied to the distortion mesh of the green channel of the see-through camera image frame. The distortion mesh of blue channel of the see-through camera image frame is likewise adjusted at block 635. From blocks 633 and 635, finally corrected meshes for rendering the blue and green channels of the VST XR frame may be obtained. At block 640, a VST XR frame comprising data in each of the three color channels may be rendered based on the corrected distortion meshes obtained at blocks 631, 633, and 635.

[0159] Although FIGURE 6 illustrates an exemplary pipeline 600 for vertex pose adjustment with passthrough and time-warp transformations in VST XR, other embodiments or variations are possible and within the contemplated scope of this disclosure. For example, the vertex adjustment values for chromatic aberration could, instead of being performed as part of the first batch of corrections performed at first stage 610, be performed as part of the individual color channel corrections performed in the second stage, shown as block 620.

[0160] FIGURE 7 illustrates an exemplary XR device 700 for vertex pose adjustment with passthrough and time-warp transformation in VST XR. For ease of explanation, the XR device 700 of FIGURE 7 is described as a set of processes that can be implemented using the electronic device 101 in the network configuration 100 of FIGURE 1.

[0161] An extended reality (XR) device 700 may comprise at least one display 710, a see-through camera 720 and at least one processing device 730.

[0162] The at least one display 710 may display the XR virtual frame rendered by the XR device 700.

[0163] The see-through camera 720 may contain at least one lens for capturing data across wide fields of view.

[0164] The processing device 730 may determine a first set of vertex adjustment values of a distortion mesh. The processing device 730 may receive image frame data of a scene captured at a first time and at a first head pose by the see-through camera 720. The processing device 730 may apply the first set of vertex adjustment values of the distortion mesh to the image frame data to obtain intermediate image data;

[0165] The processing device 730 may predict, a second head pose at a second time subsequent to the first time. The processing device 730 may generate, based on the predicted second head pose, a second set of vertex adjustment values of the distortion mesh. The processing device 730 may apply the second set of vertex adjustment values of the distortion mesh to the intermediate image data to generate a rendered virtual frame.

[0166] The processing device 730 may display at the at least one display 710, the virtual frame rendered by the XR device 700 at the second time. The rendered virtual frame may comprise a corrected view of the scene.

[0167] According to an exemplary embodiment, the XR device 700 may further comprise a depth sensor 740 and a motion sensor 750. The processing device 730 may obtain depth data of the scene captured by the depth sensor 740 of the XR device 700 at the first time. The processing device 730 may obtain motion data captured by a motion sensor 750 of the XR device 700 at the first time, and predict the second head pose based on the depth data and the motion data. The second head pose may comprise a prediction of a rotational position and a translational position of the XR device 700 at the second time.

[0168] The operations of the XR device 700 is described in detail through the exemplary method 800 in FIGURE 8 below.

[0169] FIGURE 8 illustrates an exemplary method 800 for vertex pose adjustment with passthrough and time-warp transformation in VST XR. For ease of explanation, the method 800 of FIGURE 8 is described as a set of processes that can be implemented using the electronic device 101 in the network configuration 100 of FIGURE 1 or the XR device 700 of FIGURE 7. However, method 800 may be implemented using any other suitable device(s), such as the server 106, and in any other suitable system(s).

[0170] As shown in FIGURE 8, at step 801, the processing device 730 (for example, processor 120) in FIGURE 1 may generate a distortion map (for example, distortion mesh 203 in FIGURE 2), wherein the distortion map may comprise vertices defining coordinates within image data at locations that, due to intrinsic, static properties of the see-through camera 720 of the XR device 700, are distorted, and shifted from desired locations in the VST XR display 710. Examples of such intrinsic or static sources of distortion may include, without limitation, distortions due to the shape of the lens of the see-through camera 720 from which image data is obtained. Examples of such intrinsic or static sources of distortion may viewpoint disparities arising from the separation between an XR device user's eye(s) and the see-through camera 720. At step 701, a first set of vertex adjustment values of the distortion mesh may be generated - for example, as described with reference to equations (1)-(8) of this disclosure. The first set of vertex adjustment values may define coordinate shifts for the vertices of the distortion map to correct one or more sources of static or intrinsic distortion To reduce latency, step 701 may be performed as part of a calibration or initialization routine for the XR device 700, with the first set of vertex adjustment values stored in a memory for ready access.

[0171] At step 803, the processing device 700 providing the VST XR display 710 may receive image frame data from the see-through camera 720. The image frame data may be captured at a first time associated with a first head pose of a user wearing the XR device 700. Depending on embodiments, the image frame received at step 703 may be received as a straight out of camera ("SOOC") .RAW file. In some embodiments, the image frame may be provided in discrete sets of data corresponding to the component color channels of the color space used by the see-through camera 720. Additionally, in some embodiments, the image data received at step 703 may just be image data of a first camera of a stereoscopic pair of see-through cameras 720, wherein two instances of method 800 are performed in parallel.

[0172] At step 805, the processing device 700 applies a distortion mesh which has been corrected by the first set of vertex adjustment values to obtain intermediate image data which has been corrected for intrinsic and static sources of distortion. Depending on the format of the image frame data received at 803, step 805 can comprise multiple applications of the first of vertex adjustment values. For example, where the image data is provided as multiple individual and discrete sets of image data corresponding to each of a plurality of color channels, vertex adjustment values may have to be applied to each set of image data.

[0173] At step 807, the processing device 700 predicts the head pose of the user at a second time, wherein the second time is subsequent to the first time. The second time can correspond to the estimated time of conclusion of a correction interval. In some embodiments, the duration of the correction interval can be determined by estimating the time to clear one or more major processing bottlenecks (for example, receiving image data from the see-through camera 720, rendering a frame of XR content based on the image data, and / or displaying the rendered frame to the user). Depending on the application, available data, and processing resources, determining the second time can be performed programmatically or dynamically calculated.

[0174] Similarly, depending on, without limitation, the available data regarding the user's head pose at the first time, the sensors available at the XR device 700, and the current processing load at the processing device 700, the user's head can be predicted according to one or more of the following methods: extrapolation from sensor data indicating the direction and magnitude of the change in the user's pose at the first time or by machine learning techniques identifying a predicted movement based on sensor data, frame-over-frame changes in the image data or combinations thereof. Predicting the user's head pose can, in some embodiments, be limited to predicting a rotational change in the user's pose (for example, as described with reference to block 539 of FIGURE 5). Additionally or alternatively, predicting the user's head pose at the second time can include a prediction of translational changes in the user's pose based on the motion sensor 750 including one or more of acceleration sensor (for example, a 6DOF accelerometer) data or depth data (for example, depth data obtained from the depth sensor 740, as like a TOF depth sensor).

[0175] At step 809, a second set of vertex adjustment values, based on the predicted second head pose are generated. In some embodiments, the second set of vertex adjustment values can be generated according to the transformation described with reference to Equations (9)-(13) of this disclosure.

[0176] At step 811, the second set of vertex adjustment values are applied (for example, as described with reference to block 545 of FIGURE 5 and block 640 of FIGURE 6) as part of a rendering operation to generate a virtual frame for display to the user. The rendered virtual frame comprises visual content based on the see-through camera frame received at step 803, and which has been corrected for one or more static or intrinsic distortions created by the see-through-camera 720, and reprojected from the pose-dependent viewpoint at the first time, to a second pose-dependent viewpoint based on the prediction of the user's head pose performed at step 807. At step 811, the processing device can also make further changes to the corrected image data, such as adding items of virtual content into the display 710.

[0177] At step 813, the processing device causes the XR device 700 to display the rendered virtual frame at the second time. By predicting the head pose of the user at step 807, disparities between the pose-dependent viewpoint of the XR display 700 provided at step 813 and the user's native, proprioception-based understanding of the user's pose and viewpoint are minimized, resulting in an improved, less motion sickness-inducing XR experience.

[0178] Although FIGURE 8 illustrates one exemplary method for vertex pose adjustment with passthrough and time-warp transformations in VST XR, various changes may be made to FIGURE 8. For example, while shown as a series of steps, various steps in FIGURE 8 could overlap, occur in parallel, occur in a different order, or occur any number of times (including zero times). Depending on the specific requirements (for example, prioritizing maintaining a steady frame rate) of the implementation, as well as the sensor data and processing capacity of the processing device, operations of method 800 can be sequenced differently or merged. As one specific example, where depth information and translational movement is available, and the processing device(s) include a GPU, applying the first and second sets of vertex adjustment values may be combined in a single step performed as part of a depth-based reprojection performed by a vertex shader of the GPU. In such embodiments, the creation and use of intermediate data, as described with reference to step 705, may be eliminated.

[0179] It should be noted that the functions shown in or described with respect to FIGURES 2 through 8 can be implemented in an electronic device 101, 102, 104, server 106, or other device(s) in any suitable manner. For example, at least some of the functions shown in or described with respect to FIGURES 2 through 8 can be implemented or supported using one or more software applications or other software instructions that are executed by the processor 120 of the electronic device 101, 102, 104, server 106, or other device(s). In other embodiments, at least some of the functions shown in or described with respect to FIGURES 2 through 8 can be implemented or supported using dedicated hardware components. In general, the functions shown in or described with respect to FIGURES 2 through 8 can be performed using any suitable hardware or any suitable combination of hardware and software / firmware instructions. Also, the functions shown in or described with respect to FIGURES 2 through 8 can be performed by a single device or by multiple devices.

[0180] Although this disclosure has been described with exemplary embodiments, various changes and modifications may be suggested to one skilled in the art. It is intended that this disclosure encompass such changes and modifications as fall within the scope of the appended claims.

[0181] It is understood that the specific order or hierarchy of steps, operations, or processes disclosed is an illustration of exemplary approaches. Unless explicitly stated otherwise, it is understood that the specific order or hierarchy of steps, operations, or processes may be performed in different order. Some of the steps, operations, or processes may be performed simultaneously or may be performed as a part of one or more other steps, operations, or processes. The accompanying method claims, if any, present elements of the various steps, operations or processes in a sample order, and are not meant to be limited to the specific order or hierarchy presented. These may be performed in serial, linearly, in parallel or in different order. It should be understood that the described instructions, operations, and systems can generally be integrated together in a single software / hardware product or packaged into multiple software / hardware products.

[0182] The disclosure is provided to enable any person skilled in the art to practice the various aspects described herein. In some instances, well-known structures and components are shown in block diagram form in order to avoid obscuring the concepts of the subject technology. The disclosure provides various examples of the subject technology, and the subject technology is not limited to these examples. Various modifications to these aspects will be readily apparent to those skilled in the art, and the principles described herein may be applied to other aspects.

[0183] All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims.

[0184] The title, background, brief description of the drawings, abstract, and drawings are hereby incorporated into the disclosure and are provided as illustrative examples of the disclosure, not as restrictive descriptions. It is submitted with the understanding that they will not be used to limit the scope or meaning of the claims. In addition, in the detailed description, the description may provide illustrative examples and the various features may be grouped together in various implementations for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed subject matter requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed configuration or operation. The following claims are hereby incorporated into the detailed description, with each claim standing on its own as a separately claimed subject matter.

[0185] The embodiments are provided solely as examples for understanding the invention. They are not intended and are not to be construed as limiting the scope of this invention in any manner. Although some embodiments and examples have been provided, it will be apparent to those skilled in the art based on the disclosures herein that changes in the embodiments and examples shown may be made without departing from the scope of this invention.

[0186] The claims are not intended to be limited to the aspects described herein, but are to be accorded the full scope consistent with the language claims and to encompass all legal equivalents. Notwithstanding, none of the claims are intended to embrace subject matter that fails to satisfy the requirements of the applicable patent law, nor should they be interpreted in such a way.

Claims

1.A method comprising:determining, using at least one processing device of an extended reality (XR) device, a first set of vertex adjustment values of a distortion mesh;receiving, using the at least one processing device, image frame data of a scene captured at a first time and at a first head pose using a see-through camera of the XR device;applying, using the at least one processing device, the first set of vertex adjustment values of the distortion mesh to the image frame data to obtain intermediate image data;predicting, using the at least one processing device, a second head pose at a second time subsequent to the first time;generating, using the at least one processing device, based on the predicted second head pose, a second set of vertex adjustment values of the distortion mesh;applying, using the at least one processing device, the second set of vertex adjustment values of the distortion mesh to the intermediate image data to generate a rendered virtual frame; anddisplaying the rendered virtual frame by the XR device at the second time, the rendered virtual frame comprising a corrected view of the scene.2.The method of Claim 1, wherein:the first set of vertex adjustment values compensates for static differences between the image frame data and the corrected view of the scene; andthe static differences comprise at least one of: a distortion due to a lens of the see-through camera, a chromatic aberration due to the lens of the see-through camera, or a viewpoint difference due to a separation between a location of the see-through camera and a location of a user's eye.3.The method of Claim 1, wherein the second head pose comprises a time-warp transformation of the first head pose.4.The method of Claim 1, further comprising:obtaining depth data of the scene captured by a depth sensor of the XR device at the first time;obtaining motion data captured by a motion sensor of the XR device at the first time; andpredicting the second head pose based on the depth data and the motion data,wherein the second head pose comprises a prediction of a rotational position and a translational position of the XR device at the second time.5.The method of Claim 4, wherein predicting the second head pose comprises performing a depth-based reprojection to predict the translational position of the XR device at the second time.6.The method of Claim 1, wherein applying the first set of vertex adjustment values of the distortion mesh and applying the second set of vertex adjustment values of the distortion mesh are performed by at least one of: a vertex shader or a fragment shader.7.The method of Claim 1, further comprising:estimating a correction interval comprising at least one of: an estimated latency interval for predicting the second head pose, an estimated latency interval for generating the rendered virtual frame, or an estimated latency interval for displaying the rendered virtual frame; anddetermining the second time based on the correction interval.8.An extended reality (XR) device comprising:at least one display;a see-through camera; andat least one processing device configured to:determine a first set of vertex adjustment values of a distortion mesh;receive image frame data of a scene captured at a first time and at a first head pose by the see-through camera;apply the first set of vertex adjustment values of the distortion mesh to the image frame data to obtain intermediate image data;predict, a second head pose at a second time subsequent to the first time;generate, based on the predicted second head pose, a second set of vertex adjustment values of the distortion mesh;apply the second set of vertex adjustment values of the distortion mesh to the intermediate image data to generate a rendered virtual frame; anddisplay at the at least one display, the rendered virtual frame by the XR device at the second time, the rendered virtual frame comprising a corrected view of the scene.9.The XR device of Claim 8, wherein:the first set of vertex adjustment values compensates for static differences between the image frame data and the corrected view of the scene; andthe static differences comprise at least one of: a distortion due to a lens of the see-through camera, a chromatic aberration due to the lens of the see-through camera, or a viewpoint difference due to a separation between a location of the see-through camera and a location of a user's eye.10.The XR device of Claim 8, wherein the second head pose comprises a time-warp transformation of the first head pose.11.The XR device of Claim 8, further comprising:a depth sensor; anda motion sensor,wherein the at least one processing device is further configured to:obtain depth data of the scene captured by the depth sensor at the first time;obtain motion data captured by the motion sensor at the first time; andpredict the second head pose based on the depth data and the motion data,wherein the second head pose comprises a prediction of a rotational position and a translational position of the XR device at the second time.12.The XR device of Claim 11, wherein the at least one processing device is configured to predict the second head pose by performing a depth-based reprojection to predict the translational position of the XR device at the second time.13.The XR device of Claim 8, wherein applying the first set of vertex adjustment values of the distortion mesh and applying the second set of vertex adjustment values of the distortion mesh are performed by at least one of: a vertex shader or a fragment shader.14.The XR device of Claim 8, wherein the at least one processing device is further configured to:estimate a correction interval comprising at least one of: an estimated latency interval for predicting the second head pose, an estimated latency interval for generating the rendered virtual frame, or an estimated latency interval for displaying the rendered virtual frame; anddetermine the second time based on the correction interval.

Images