Straight-through assembly line

A straight-through pipeline processor in HMDs processes images in real-time to reduce latency, addressing user disorientation and discomfort by integrating virtual content with physical environments effectively.

CN119071456APending Publication Date: 2024-12-03APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410692894.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-06-02
Filing Date
2024-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing head-mounted devices (HMDs) in extended reality environments introduce latency due to image processing delays, causing user disorientation and discomfort by processing passthrough images after capture.

Method used

Implement a straight-through pipeline processor that processes captured images in a pipelined manner, allowing simultaneous capture and display of images while applying chromatic aberration correction, blending with virtual content, and hole filling, thereby reducing latency.

Benefits of technology

The solution reduces latency by processing images in real-time, ensuring seamless integration of virtual content with physical environments without user discomfort.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071456A_ABST
    Figure CN119071456A_ABST
Patent Text Reader

Abstract

The invention relates to a through pipeline. A pipelined method of mixing an image with virtual content is performed by a device including an image sensor, a display, a processor, and a non-transitory memory. The method includes capturing a first portion of an image of a physical environment using an image sensor; warping a first portion of the image of the physical environment to generate a warped first portion; mixing the twisted first portion with the first portion of the virtual content to generate a mixed first portion; displaying the mixed first portion on a display; capturing a second portion of the image of the physical environment using the image sensor; warping a second portion of the image of the physical environment to generate a warped second portion; mixing the twisted second portion with a second portion of the virtual content to generate a mixed second portion; and displaying the mixed second portion on the display, wherein the display of the mixed first portion is initiated before the mixing of the twisted second portion is completed.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 470,689, filed on June 2, 2023, which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to systems, methods, and devices for displaying virtual content in conjunction with images of an environment with low latency. Background Art

[0004] In various embodiments, an extended reality (XR) environment is presented by a head mounted device (HMD). Various HMDs include a scene camera that captures an image of the physical environment (e.g., a scene) in which a user is present and a display that displays the image to the user. In some cases, the image (sometimes referred to as a passthrough image) or a portion thereof may be combined with one or more virtual objects to present an XR experience to the user. In other cases, the HMD may operate in a passthrough mode in which the passthrough image or a portion thereof is presented to the user without adding virtual objects. In various embodiments, the passthrough image is processed after being captured and before being displayed. This processing introduces a delay between capture and display, which may cause disorientation and / or discomfort to the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] So that the present disclosure can be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0006] Figure 1 is a block diagram of an exemplary operating environment according to some implementations.

[0007] Figure 2 A functional block diagram of an electronic device according to some implementations is shown.

[0008] Figure 3A A timing diagram of an electronic device according to some specific implementations is shown.

[0009] Figure 3B is a flowchart representation of a method for displaying a through-the-lens image stream with low latency according to some specific implementations.

[0010] Figure 4 is a flowchart representation of a method of performing chromatic aberration correction according to some specific implementations.

[0011] Figure 5 is a flowchart representation of a method for blending an image with virtual content and hole filling color according to some specific implementations.

[0012] Figure 6 is a flowchart representation of a method for pipelined blending of images and virtual content according to some specific implementations.

[0013] Figure 7 is a block diagram of an example controller according to some implementations.

[0014] Figure 8 is a block diagram of an exemplary electronic device according to some implementations.

[0015] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention

[0016] Various embodiments disclosed herein include devices, systems, and methods for performing chromatic aberration correction. In various embodiments, the method is performed by a device including an image sensor, a display, one or more processors, and a non-volatile memory. The method includes capturing an image of a physical environment using the image sensor. The method includes warping the image of the physical environment to correct the distortion caused by the lens of the image sensor and / or the lens of the display to obtain a corrected image. The method includes mixing the corrected image with virtual content to obtain a mixed image including a first color channel and a second color channel. The method includes warping the second color channel to correct the chromatic aberration, thereby obtaining a display image including the first color channel and the warped second color channel. The method includes displaying the display image on the display.

[0017] Various embodiments disclosed herein include devices, systems, and methods for blending an image with virtual content and a hole fill color. In various embodiments, the method is performed by a device including an image sensor, a display, one or more processors, and a non-volatile memory. The method includes capturing an image of a physical environment using the image sensor. The method includes distorting the image of the physical environment to generate a distorted image, wherein the distorted image includes a hole. The method includes obtaining a hole fill color. The method includes blending the distorted image with virtual content to generate a blended image, wherein blending the distorted image includes filling the hole by setting the pixel value of the pixel of the hole to the hole fill color. The method includes displaying the blended image on the display.

[0018] Various embodiments disclosed herein include devices, systems, and methods for pipelined blending of images and virtual content. In various embodiments, the method is performed by a device including an image sensor, a display, one or more processors, and a non-volatile memory. The method includes capturing a first portion of an image of a physical environment using the image sensor. The method includes distorting the first portion of the image of the physical environment to generate a distorted first portion. The method includes blending the distorted first portion with a first portion of virtual content to generate a blended first portion. The method includes displaying the blended first portion on the display. The method includes capturing a second portion of the image of the physical environment using the image sensor. The method includes distorting the second portion of the image of the physical environment to generate a distorted second portion. The method includes blending the distorted second portion with a second portion of the virtual content to generate a blended second portion. The method includes displaying the blended second portion on the display, wherein displaying the blended first portion is initiated before blending the distorted second portion is completed.

[0019] Various embodiments disclosed herein include devices, systems, and methods for displaying a through-image stream with low latency. In various embodiments, the method is performed by a device including an image sensor, a display, one or more processors, and a non-volatile memory. The method includes capturing multiple images of a physical environment at a first frame rate using the image sensor. The method includes transmitting a subset of the multiple images of the physical environment. The method includes receiving a control parameter based on the subset of the multiple images of the physical environment. The method includes processing the multiple images of the physical environment based on the control parameter. The method includes displaying multiple processed images of the physical environment on the display at the first frame rate.

[0020] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors. One or more programs include instructions for executing or causing the execution of any method described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to execute or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and a device for executing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION

[0021] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.

[0022] As described above, processing of the pass-through image may introduce a delay between capturing the pass-through image and displaying the processed pass-through image. The processing may include, for example, filtering, denoising, performing color space conversion, adding virtual content, performing viewpoint correction, and / or correcting lens distortion. Additional processing that may be performed is further described below. To reduce latency, the pass-through image is processed in a pipelined manner, where portions of the pass-through image are processed while other portions are captured.

[0023] Figure 1 1 is a block diagram of an exemplary operating environment 100 according to some implementations. Although relevant features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100 includes a controller 110 and an electronic device 120.

[0024] In some implementations, the controller 110 is configured to manage and coordinate the user's XR experience. In some implementations, the controller 110 includes a suitable combination of software, firmware, and / or hardware. Figure 7 Controller 110 is described in more detail. In some implementations, controller 110 is a computing device located locally or remotely relative to physical environment 105. For example, controller 110 is a local server located within physical environment 105. In another example, controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside physical environment 105. In some implementations, controller 110 is communicatively coupled to electronic device 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE802.16x, IEEE 802.3x, etc.). As another example, controller 110 is included in a housing of electronic device 120. In some implementations, the functionality of controller 110 is provided by electronic device 120 and / or is combined with the electronic device.

[0025] In some implementations, the electronic device 120 is configured to provide an XR experience to the user. In some implementations, the electronic device 120 includes a suitable combination of software, firmware, and / or hardware. According to some implementations, the electronic device 120 presents XR content to the user via a display 122 while the user is physically present within a physical environment 105, which includes a table 107 within a field of view 111 of the electronic device 120. In some implementations, the user holds the electronic device 120 in one or both of his / her hands. In some implementations, when providing XR content, the electronic device 120 is configured to display an XR object (e.g., an XR cylinder 109) and implement video pass-through of the physical environment 105 (e.g., including a representation 117 of the table 107) on the display 122. The following description is relative to Figure 8 The electronic device 120 is described in more detail.

[0026] According to some implementations, the electronic device 120 provides an XR experience to the user while the user is virtually and / or physically present within the physical environment 105 .

[0027] In some implementations, the user wears the electronic device 120 on his / her head. For example, in some implementations, the electronic device includes a head-mounted system (HMS), a head-mounted device (HMD), or a head-mounted housing (HME). Therefore, the electronic device 120 includes one or more XR displays configured to display XR content. For example, in various implementations, the electronic device 120 surrounds the user's field of view. In some implementations, the electronic device 120 is a handheld device (such as a smart phone or a tablet computer) configured to present XR content, and the user no longer wears the electronic device 120 but holds the device with the display facing the user's field of view and the camera facing the physical environment 105. In some implementations, the handheld device may be placed in a housing that can be worn on the user's head. In some implementations, the electronic device 120 is replaced with an XR cabin, housing, or room configured to present XR content, in which the user no longer wears or holds the electronic device 120.

[0028] Figure 2A functional block diagram of an electronic device 200 according to some implementations is shown. The electronic device 200 includes an image sensor 201 that captures an image of a physical environment, a straight-through pipeline processor 202 that processes the captured image, and a display 203 that displays the processed image. The electronic device 200 includes a control processor 204 that generates parameters for controlling the processing of the image captured by the straight-through pipeline processor 202. In turn, the straight-through pipeline processor 202 processes the captured image based on the parameters most recently received from the control processor 204. In various implementations, the straight-through pipeline processor 202 and the control processor 204 are independent processors of the same electronic device 200. In various implementations, the control processor 204 is an independent processor of a different electronic device independent of the electronic device 200.

[0029] The straight-through pipeline processor 202 includes several modules, including a plurality of optional image processing modules 220A to 220F. Each of the optional image processing modules 220A to 220F performs one or more image processing functions on an input image to generate an output image. In various specific implementations, the image processing functions include filtering, noise reduction, dithering, gamma correction, color space conversion, demosaicing, or shifting. The straight-through pipeline processor 202 also includes a first warping module 221A, an optional second warping module 221B, and an optional third warping module 221C. Each of the warping modules 221A to 221C warps the input image into an output image based on a warp definition. The warp definition defines a mapping between a destination pixel position of the output image and a source pixel position of the input image. In particular, although the pixel value of a particular pixel of the output image of one of the optional image processing modules 220A to 220F may be based on the pixel values ​​of other pixels of the input image, the identification of the other pixels is not provided separately. Conversely, the pixel value of a particular pixel of the output image of one of the warping modules 221A-221C may be based on the pixel values ​​of other pixels of the input image defined by the warp definition.

[0030] The image sensor 201 captures an image of the physical environment in which the electronic device 200 exists at a first frame rate. In various specific implementations, the first frame rate is 90 frames per second. The image captured by the image sensor 201 is fed into a first optional image processing module 220A as an input image. In various specific implementations, the image sensor 201 provides the input image to the first optional image processing module 220A in a raw image format. In various specific implementations, the straight-through pipeline processor 202 obtains the captured image via direct memory access. In various specific implementations, the first optional image processing module 220A performs one or more of sensor linearization, gain offset correction, pixel defect correction, chromatic aberration restoration (of lateral chromatic aberration), front-end lateral scaling for one-dimensional highlight restoration, noise reduction (such as raw noise filtering using a 3x3 bilateral filter, fixed pattern noise reduction, and / or multi-band noise reduction), raw vertical filtering, lens shading correction, demosaicing, global and / or local tone mapping, global and / or local contrast enhancement, tone preserving post-processing, sharpening, color post-processing, and highlight restoration on the input image to generate an output image. In various specific implementations, the output image of the first optional image processing module 220A is in a YCbCr image format.

[0031] The output image of the first optional image processing module 220A is fed as an input image into the first warping module 221A at a first frame rate. The video stream is also sampled or sampled by the control processor 204 at a second frame rate that is less than the first frame rate. For example, in various implementations, every other output image or every third output image of the first optional image processing module 220A is sent to the control processor 204. In various implementations, the video stream is sampled asynchronously. In various implementations, the output image is spatially downsampled before being sent to the control processor 204.

[0032] The control processor 204 includes a parameter generation module 241 that generates parameters for controlling the processing of images by the straight-through pipeline processor 204. For example, in various implementations, the parameter generation module 241 generates parameters for controlling image processing performed by the first optional image processing module 220A. In various implementations, the parameter generation module 241 generates warp definitions for the first warp module 221A, the optional second warp module 221B, and / or the optional third warp module 221C.

[0033] although Figure 2A parameter generation module 241 is shown providing parameters to the first optional image processing module 220A, the first warping module 221A, and the optional second warping module 221B, but it should be understood that the parameter generation module 241 may provide parameters or other information to any module of the straight-through pipeline processor 202, the image sensor 201, or the display 203.

[0034] As described above, the output image of the first optional image processing module 220A is fed as an input image into the first warping module 221A at a first frame rate. In various implementations, the first warping module 221A warps the input image into the output image based on a first warp definition that defines a mapping between destination pixel positions of the output image and source pixel positions of the input image. In various implementations, the first warp module 221A receives the first warp definition from the parameter generation module 241 of the control processor 204.

[0035] In various implementations, the mapping is a forward mapping, wherein for each pixel of the input image at a pixel location in the input space, a destination pixel location in the output space of the output image is determined. In various implementations, the warping is a backward mapping, wherein for each pixel of the output image at a pixel location in the output space, a source pixel location in the input space is determined.

[0036] In various implementations, the warp definitions are generated to perform point of view correction (POVC) on the input image to at least partially correct for the difference in viewing angles of the image sensor 201 at the capture time and the display 203 at the display time. In various implementations, the transformation matrices are generated to perform lens distortion correction on the input image to at least partially correct for distortion caused by a lens of the image sensor 201 and / or a lens of the display 203. In various implementations, the warp definitions are generated to perform optical crosstalk correction on the input image to at least partially correct for incomplete isolation of left and right image channels. In various implementations, the warp definitions are generated to perform foveation on the input image. In various implementations, the transformation matrices are generated to perform fewer or additional functions.

[0037] The output image of the first warping module 221A is fed as an input image into a second optional image processing module 220B. In various specific implementations, the second image processing module 220B performs color space conversion, gamma correction and filtering on the input image to generate an output image. In various specific implementations, the output image of the second image processing module 220B is in RGB format.

[0038] The output image of the second optional image processing module 220B, referred to as the pass-through image, is fed into the blending module 223 to be combined with the virtual content. The virtual content is obtained from the rendering module 242 of the control processor 204, which generates images of the virtual content based on a sampled video feed of images captured by the image sensor 201 and processed by the first optional image processing module 220A. To this end, the rendering module 242 receives images or parameters based on images from the parameter generation module 241.

[0039] The display 203 includes M at corresponding positions in the display space. D ×N D In various implementations, display 203 includes two such matrices, one located in front of the user's left eye and the other located in front of the user's right eye.

[0040] In various specific implementations, in order to render the image of the virtual content to be mixed with the pass-through image for display on the display 203, the rendering module 242 is M D ×N D Each pixel of the image generates M D ×N D pixel values. In various implementations, rendering module 242 renders two such images, one for each eye of the user. Thus, each pixel of the rendered image corresponds to a pixel of display 203 having a corresponding position in the display space. Thus, rendering module 242 renders M pixels evenly spaced in a grid pattern in the display space. D ×N D Generate pixel values ​​for each pixel position.

[0041] Rendering M D ×N D pixel values ​​may be computationally expensive. In addition, as the size of the rendered image increases, the amount of bandwidth required to transfer the rendered image from the control processor 204 to the straight-through pipeline processor 202 and the amount of processing required to process the image at the straight-through pipeline processor 202 also increases.

[0042] In various implementations, in order to reduce the size of the rendered image without degrading the user experience, foveal focus (e.g., foveal focused imaging) is used. Foveal focus is a digital image processing technique in which the image resolution or amount of detail varies across the image. Therefore, a foveal focused image has different resolutions at different parts of the image. Humans typically have relatively weak peripheral vision. According to one model, the user's resolvable resolution is maximum at the fovea (e.g., the area where the user is looking) and decreases in an inverse linear manner. Therefore, in one implementation, the displayed image displayed by the display 203 is a foveal focused image having a maximum resolution at the fovea and a resolution that decreases in an inverse linear manner proportional to the distance from the fovea.

[0043] Since some parts of the image have lower resolution, M D ×N D The foveal focused image includes D ×N D Therefore, in various embodiments, the rendering module 242 generates the foveal focused image in the form of a rendered image. D ×N D The rendering module 242 can generate the M image faster and with less processing power (and battery charge) than the non-foveal focused image. D ×N D The fovea focuses the image. In addition, M D ×N D The foveal focused image can be used with a ratio of M D ×N D In other words, M D ×N D The size of the fovea focused image file should be smaller than M D ×N D In various embodiments, various compression techniques are used to compress M D ×N D A foveally focused image will produce a compression ratio of M D ×N D The non-foveal focused image has fewer bits.

[0044] The foveal focus ratio R can be defined as M D ×N D The amount of information in the non-foveal focused image is divided by M D ×N DIn some embodiments, the foveal focus ratio is between 1.5 and 10. For example, in some embodiments, the foveal focus ratio is 2. In some embodiments, the foveal focus ratio is 3 or 4. In some embodiments, the foveal focus ratio is constant between images. In some embodiments, the foveal focus ratio is determined for the image to be rendered. For example, in various embodiments, the amount of information that the rendering module 242 is able to render in a frame period may be reduced due to a thermal event (e.g., when the process of calculating additional pixel values ​​would cause the processor to overheat).

[0045] In some specific implementations, in order to render an image to be displayed on the display 203, the rendering module 242 is M D / R×N D Each pixel of the R distorted images generates M D / R×N D Each pixel of the distorted image corresponds to a larger area than the pixel of the display 203 at the corresponding position in the display space. Therefore, the rendering module 242 renders M pixels in the display space that are not evenly distributed in the grid pattern. D / R×N D Each of the / R positions generates a pixel value. The corresponding area in the display space corresponding to each pixel value is defined by the corresponding position in the display space (rendering position) and the scaling factor (or a set of horizontal scaling factor and vertical scaling factor).

[0046] In various implementations, the rendering module 242 generates the distorted image in the form of a rendered image. In various implementations, the distorted image includes M pixels evenly spaced in a grid pattern in a distorted space different from the display space. D / R×N D / M in R positions D / R×N D Specifically, the distorted image includes M pixels in the display space that are not evenly distributed in the grid pattern. D / R×N D / M in R positions D / R×N D / R pixel values. Therefore, although the resolution of the warped image is uniform in the warp space, the resolution varies in the display space.

[0047] The output image of the rendering module 242 is fed as an input image to the third optional image processing module 220C. The output image of the third optional image processing module 220C is fed to the optional second warping module 221B, which warps the input image into an output image based on the second warp definition. In various specific implementations, the optional second warp module 221B receives the second warp definition from the parameter generation module 241 of the control processor 204.

[0048] In various implementations, the warp definition is generated to perform point of view correction (POVC) on the input image to at least partially correct for the difference in viewing angle between the virtual camera at rendering time and the display 203 at display time. In various implementations, the transformation matrix is ​​generated to perform lens distortion correction on the input image to at least partially correct for distortion caused by the lens of the display 203. In various implementations, the warp definition is generated to perform optical crosstalk correction on the input image to at least partially correct for incomplete isolation of left and right image channels. In various implementations, the transformation matrix is ​​generated to perform fewer or additional functions. The output image of the optional second warp module 221B is fed as an input image into the optional fourth image processing module 220D.

[0049] The output image of the fourth optional image processing module 220D, referred to as the virtual content image, is fed into the blending module 223 to be combined with the pass-through image.

[0050] In various implementations, the blending module 223 combines the through image and the virtual content image to generate a blended image. In various implementations, the through image, the virtual content image, and the blended image have the same size and image format. For example, in various implementations, the virtual content image is an RGB format. D / R×N D Therefore, in various implementations, the pass-through image is M in RGB format. D / R×N D In various specific implementations, the output image of the first image processing module 220A is an M image in a YCbCr format or an RGB format. C ×M N The first distortion module 221A distorts the input image to focus the central fovea of ​​the input image to M. D / R×N D / R images, and in various specific implementations, the optional second image processing module 220B performs color space conversion to convert the direct image into RGB format.

[0051] In various specific implementations, the blending module 223 generates a blended image by combining the through image and the virtual content image according to an alpha mask of the same size as the virtual content image. For example, the pixel value of a particular pixel at a pixel position of the blended image is equal to a weighted average of the pixel value of the corresponding pixel at the pixel position of the through image and the pixel value of the corresponding pixel at the pixel position of the virtual content image, where the weight is defined by the alpha value of the corresponding pixel at the pixel position of the alpha mask.

[0052] In various implementations, the blending module 223 generates an alpha mask based on a rendered alpha mask received from the rendering module 242 and a matte alpha mask received from the mask generation module 243. The rendered alpha mask describes the opacity of the virtual content, ranging from 0 for invisible to 1 for opaque, and includes intermediate values ​​for semi-transparent content (such as virtual glass). The matte alpha mask describes the areas of the virtual content that will be occluded by dynamic objects in the physical environment (such as the hands of a user or other person). In various implementations, the mask generation module 243 generates the matte alpha mask by applying mask coefficients from a previously captured image to a currently captured image. Thus, the mask generation module 243 predicts the position of the dynamic object in the currently captured image. In various implementations, the mask generation module 243 predicts the position of the dynamic object using other methods, such as re-projection based on the predicted pose and movement of the dynamic object.

[0053] In various implementations, the straight-through pipeline processor 202 includes an optional third warp module 221C between the mask generation module 243 and the blending module 223, which warps the input mask alpha mask into an output mask alpha mask based on a third warp definition. In various implementations, the optional third warp module 221B receives the third warp definition from the parameter generation module 241 of the control processor 204.

[0054] In various implementations, the first warping module 221A performs a forward mapping, wherein for each pixel of the input image at a pixel location in the input space, a destination pixel location is determined in the output space of the output image. However, in some cases, not every destination pixel location is filled, resulting in one or more holes. In various implementations, the first warping module 221A performs a backward mapping, wherein for each pixel of the output image at a pixel location in the output space, a source pixel location is determined in the input space. However, in some cases, the source pixel location cannot be identified or does not have a pixel value, resulting in one or more holes.

[0055] In various implementations, the first warping module 221A uses interpolation to fill the hole. In various implementations, once the blending module 223 blends the hole filling virtual content with the pass-through image, the rendering module 242 renders the hole filling virtual content to cover the hole. In various implementations, the hole filling virtual content is the size and shape of the hole. In various implementations, the hole filling virtual content is based on a previously captured image of the physical environment. In various implementations, the hole filling virtual content is a hole filling color (e.g., a color triplet).

[0056] In various implementations, the hole fill color is a default hole fill color, which can be, for example, white, black, or gray. In various implementations, the hole fill color is the average of the through image (or the average of a region of the through image, such as a region around the user's gaze position). In various implementations, the hole fill color is user-defined or application-defined.

[0057] The blending module 223 feeds the blended image as an input image to the fifth optional image processing module 220E. In various implementations, the fifth optional image processing module 220E performs gamma correction and filtering on the input image to generate an output image.

[0058] The output image of the fifth optional image processing module 220E is fed as an input image into a chromatic aberration correction (CAC) module 225. In various implementations, the first warping module 221A, the optional second warping module 221B, and / or the optional third warping module 221B perform a transformation to correct for lens distortion caused by the lens of the image sensor 201 and / or the lens of the display 203. However, the lens distortion caused by the lens of the image sensor 201 and / or the lens of the display 203 varies with wavelength (e.g., color), which is called chromatic aberration. The CAC module 225 performs a warping function on at least one color channel of the output image to correct for chromatic aberration. In various implementations, the CAC module 225 performs a warping function on less than all color channels of the output image. For example, in various implementations, the output image has three color channels and the CAC module 225 performs a warping function on only one or two of the color channels. For example, in various implementations, the output image has a red channel, a green channel, and a blue channel, and the CAC module 225 performs the warping function only on the red channel and the blue channel.

[0059] In various implementations, the first warping module 221A, the optional second warping module 221B, and / or the optional third warping module 221B perform a transformation to correct lens distortion so that a particular color (eg, green) is least distorted. The CAC module 225 then performs a transformation on the red and blue channels to correct chromatic aberration.

[0060] The output image of the CAC module 225 is fed as an input image into the optional sixth image processing module 220F. The optional sixth image processing module 220F performs optical crosstalk correction, dithering and shifting on the input image to generate an output image. The output image of the optional sixth image processing module 220F (referred to as a display image) is fed to the display 203 and displayed by the display 203 at a first frame rate.

[0061] Because the straight-through pipeline processor 202 processes the captured image based on the parameters most recently received from the control processor 204, if the control processor 204 does not provide updated parameters for the current frame, the captured image is still processed (using the parameters of the previous frame) and displayed without introducing delay. Therefore, if an error occurs at the control processor 204 (e.g., the control processor 204 crashes) and the control processor 204 fails to provide the updated parameters to the straight-through pipeline processor 202, the captured image is still processed by the straight-through pipeline processor 202 and displayed at the first frame rate.

[0062] In various implementations, the image sensor 201 is a rolling shutter camera. Therefore, in various implementations, the image sensor 201 captures images over a capture period. For example, in various implementations, each row of the image sensor 201 is exposed over a different exposure period having an exposure period length, and after the exposure period, the resulting values ​​are read out over a corresponding readout period having a readout period length. The exposure period of each row after the first row starts the readout period length after the start of the exposure period of the previous row.

[0063] Similarly, in various implementations, the display 203 is a scrolling display. Thus, in various implementations, the display 203 displays the display image over a display time period. For example, in various implementations, each row of the display image is loaded at a different loading time period having a loading time period length, and after the loading time period, the row is sustained over a corresponding duration period having a duration period length. The loading time period for each row after the first row starts the loading time period length after the loading time period for the previous row starts.

[0064] Therefore, in various specific implementations, the processing of the earlier captured portion of the captured image can begin before the later captured portion of the captured image is captured. In addition, the processing of generating a display image can continue on the later displayed portion of the display image while displaying the earlier displayed portion of the display image.

[0065] In various implementations, the parameter generation module 241 generates parameters based on predicted conditions for display time within a display time period. In various implementations, the display time is the middle of the display time period (or in various implementations, the middle of the duration period). For example, in various implementations, the parameter generation module 241 generates parameters based on predicted pupil position for display time, predicted gaze for display time, or predicted device posture for display time. In various implementations, the parameter generation module 241 predicts pupil position, gaze, and / or posture for display time based on previously captured images.

[0066] In various implementations, the parameter generation module 241 generates a first distortion definition to perform viewpoint correction based on the predicted pose of the device at display time. In particular, the parameter generation module 241 generates the first distortion definition based on the difference between the predicted pose at display time and the predicted pose at capture time within the capture time period (due to the rolling shutter camera, the difference may be different for different parts of the image). In various implementations, the parameter generation module 241 generates the first distortion definition to perform foveal focus based on the predicted gaze at display time. In various implementations, the parameter generation module 241 generates the distortion definition to perform lens distortion correction based on the predicted pupil position at display time. In various implementations, the parameter generation module 241 provides parameters to the optional sixth image processor 220F to perform optical crosstalk correction based on the predicted pupil position at display time.

[0067] Figure 3A An electronic device (eg, Figure 2 The timing diagram 301 of the electronic device 200 is shown. The time is divided into a plurality of frames, each frame having a frame time period length T F During the first frame (from t0 to t1 = t0 + T F ), the image sensor has a first capture time period length T C1 The first capture time period (from t0 to t0+T C1 ) captures a first image. As described above, in various embodiments, the image sensor is a rolling shutter camera. For example, in various embodiments, the first exposure time period length T x1 The n rows of the image sensor are exposed at different exposure time periods (in Figure 3A Each row in the figure is shown as 5 rows, and after the exposure period, a readout period length T r The exposure time period of each row after the first row starts after the exposure time period of the previous row. The readout time period length T r .

[0068] During the first frame, the warp generator is in a state with a warp generation time period length T G The first distortion generation time period (from t0 to t0+T G ). In various implementations, the first warp definition is generated to perform viewpoint correction based on a predicted pose of the device at display time (in a first display time period during the second frame), lens distortion correction based on a predicted pupil position at display time, foveal focus based on a predicted gaze at display time, or any other transformation. In various implementations, during the first warp generation time period, the warp generator also generates a warp definition for an image and / or a mask alpha mask of the virtual content image.

[0069] Furthermore, starting from the first frame, after a number of lines of the first image have been read out, the warp processor uses the first warp definition to read the first image over a warp processing time period length T W In various implementations, each row has a row warping time period length T w The row warping process time period of each row after the first row starts after the row warping process time period of the previous row starts. r .

[0070] In various implementations, the first warping processing period is before the end of the first capture period (at t c2 ) starts (at t w1 ).

[0071] In various implementations, during the first warp processing time period, the warp processor also warps the image of the virtual content of the rendering module (using the corresponding warp definition). In various implementations, during the first warp processing time period, the warp processor also warps the mask alpha mask of the mask generation module (using the corresponding warp definition). In various implementations, such as the through image, each row is warped in a different row warp processing time period. Therefore, the warping of the through image, the virtual content image and / or the mask alpha mask can be performed in parallel.

[0072] Also starting in the first frame, after the first row of the first image has been warped (and, in some implementations, subjected to additional non-warping image processing, such as color space conversion, filtering, and / or gamma correction), the blending processor performs a blending process for a blending process period of length T B In various embodiments, the first mixed image is generated during the first mixed processing time period having a length of T. bIn different line mixing processing time periods, each line of the first image is mixed with the corresponding line of the virtual content according to the corresponding line of the mask α mask. The line mixing processing time period of each line after the first line starts after the line mixing processing time period of the previous line starts to read out the time period length T r .

[0073] In various implementations, the first blending process period ends at the end of the first warping process period (at t w2 ) before (at t b1 ).

[0074] During the second frame, the display is in a display period with a length of T D The first display time period (from t1 to t1+T D ) starts display of the first mixed image. In various specific implementations, the display is a scrolling display. For example, in various specific implementations, in a loading time period having a length T l m rows of the first mixed image are loaded at different loading time periods (in Figure 3A Each row in the table is shown as 5 rows in FIG. 1 , and after the loading period, the row is in a duration period length T p The loading time period of each row after the first row starts after the loading time period of the previous row starts. The loading time period length T l .

[0075] During the second frame (from t1 to t2 = t1 + T f ), the image sensor has a second capture time period length T C2 The second capture time period (from t1 to t1+T C2 ) captures the second image. In various embodiments, due to a longer or shorter second exposure time period length T x2 , the length of the second capture time period T C2 Than the first capture time period length T C1 Long or short.

[0076] During the second frame, the warp generator is operated in a state with a warp generation time period length T G The second distortion generation time period (from t1 to t1+T G ). In various implementations, the second warp definition is generated to perform viewpoint correction based on a predicted pose of the device at display time (within a second display time period during the third frame), lens distortion correction based on a predicted pupil position at display time, foveal focus based on a predicted gaze at display time, or any other transformation.

[0077] Furthermore, starting from the second frame, after a number of lines of the second image have been read out, the warp processor uses the second warp definition to read the second image over a warp processing time period length T W A second warped image is generated during a second warping processing time period.

[0078] Also starting in the second frame, after the first row has been warped (and, in some implementations, subjected to additional non-warping image processing, such as color space conversion, filtering, and / or gamma correction), the blending processor performs a blending process with a blending processing period length T B A second mixed image is generated during a second mixing processing time period.

[0079] During the third frame, the display is in a display period with a length of T D The second display time period (from t2 to t2+T D ) starts display of the second warped image.

[0080] although Figure 3A Some processing that can be applied to an image is shown, but in various implementations, additional processing is performed. For example, for the first row of the first image, after capturing the first row (at t x1 +T r ) and before the first line of distortion (at t w1 ), which can be compared with Figure 2 The image processing is performed as described in the first optional image processing module 220A. For another example, for the first row of the first image, after the first row has been distorted (at t w1 +T w ) and before the first row is mixed (at t b1 ), which can be compared with Figure 2 The image processing is performed as described by the second optional image processing module 220B. For another example, for the first line of the first image, after the first line has been mixed (at t b1 +T b ) and before the first row is displayed (at t1), it can be as relative to Figure 2 The fifth optional image processing module 220E, the CAC module 225 and the sixth optional image processing module 220F perform image processing as described.

[0081] As noted above, in various implementations (and as Figure 3A ), the warping of the first row of the image begins before the capture of the last row is completed; the blending of the first row begins before the warping of the last row is completed, and the display of the first row begins before the blending of the last row is completed. In various specific implementations (although not shown in Figure 3A ), the blending can be performed quickly enough so that the display of the first row begins before the distortion of the last row ends.

[0082] In various implementations, the first capture time period overlaps with the first warp processing time period, the first warp processing time period overlaps with the first blending processing time period, and the first blending processing time period overlaps with the first display time period. In various implementations, blending can be performed fast enough so that the first warp processing time period overlaps with the first display time period.

[0083] In various implementations, the time between starting to capture the first line of the image and completing the mixing of the last line of the image is greater than the frame period. Similarly, in various implementations, the time between starting to capture the first line of the image and starting to display the last line of the image is greater than the frame period.

[0084] Although the above has been described with respect to the rows of the image Figure 3A , but it should be appreciated that similar pipelining may be performed for slices, tiles, or other portions of an image having different shapes.

[0085] Figure 3B 300 is a flowchart representation of a method for displaying a through-the-lens image stream with low latency according to some specific implementations. In various specific implementations, the method 300 is performed by a device (e.g., Figure 1 In some implementations, the method 300 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 300 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0086] The method 300 begins at block 310, where a device captures a plurality of images of a physical environment at a first frame rate using an image sensor. Figure 2 , the image of the physical environment may correspond to an output image of the image sensor 201 or the first optional image processing module 220A. Thus, in various implementations, capturing multiple images includes capturing raw images and performing one or more image processing functions (e.g., as described above with respect to the first optional image processing module 220A) on the raw images to generate an image of the physical environment.

[0087] Capturing images of the physical environment at a first frame rate. For example, in various implementations, images are captured at a first frame rate of 90 frames per second. Thus, 90 images of the physical environment are captured per second. In addition, a corresponding start time for capturing each of the plurality of images is a frame period after a corresponding start time for capturing a previous image in the plurality of images.

[0088] Method 300 continues at block 320, where the device transmits a subset of the plurality of images of the physical environment. Figure 2 In the embodiment of the present invention, the straight-through pipeline processor 202 transmits a subset of the output images from the first optional image processing module 220A to the control processor 204. In various embodiments, transmitting the subset of the plurality of images of the physical environment includes transmitting the subset of the plurality of images of the physical environment at a second frame rate that is less than the first frame rate. For example, in various embodiments, every other or every third image is transmitted. In various embodiments, transmitting the subset of the plurality of images of the physical environment includes transmitting a downsampled version of at least one of the plurality of images of the physical environment.

[0089] The method 300 continues at block 330, where the device receives a control parameter based on a subset of the plurality of images of the physical environment. Figure 2 In the embodiment, the straight-through pipeline processor 202 receives control parameters from the control processor 204. In various embodiments, the control parameters include parameters for one or more image processing functions. Figure 2 In block 310, the first image processing module 220A receives the control parameters from the parameter generation module 241. Thus, a feedback loop is generated between the first optional image processing module 220A and the parameter generation module 241. Thus, in various implementations, capturing (in block 310) at least one of the plurality of images of the physical environment is based on receiving (in block 330) the control parameters based on another of the plurality of images of the physical environment.

[0090] In various implementations, control parameters for the current frame are based on predicted conditions at display time of the current frame. In various implementations, the method includes predicting conditions for the current frame at display time based on measurements made during one or more previous frames. In particular, in various implementations, the control parameters are based on a predicted posture of the user's eyes. For example, in various implementations, the control parameters are based on a predicted posture of the user's eyes relative to the device (e.g., predicted pupil position). In various implementations, the control parameters are based on a predicted posture of the user's eyes relative to the user's head (e.g., predicted gaze). In various implementations, the control parameters are based on a predicted posture of the user's eyes relative to the physical environment (e.g., a predicted posture of the device, where the device, the user's head, and the user's eyes are in space).

[0091] The method 300 continues in block 340, where the device processes multiple images of the physical environment based on the control parameter. In various implementations, the control parameter includes a distortion definition. Thus, in various implementations, processing the multiple images of the physical environment based on the control parameter includes distorting at least one of the multiple images of the physical environment based on the distortion definition. In various implementations, the control parameter includes virtual content. Thus, in various implementations, processing the multiple images of the physical environment includes combining at least one of the multiple images of the physical environment with virtual content. In various implementations, the control parameter includes an alpha mask. Thus, in various implementations, processing the multiple images of the physical environment includes combining at least one of the multiple images of the physical environment with another image (e.g., of virtual content) according to the alpha mask. In various implementations, the control parameter includes a hole fill color. Thus, in various implementations, processing the multiple images of the physical environment includes filling holes in at least one of the multiple images of the physical environment with the hole fill color.

[0092] The method 300 continues in block 350, where the device displays the plurality of processed images of the physical environment on the display at the first frame rate. In various implementations, a respective start time of displaying each of the plurality of processed images of the physical environment is a fixed time after a respective start time of capturing a corresponding one of the plurality of images of the physical environment. In particular, the fixed time is a fixed frame period (the inverse of the first frame rate).

[0093] In contrast, in various implementations, the respective start times for transmitting at least some of the subset of the plurality of images of the physical environment are variable times after the respective start times for capturing corresponding ones of the plurality of images of the physical environment. For example, the images may be transmitted after a variable capture time period (attributable to a variable exposure time period). As another example, the images may be transmitted after a variable processing time period to perform one or more image processing functions on the raw images. Similarly, the respective start times for receiving control parameters based on at least some of the subset of the plurality of images of the physical environment are variable times after the respective start times for capturing each of the plurality of images of the physical environment. Thus, while the capturing (in block 310) and display (in block 350) are synchronized, thereby ensuring that each captured image is processed and displayed after a fixed time frame period, the transmitting (in block 320) and receiving (in block 330) are performed asynchronously.

[0094] Figure 4 4 is a flowchart representation of a method for performing chromatic aberration correction according to some specific implementations. In various specific implementations, method 400 is performed by a device including an image sensor, a display, one or more processors, and a non-volatile memory (e.g., Figure 1In some implementations, the method 400 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 400 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., a memory).

[0095] Method 400 begins in block 410, where a device captures an image of a physical environment using an image sensor. Figure 2 In , the image of the physical environment may correspond to the output image of the image sensor 201 or the first optional image processing module 220A.

[0096] Method 400 continues at block 420, where the device distorts the image of the physical environment to correct for distortion caused by a lens of the image sensor and / or a lens of the display to obtain a corrected image. Figure 2 , the corrected image may correspond to the output image of the first warping module 221A or the second optional image processing module 220B.

[0097] In various implementations, the distorted image of the physical environment is based on a distortion of the image of the physical environment by a lens of the image sensor. In various implementations, the distorted image of the physical environment is based on a distortion of the displayed image by a lens of the display. To this end, in various implementations, the distorted image of the physical environment is based on a position of a pupil of a user's eye relative to a lens of the display.

[0098] In various implementations, the distortion caused by the lens of the image sensor and / or the lens of the display is color-dependent. Thus, in various implementations, the image of the distorted physical environment is based on color. For example, in various implementations, the image of the physical environment is based on the color of a color channel, for example to minimize distortion on the color channel.

[0099] In various implementations, distorting the first color channel is performed as part of the transformation to perform viewpoint correction to at least partially correct for differences in viewing angles between the image sensor and the display. In various implementations, distorting the first color channel is performed as part of the transformation to perform foveal focusing of the image of the physical environment.

[0100] The method 400 continues in block 430, where the device blends the corrected image with the virtual content to obtain a blended image including a first color channel and a second color channel. In various implementations, distorting the image of the physical environment (in block 420) is based on the color of the first color channel. In various implementations, distorting the image of the physical environment (in block 420) reduces distortion of the first color channel more than distortion of the second color channel. In various implementations, blending the corrected image of the physical environment with the virtual content includes performing a color space conversion on the corrected image so that the converted image has the same image format as the virtual content.

[0101] In various specific implementations, mixing the corrected image of the physical environment with the virtual content includes: for a specific pixel of the image at a pixel position, generating a weighted average of the pixel value of the corresponding pixel at the pixel position of the partially corrected image of the physical environment and the pixel value of the corresponding pixel at the pixel position of the virtual content, wherein the weight is defined by the alpha value of the corresponding pixel at the pixel position of an alpha mask.

[0102] Thus, in various implementations, the first color channel is a weighted average of the first color channel of the corrected image and the corresponding first color channel of the virtual content (based on the first alpha mask), and the second color channel is a weighted average of the second color channel of the corrected image and the corresponding second color channel of the virtual content (based on the second alpha mask, which may be the same as or different from the first alpha mask).

[0103] The method 400 continues at block 440, where the device distorts the second color channel to correct the chromatic aberration, thereby obtaining a displayed image including the first color channel and the distorted second color channel. Figure 2 , the displayed image may correspond to the output image of the CAC module 225 or the sixth optional image processing module 220F.

[0104] Thus, in various implementations, all color channels are warped (in block 420) prior to blending (in block 430), and the second color channel is further warped (in block 440) after blending (in block 430), while the first color channel is not further warped (in block 440) after blending (in block 430). Thus, in various implementations, warping the second color channel does not include warping the first color channel.

[0105] In various implementations, the blended image further includes a third color channel. In various implementations, the third color channel is distorted (in block 440) after blending (in block 430). Thus, in various implementations, distorting the second color channel includes distorting the third color channel. In various implementations, the third color channel is not distorted (in block 440) after blending (in block 430). Thus, in various implementations, distorting the second color channel does not include distorting the third color channel.

[0106] For example, in various implementations, the mixed image includes a red channel, a blue channel, and a green channel, and distorting the second color channel includes distorting the red channel and the blue channel without distorting the green channel. As another example, in various implementations, the mixed image includes a luma channel, a first chroma channel, and a second chroma channel, and distorting the second color channel includes distorting the first chroma channel without distorting the luma channel or the second chroma channel.

[0107] Method 400 continues at block 450 where the device displays the display image on a display.

[0108] Figure 5 5 is a flowchart representation of a method for blending an image with hole-filling virtual content according to some specific implementations. In various specific implementations, the method 500 is performed by a device including an image sensor, a display, one or more processors, and a non-volatile memory (e.g., Figure 1 In some implementations, the method 500 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 500 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0109] Method 500 begins at block 510, where a device captures an image of a physical environment using an image sensor. Figure 2 In , the image of the physical environment may correspond to the output image of the image sensor 201 or the first optional image processing module 220A.

[0110] Method 500 continues at block 520, where the device distorts the image of the physical environment to generate a distorted image. Figure 2 In the embodiment, the distorted image may correspond to the output image of the first distortion module 221A or the second optional image processing module 220B. In various specific implementations, the distorted image includes a hole. In various specific implementations, the distorted image includes a plurality of holes.

[0111] In various specific implementations, warping the image of the physical environment includes performing a forward mapping, wherein for each pixel of the image of the physical environment at a pixel location in an input space, a destination pixel location is determined in an output space of the warped image. However, in some cases, not each destination pixel location is filled, resulting in one or more holes. In various specific implementations, warping the image of the physical environment includes performing a backward mapping, wherein for each pixel of the warped image at a pixel location in an output space, a source pixel location is determined in the input space. However, in some cases, the source pixel location cannot be identified or the source pixel location does not have a pixel value (e.g., the source pixel location is outside the image of the physical environment), resulting in one or more holes.

[0112] In various implementations, distorting the image of the physical environment includes identifying pixels of holes. For example, in various implementations, distorting the image of the physical environment includes determining a pixel value for each pixel of the distorted image, wherein the pixel value of the hole is a hole value, which can be 0, NaN, or any other value that identifies the pixel as a pixel of a hole.

[0113] Method 500 continues in box 530, where the device obtains a hole fill color. In various specific implementations, the hole fill color is represented by a color triplet (e.g., an RGB value or a YCbCr value). In various specific implementations, the hole fill color is a default hole fill color, which can be, for example, white, black, or gray. In various specific implementations, the hole fill color is an average of at least a portion of the image of the physical environment. For example, in various specific implementations, the hole fill color is an average of the entire image of the physical environment. As another example, in various specific implementations, the hole fill color is an average of an area surrounding the user's gaze location. In various specific implementations, the hole fill color is user-defined or application-defined.

[0114] The method 500 continues at block 540, where the device blends the distorted image with virtual content to generate a blended image, where blending the distorted image includes filling the holes by setting pixel values ​​of pixels of the holes to a hole fill color. In various implementations, the virtual content includes hole-filling virtual content having the hole fill color and the size and shape of the holes. For example, in Figure 2 , the mixed image may correspond to the output image of the mixing module 223, the fifth optional image processing module 220E, the CAC module 225, or the sixth optional image processing module 220F.

[0115] Method 500 continues at block 550 where the device displays the display image on a display.

[0116] Figure 66 is a flowchart representation of a method for pipelined blending of images and virtual content according to some specific implementations. In various specific implementations, method 600 is performed by a device including an image sensor, a display, one or more processors, and a non-volatile memory (e.g., Figure 1 In some implementations, the method 600 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 600 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0117] Method 600 begins at block 610, where a device captures a first portion of an image of a physical environment using an image sensor. Figure 3A The first part of the image of the physical environment may correspond to the time between t0 and t0+T x1 +T r between the first row of the first image captured.

[0118] The method 600 continues at block 620, where the device distorts a first portion of the image of the physical environment to generate a distorted first portion. Figure 3A The first part of the distortion may correspond to the w1 With t w1 +T w The first row of the first warped image is warped between the two.

[0119] In various implementations, distorting the first portion includes distorting the first portion to correct for a difference in viewing angle between an image sensor and a display. In various implementations, distorting the first portion includes distorting the first portion to correct for distortion caused by a lens of the image sensor and / or a lens of the display. In various implementations, distorting the first portion includes distorting the first portion to correct for optical crosstalk. In various implementations, distorting the first portion includes distorting the first portion to perform foveal focusing.

[0120] In various implementations, warping the first portion includes warping the first portion of the virtual content. In various implementations, warping the first portion includes warping the first portion of the alpha mask.

[0121] The method 600 continues at block 630, where the device blends the warped first portion with the first portion of the virtual content to generate a blended first portion. Figure 3A The first part of the mixture may correspond to the b1 With t b1 +T b In various implementations, blending the first portion is based on the warped first portion of the alpha mask.

[0122] Method 600 continues at block 640 where the device displays the first portion of the mixture on a display. Figure 3A In , the mixed first portion may correspond to the first line of the first mixed image displayed between t1 and t2.

[0123] Method 600 continues at block 650, where the device captures a second portion of the image of the physical environment using the image sensor. Figure 3A , the second portion of the image of the physical environment may correspond to the image of the physical environment at T c1 –T x1 –T r With T c1 between the last row of the first image captured.

[0124] Method 600 continues at block 660, where the device distorts a second portion of the image of the physical environment to generate a distorted second portion. Figure 3A The second part of the distortion may correspond to the w2 –T w With t w2 Warp between the last row of the first warped image.

[0125] Method 600 continues at block 670, where the device blends the warped second portion with the second portion of the virtual content to generate a blended second portion. Figure 3A The second part of the mixture may correspond to the b2 –T b With t b2 The last row of the first blended image is blended between.

[0126] Method 600 continues at block 680, where the device displays the second portion of the blend on a display, wherein displaying the first portion of the blend (at block 640) is initiated before blending the second portion of the warp (at block 670) is completed. Figure 3A The second part of the mixture corresponds to the time at t1+T d –T e –T p With t1+T d The last line of the first mixed image is displayed between.

[0127] In various implementations, blending the first portion of the warping (in block 630) is initiated before the second portion of the image of the warped physical environment is completed (in block 660). In various implementations, the first portion of the image of the warped physical environment is initiated (in block 620) before the second portion of the image of the captured physical environment is completed (in block 650). In various implementations, blending may be performed quickly enough that display of the blended first portion is initiated (in block 640) before the second portion of the image of the warped physical environment is completed (in block 660).

[0128] In various implementations, the time between starting to capture the first portion of the image of the physical environment and completing the blending of the second warped portion is greater than a frame period, e.g., the amount of time between the start of the capture time period. Similarly, in various implementations, the time between starting to capture the first portion of the image of the physical environment and starting to display the second blended portion is greater than a frame period, e.g., the amount of time between the start of the capture time period.

[0129] In various implementations, the first portion of the distortion includes one or more holes. In various implementations, method 600 further includes obtaining a hole fill color. In various implementations, blending the first portion of the distortion with the first portion of the virtual content includes filling at least one of the one or more holes by setting pixel values ​​of pixels of at least one of the one or more holes to the hole fill color.

[0130] Figure 7 is a block diagram of an example of a controller 110 according to some implementations. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the controller 110 includes one or more processing units 702 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 706, one or more communication interfaces 708 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 710, a memory 720, and one or more communication buses 704 for interconnecting these components and various other components.

[0131] In some implementations, the one or more communication buses 704 include circuits for interconnecting system components and controlling communications between system components. In some implementations, the one or more I / O devices 706 include at least one of a keyboard, a mouse, a touch pad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0132] The memory 720 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, the memory 720 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 720 optionally includes one or more storage devices remotely located from the one or more processing units 702. The memory 720 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 720 or the non-transitory computer-readable storage medium of the memory 720 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 730 and an XR experience module 740.

[0133] The operating system 730 includes processes for handling various basic system services and for performing hardware-related tasks. In some specific implementations, the XR experience module 740 is configured to manage and coordinate single or multiple XR experiences of one or more users (e.g., a single XR experience of one or more users, or multiple XR experiences of corresponding groups of one or more users). To this end, in various specific implementations, the XR experience module 740 includes a data acquisition unit 742, a tracking unit 744, a coordination unit 746, and a data transmission unit 748.

[0134] In some specific implementations, the data acquisition unit 742 is configured to at least Figure 1 The electronic device 120 obtains data (eg, presentation data, interaction data, sensor data, location data, etc.). To this end, in various specific implementations, the data acquisition unit 742 includes instructions and / or logic components for instructions and heuristics and metadata for heuristics.

[0135] In some implementations, the tracking unit 744 is configured to map the physical environment 105 and to track at least the electronic device 120 relative to the physical environment 105. Figure 1 The location / position of the physical environment 105. To this end, in various implementations, the tracking unit 744 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0136] In some implementations, the coordination unit 746 is configured to manage and coordinate the XR experience presented to the user by the electronic device 120. To this end, in various implementations, the coordination unit 746 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0137] In some implementations, the data transfer unit 748 is configured to transfer data (e.g., presentation data, location data, etc.) to at least the electronic device 120. To this end, in various implementations, the data transfer unit 748 includes instructions and / or logic components for instructions as well as heuristics and metadata for the heuristics.

[0138] Although the data acquisition unit 742, tracking unit 744, coordination unit 746, and data transfer unit 748 are shown as residing on a single device (e.g., controller 110), it should be understood that in other specific implementations, any combination of the data acquisition unit 742, tracking unit 744, coordination unit 746, and data transfer unit 748 may be located in separate computing devices.

[0139] also, Figure 7 It serves more as a functional description of various features that may be present in a particular implementation, rather than as a structural schematic diagram of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 7 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a specific implementation.

[0140] Figure 8is a block diagram of an example of an electronic device 120 according to some implementations. While some specific features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. For this purpose, as a non-limiting example, in some embodiments, the electronic device 120 includes one or more processing units 802 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 806, one or more communication interfaces 808 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 810, one or more XR displays 812, one or more optional internal-facing and / or external-facing image sensors 814, memory 820, and one or more communication buses 804 for interconnecting these and various other components.

[0141] In some implementations, one or more communication buses 804 include circuits that interconnect and control communications between system components. In some implementations, one or more I / O devices and sensors 806 include an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, and / or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0142] In some implementations, one or more XR displays 812 are configured to provide an XR experience to the user. In some implementations, one or more XR displays 812 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field effect transistor (OLET), organic light-emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some implementations, one or more XR displays 812 correspond to diffraction, reflection, polarization, holographic and other waveguide displays. For example, the electronic device 120 includes a single XR display. As another example, the electronic device includes an XR display for each eye of the user. In some implementations, one or more XR displays 812 can present MR and VR content.

[0143] In some implementations, the one or more image sensors 814 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and thus may be referred to as an eye-tracking camera). In some implementations, the one or more image sensors 814 are configured to face forward in order to acquire image data corresponding to the physical environment that the user would see when the electronic device 120 is not present (and may be referred to as a scene camera). The one or more optional image sensors 814 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, and / or one or more event-based cameras, etc.

[0144] The memory 820 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some specific implementations, the memory 820 includes a non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 820 optionally includes one or more storage devices remotely located from the one or more processing units 802. The memory 820 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 820 or the non-transitory computer-readable storage medium of the memory 820 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 830 and an XR rendering module 840.

[0145] The operating system 830 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the XR rendering module 840 is configured to present XR content to a user via one or more XR displays 812. To this end, in various implementations, the XR rendering module 840 includes a data acquisition unit 842, an XR mixing unit 844, an XR rendering unit 846, and a data transmission unit 848.

[0146] In some specific implementations, the data acquisition unit 842 is configured to at least Figure 1 The controller 110 obtains data (eg, presentation data, interaction data, sensor data, location data, etc.) To this end, in various implementations, the data acquisition unit 842 includes instructions and / or logic components for instructions and heuristics and metadata for heuristics.

[0147] In some implementations, the XR blending unit 844 is configured to blend the pass-through image with the virtual content to generate an XR image. To this end, in various implementations, the XR blending unit 844 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0148] In some implementations, the XR rendering unit 846 is configured to display XR images via one or more XR displays 812. To this end, in various implementations, the XR rendering unit 846 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0149] In some implementations, the data transmission unit 848 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110. In some implementations, the data transmission unit 848 is configured to transmit authentication credentials to the electronic device. To this end, in various implementations, the data transmission unit 848 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0150] Although the data acquisition unit 842, the XR blending unit 844, the XR rendering unit 846, and the data transfer unit 848 are illustrated as residing on a single device (e.g., the electronic device 120), it should be understood that in other implementations, any combination of the data acquisition unit 842, the XR blending unit 844, the XR rendering unit 846, and the data transfer unit 848 may be located in separate computing devices.

[0151] also, Figure 8 It is more of a functional description of various features that may be present in a particular implementation, as opposed to a schematic diagram of the structure of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 8 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.

[0152] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.

[0153] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0154] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0155] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

Claims

1. A method, include: At a device including an image sensor, a display, one or more processors, and a non-transitory memory: capturing a first portion of an image of a physical environment using the image sensor; distorting the first portion of the image of the physical environment to generate a distorted first portion; blending the warped first portion with a first portion of virtual content to generate a blended first portion; displaying the mixed first portion on the display; capturing a second portion of the image of the physical environment using the image sensor; distorting the second portion of the image of the physical environment to generate a distorted second portion; blending the warped second portion with the second portion of the virtual content to generate a blended second portion; as well as displaying the mixed second portion on the display, Wherein displaying the first portion of the blend is initiated before completing blending the second portion of the warp. 2 . The method of claim 1 , wherein blending the first portion of the warp is initiated before completing warping the second portion of the image of the physical environment.

3. The method of claim 1 or 2, wherein distorting the first portion of the image of the physical environment is initiated before completing capturing the second portion of the image of the physical environment.

4. The method of any one of claims 1 to 3, wherein displaying the first part of the blend is initiated before completing distorting the second part of the image of the physical environment.

5. The method of any one of claims 1 to 4, wherein the time between starting to capture the first portion of the image of the physical environment and completing blending of the second warped portion is greater than a frame period. 6 . The method of claim 5 , wherein the frame period is the amount of time between starting to capture the first portion of the image and starting to capture the second portion of the image.

7. The method of any one of claims 1 to 6, wherein the time between starting to capture the first portion of the image of the physical environment and starting to display the second mixed portion is greater than a frame period.

8. The method of claim 7, wherein the frame period is an amount of time between starting to capture the first portion of the image and starting to capture the second portion of the image.

9. The method according to any one of claims 1 to 8, wherein distorting the first portion of the image of the physical environment comprises distorting the first portion of the image of the physical environment to correct for distortion caused by a lens of the image sensor and / or a lens of the display, wherein the mixed first portion includes a first color channel and a second color channel, The method also includes distorting the second color channel to correct for chromatic aberration such that the mixed first portion includes the first color channel and the distorted second color channel.

10. The method according to any one of claims 1 to 9, wherein the first portion of the distortion comprises a void, The method further includes obtaining a hole filling color, Wherein blending the warped first portion with the first portion of the virtual content comprises filling the hole by setting pixel values ​​of pixels of the hole to the hole fill color.

11. The method of any one of claims 1 to 10, wherein distorting the first portion comprises distorting the first portion to correct for differences in viewing angle between the image sensor and the display.

12. The method of any one of claims 1 to 11, wherein distorting the first portion comprises distorting the first portion to correct for distortion caused by a lens of the image sensor and / or a lens of the display.

13. The method of any one of claims 1 to 12, wherein distorting the first portion comprises distorting the first portion to correct for optical crosstalk.

14. The method of any one of claims 1 to 13, wherein distorting the first portion comprises distorting the first portion to achieve foveal vision.

15. The method of any one of claims 1 to 14, wherein distorting the first portion comprises distorting the first portion of virtual content.

16. The method of any one of claims 1 to 15, wherein distorting the first portion comprises distorting a first portion of an alpha mask. The method of claim 16 , wherein blending the first portion is based on a warped first portion of the alpha mask.

18. A device, include: Image sensor; monitor; non-transitory memory; and One or more processors configured to: capturing a first portion of an image of a physical environment using the image sensor; distorting the first portion of the image of the physical environment to generate a distorted first portion; blending the warped first portion with a first portion of virtual content to generate a blended first portion; displaying the mixed first portion on the display; capturing a second portion of the image of the physical environment using the image sensor; distorting the second portion of the image of the physical environment to generate a distorted second portion; blending the warped second portion with the second portion of the virtual content to generate a blended second portion; as well as displaying the mixed second portion on the display, Wherein displaying the first portion of the blend is initiated before completing blending the second portion of the warp.

19. The device of claim 18, wherein the one or more processors are configured to distort the first portion to correct for differences in viewing angle between the image sensor and the display.

20. A non-transitory memory storing one or more programs which, when executed by one or more processors of a device comprising an image sensor and a display, cause the device to perform any of the methods of claims 1 to 17.