Low latency frame delivery

Abstract

An example is described of marking a designated area of ​​stored image frame buffer data in an image frame buffer. The imaging system can read the designated area of ​​the image frame buffer to identify whether the marking has been overwritten. Thus, the imaging system can efficiently identify how much of the image frame buffer has been overwritten with data from a new image frame. Based on this, the imaging system can retrieve partial image frame data from the image frame buffer and process the partial image frame data, for example, to composite the partial image frame data with virtual content and / or to perform distortion compensation. The processed partial image frame data can be uploaded to a display buffer and displayed by a display, or the frame can be captured and processed again. The imaging system can also perform auto-exposure using the partial image frame data.

Description

[Technical field]

[0001]

[0001] This application relates to image processing. More particularly, aspects of the application relate to systems and methods for efficiently processing buffered image data for display. [Background technology]

[0002]

[0002] An extended reality (XR) device is a device that displays an environment to a user, for example through a head-mounted display (HMD) or other device. The environment is at least partially different from the real-world environment in which the user is located. The user can generally interactively change the view of the environment, for example by tilting or moving the HMD or other device. Virtual reality (VR) and augmented reality (AR) are examples of XR.

[0003]

[0003] An image sensor of an image capture device can capture an image of a scene. Traditionally, an entire image frame must be completed to be captured and processed before a software application running on a processor receives access to the image of the scene from the image sensor. Traditionally, negotiation processes between hardware drivers, an operating system, and a software application must also be completed before the software application receives access to the image frame. All of these processes result in a lag between the capture of an image frame and the display of the same image frame. Applications such as XR can introduce additional lag by processing the image frame (e.g., to insert virtual content) before displaying the image frame. The delay between real-world movement and the display of a representation of that movement on a display screen can be referred to as motion-to-photon latency. In applications where latency between the camera and the display is a priority, such as XR, such delays can cause the displayed environment to appear out of sync with the real world, can make movement in the displayed environment appear jittery or stuttery, and can make the user feel nauseous. Summary of the Invention

[0004]

[0004] Described herein are systems and techniques for image processing for rapid receipt of partial image frames in an application. The image processing system marks existing image frame buffer data in an image frame buffer with a watermark (and / or other metadata) in one or more predetermined regions. An image sensor captures image frame data corresponding to an image frame and gradually fills the image frame buffer with the image frame data as the image sensor captures the image frame and / or when some initial image processing tasks (e.g., demosaicing and / or color space conversion) are completed. The image processing system can read memory in one or more of the predetermined regions to identify which of the watermarks (and / or other metadata) still remain in the image frame buffer and which of the watermarks have been overwritten by new or replacement data, such as image frame data. The image processing system can efficiently identify that at least a predetermined amount (e.g., a percentage) of the image frame has been captured and stored in the image frame buffer based on which watermarks have been overwritten by image frame data. For example, if the image processing system determines that a watermark located at the midpoint of the image frame buffer has been overwritten by image frame data, the image processing system knows that at least 50% of the image frame has been captured and is currently stored in the image frame buffer. The image processing system can output a portion of the image frame corresponding to the predetermined percentage to a software application, a display buffer, a display controller, and / or a display.For example, if the image processing system knows that at least a first half of an image frame has been captured and is currently stored in the image frame buffer, the image processing system can output the first half of the image frame to a software application, which can process the first half of the image frame (e.g., to introduce virtual content) and send the processed half of the image frame to a display buffer. When the remainder of the image frame (e.g., the second half of the image frame) is received in the image frame buffer, the software application can process the remainder of the image frame and send the processed remainder of the image frame to the display buffer. The display buffer can display the image frame. In an illustrative example, the use of partial image frame access through the image frame buffer watermark can reduce the delay from capture to display of an image frame from 67 milliseconds (ms) to 14.2 ms.

[0005] In one example, an apparatus for image processing is provided. The apparatus includes a memory and one or more processors coupled to the memory (e.g., implemented in a circuit). The one or more processors are configured and capable of: storing data marked with metadata in a predetermined region of the image frame buffer in the image frame buffer; receiving image frame data corresponding to an image frame from an image sensor; overwriting at least a portion of the stored data in the image frame buffer with the image frame data; determining that the image frame data includes at least a predetermined amount of the image frame based on a read of a predetermined region of the image frame buffer indicating that the predetermined region stores additional data different from the metadata; and outputting at least a first portion of the image frame data corresponding to the predetermined amount of the image frame.

[0006]

[0006] In another example, a method of image processing is provided that includes storing data marked with metadata in a predetermined region of an image frame buffer in an image frame buffer, receiving image frame data from an image sensor corresponding to an image frame, overwriting at least a portion of the stored data in the image frame buffer with the image frame data, determining that the image frame data includes at least a predetermined amount of the image frame based on a reading of the predetermined region of the image frame buffer indicating that the predetermined region stores additional data different from the metadata, and outputting at least a first portion of the image frame data corresponding to the predetermined amount of the image frame.

[0007]

[0007] In another example, a non-transitory computer readable medium is provided that has instructions stored thereon that, when executed by one or more processors, cause the one or more processors to store data in an image frame buffer marked with metadata in a predetermined area of ​​the image frame buffer; receive image frame data from an image sensor corresponding to an image frame; overwrite at least a portion of the stored data in the image frame buffer with the image frame data; determine that the image frame data includes at least a predetermined amount of the image frame based on a reading of a predetermined area of ​​the image frame buffer indicating that the predetermined area stores additional data different from the metadata; and output at least a first portion of the image frame data corresponding to the predetermined amount of the image frame.

[0008] In another example, an apparatus for image processing is provided that includes means for storing in an image frame buffer data marked with metadata in a predetermined region of the image frame buffer, means for receiving image frame data from an image sensor corresponding to an image frame, means for overwriting at least a portion of the stored data in the image frame buffer with the image frame data, means for determining that the image frame data includes at least a predetermined amount of the image frame based on a reading of the predetermined region of the image frame buffer indicating that the predetermined region stores additional data distinct from the metadata, and means for outputting at least a first portion of the image frame data corresponding to the predetermined amount of the image frame.

[0009] In some aspects, the additional data is at least a portion of the image frame data.

[0010] In some aspects, to output the first portion of the image frame data, the one or more processors are configured to process the first portion of the image frame data. In some aspects, to process the first portion of the image frame data, the one or more processors are configured to modify at least a portion of the first portion of the image frame data using at least one of distortion, distortion compensation, and warping.

[0011] In some aspects, to output the first portion of the image frame data, the one or more processors are configured to warp at least a portion of the first portion of the image frame data. In some aspects, to output the first portion of the image frame data, the one or more processors are configured to distort at least a portion of the first portion of the image frame data. In some aspects, to output the first portion of the image frame data, the one or more processors are configured to perform distortion compensation on at least a portion of the first portion of the image frame data.

[0012] In some aspects, to output the first portion of the image frame data, the one or more processors are configured to process the first portion of the image frame data using an extended reality (XR) application. In some aspects, to output the first portion of the image frame data, the one or more processors are configured to composite the first portion of the image frame data with virtual content. In some aspects, the method, apparatus, and computer-readable medium described above further comprise identifying a pose of an image sensor and rendering the virtual content based on the pose of the image sensor.

[0013] In some aspects, to output the first portion of the image frame data, the one or more processors store the first portion of the image frame data in a display buffer. In some aspects, the methods, apparatus, and computer-readable media described above further comprise a display buffer.

[0014] In some aspects, to output the first portion of the image frame data, the one or more processors display the first portion of the image frame data using a display. In some aspects, the methods, apparatus, and computer-readable media described above further comprise a display.

[0015] In some aspects, to output the first portion of the image frame data, the one or more processors send the first portion of the image frame data to a receiving device using a communication transceiver. In some aspects, the methods, apparatus, and computer-readable media described above further comprise a communication transceiver.

[0016]

[0016] In some aspects, to output the first portion of the image frame data, the one or more processors are configured to determine one or more image statistics based on at least the first portion of the image frame data, and determine an image capture setting based on the one or more image statistics. In some aspects, the above-described method, apparatus, and computer-readable medium further comprise sending the image capture setting to image capture hardware, where the image capture hardware includes an image sensor, receiving secondary image frame data corresponding to a second image frame from the image sensor, where the image sensor captures the second image frame based on the image capture setting. In some aspects, the image capture setting is an exposure setting.

[0017]

[0017] In some aspects, the methods, apparatus, and computer-readable media described above further comprise marking the stored data with metadata in a predetermined area of ​​the image frame buffer.

[0018] In some aspects, the metadata includes a pattern of colors. In some aspects, the metadata includes a frame identifier associated with the image frame. In some aspects, the metadata includes an image frame buffer identifier associated with the image frame buffer.

[0019]

[0019] In some aspects, to determine that the image frame data includes at least a predetermined amount of the image frame, the one or more processors are configured to read a predetermined region of the image frame buffer and, based on the reading of the predetermined region of the image frame buffer, determine that the read indicates that the predetermined region stores additional data, different from the metadata.

[0020]

[0020] In some aspects, the stored data includes previous image frame data from a previous image frame, the previous image frame being captured prior to capture of the image frame.

[0021] In some aspects, to output at least a first portion of the image frame data, the one or more processors are configured to output the image frame.

[0022] In some aspects, the existing data is marked with second metadata in a second predetermined region of the image frame buffer. In some aspects, the methods, apparatus, and computer-readable media described above further comprise determining that the image frame data includes at least a second predetermined amount of the image frame based on a second reading of the second predetermined region of the image frame buffer indicating that the second predetermined region stores secondary additional data different from the second metadata, and outputting at least a second portion of the image frame data corresponding to the second predetermined amount of the image frame.

[0023]

[0023] In some aspects, the methods, apparatus, and computer-readable media described above further include receiving a request for a view of the scene and automatically exposing the image sensor to light from the scene in response to receiving the request, wherein the image frame data received from the image sensor is based on the exposure of the image sensor to the light from the scene.

[0024]

[0024] In some aspects, the apparatus includes a mobile device. In some aspects, the apparatus is a wireless communication device. In some aspects, the apparatus is a head mounted display. In some aspects, the apparatus is a camera. In some aspects, the methods, apparatus, and computer readable media described above further comprise an image sensor. In some aspects, the methods, apparatus, and computer readable media described above further comprise an image frame buffer.

[0025] In some aspects, the metadata includes a watermark.

[0026] In some aspects, the predetermined amount of image frames comprises a predetermined percentage of the image frames. In some aspects, the predetermined amount of new image frames comprises a predetermined proportion of the image frames. In some aspects, the predetermined amount of image frames comprises a predetermined number of rows of the image frames. In some aspects, the predetermined amount of image frames comprises a predetermined number of columns of the image frames. In some aspects, the predetermined amount of image frames comprises a predetermined number of pixels of the image frames.

[0027] In some aspects, the device comprises a camera, a mobile device, a mobile phone, a smartphone, a mobile handset, a portable gaming device, a wireless communication device, a smart watch, a wearable device, a head mounted display (HMD), an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other device. In some aspects, the one or more processors include an image signal processor (ISP). In some aspects, the device includes a camera or multiple cameras for capturing one or more images. In some aspects, the device includes an image sensor that captures the images. In some aspects, the device further includes a display for displaying the images, one or more notifications (e.g., related to the processing of the images), and / or other displayable data. In some aspects, the display displays the images after the one or more processors have processed the images.

[0028]

[0028] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter, which subject matter should be understood with reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0029]

[0029] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0030]

[0030] Exemplary embodiments of the present application are described in detail below with reference to the following figures. [Brief description of the drawings]

[0031] [Figure 1]

[0031] Block diagram showing the architecture of an image capture and processing system, according to some examples. [Diagram 2]

[0032] FIG. 1 is a block diagram illustrating an architecture of an extended reality (XR) system, according to some examples. [Diagram 3]

[0033] 1 is a conceptual diagram illustrating an imaging system having a graphics processing unit (GPU) that determines how full an image frame buffer is with image frame data based on a watermark in the image frame buffer, according to some examples. [Figure 4]

[0034] 1 is a conceptual diagram illustrating image frame data overwriting existing image frame buffer data with a watermark, according to some examples. [Diagram 5]

[0035] A conceptual diagram illustrating a timeline from the start of capture of an image frame to the display of the image frame, where the image frame is processed and sent to a display buffer in four separate parts, according to some examples. [Figure 6]

[0036] 1 is a conceptual diagram illustrating reducing latency between capture and display of image frames, according to some examples. [Figure 7]

[0037] 1 is a flow diagram illustrating image capture, processing, and display techniques, according to some examples. [Figure 8A]

[0038] 1 is a block diagram illustrating an imaging system for auto-exposure, in accordance with some examples. [Figure 8B]

[0039] 1 is a block diagram illustrating an imaging system for auto-exposure with partial frame delivery, in accordance with some examples. [Figure 9A]

[0040] 1 is a flow diagram illustrating an image processing technique, according to some examples. [Figure 9B]

[0041] 1 is a flow diagram illustrating an image processing technique, according to some examples. [Figure 10]

[0042] FIG. 1 illustrates an example of a system for implementing some aspects of the present technology, in accordance with some examples. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032]

[0043] Some aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments can be applied independently, and some of them can be applied in combination. In the following description, for the purpose of explanation, specific details are set forth to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments can be practiced without these specific details. The figures and descriptions are not limiting.

[0033]

[0044] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application, as set forth in the appended claims.

[0034]

[0045] An image capture device (e.g., a camera) is a device that receives light and captures an image frame, such as a still image or a video frame, using an image sensor. The terms "image," "image frame," and "frame" are used interchangeably herein. An image capture device typically includes at least one lens that receives light from a scene and bends the light toward an image sensor of the image capture device. The light received by the lens passes through an aperture controlled by one or more control mechanisms and is received by the image sensor. The one or more control mechanisms can control exposure, focus, and / or zoom based on information from the image sensor and / or based on information from an image processor (e.g., a host or application process and / or an image signal processor). In some examples, the one or more control mechanisms include a motor or other control mechanism that moves a lens of the image capture device to a target lens position.

[0035]

[0046] An extended reality (XR) device is a device that displays an environment to a user, for example, through a head mounted display (HMD), a mobile handset, a wearable device, or another device. XR can include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), and combinations thereof. The environment displayed by the XR device can be at least partially different from the real-world environment in which the user is. For example, in VR, the environment displayed by the XR device can be fully virtual or almost fully virtual. In some cases, VR can still be based on the real-world environment in which the user is, since the XR device can, for example, make the boundaries of its virtual environment similar to the boundaries of the real-world environment in which the user is, or incorporate virtual character positions that match the positions of other users in the real-world environment in which the user is. In AR or MR, the environment displayed by the XR device can be a mix of the real-world environment in which the user is and virtual content that augments the user's experience. The user can generally interactively change the view of the environment, for example, by tilting and / or rotating and / or moving the XR device. If the XR device is an HMD or a mobile handset, the user can tilt and / or rotate the XR device and / or move through the real-world environment in which the user is located while the user is wearing or holding the XR device.

[0036]

[0047] An image sensor of an image capture device can capture an image of a scene. In some devices, an entire image frame must be completed before a software application running on a processor receives access to the image of the scene from the image sensor. Even an image capture device capable of a high-speed image frame capture rate, such as 90 frames per second (FPS), takes approximately 11 milliseconds (ms) to capture an image frame in its entirety. In some devices, a negotiation process between the hardware driver's application programming interface (API), the operating system (OS), and the software application must also be completed before the software application receives access to the image frame. This negotiation process may include multiple API layers. For example, the negotiation process may include a camera hardware event being sent to the OS kernel, the OS kernel communicating with the OS kernel's camera driver, the OS kernel's camera driver interacting with a user space camera driver residing in the user space, the user space camera driver communicating with a high level OS (HLOS) camera service, and the HLOS camera service finally communicating with the software application. Each layer in this negotiation process may involve API calls to these different elements. Waiting for the capture of an entire image frame combined with this negotiation process can result in a significant delay between the capture of an image frame and the display of that same image frame. Each of these API layers can require scheduling tasks that compete for common, limited CPU resources.

[0037]

[0048] Applications such as XR can introduce additional delays by processing image frames before displaying them (e.g., inserting virtual content, performing distortion compensation, processing related to camera internal calibration, processing related to camera external calibration, processing related to consistency between left and right frames in a stereoscopic system, performing color correction, etc.). Fully processing the complexities of each image and associated API calls can introduce delays in providing camera images to a GPU warp compositor and contribute significantly to photon to motion latency in see-through XR applications. In applications where latency between the camera and the display is a priority, such delays can cause the displayed environment to appear out of sync with the real world, can make movement in the displayed environment appear jittery or stuttering, can cause scheduling delays across the application and / or OS, and can cause the user to feel nauseous or otherwise unwell.

[0038]

[0049] Setting image capture settings, such as exposure, focus, or zoom, may also result in delays or latency. Automatic image capture setting adjustments, such as auto-exposure, auto-focus, and auto-zoom, may set values ​​of image capture settings based on a previous image captured by the image sensor. Such a process may require capturing a previous image, passing the previous image through a system buffer or queue, and analyzing the previous image to obtain image statistics before new image capture settings can be determined based on image statistics. Once new image capture settings are determined, the image capture settings may need to pass through various buffers, queues, kernels, and / or hardware actuators before the image capture settings are actually applied to the image. All of these steps can take additional time and potentially add additional latency.

[0039]

[0050] Described herein are systems and techniques for image processing for rapid receipt of partial image frames from an image frame buffer in a software application. The image processing system watermarks image frame buffer data stored in the image frame buffer at one or more watermark locations or regions in the image frame buffer. An image sensor captures image frame data corresponding to image frames and gradually fills the image frame buffer with image frame data as the image sensor captures image frames. The image processing system can read memory at one or more of the watermark locations or regions in the image frame buffer to identify which of the watermarks still remain in the image frame buffer and which of the watermarks have been overwritten by other data (image frame data). The image processing system can efficiently identify that at least a predetermined amount (e.g., a percentage, a proportion, a ratio) of the image frame has been captured and stored in the image frame buffer based on which watermarks have been overwritten by the image frame data. For example, if the image processing system determines that a watermark located at the midpoint of the image frame buffer has been overwritten with image frame data, the image processing system knows that at least 50% of the image frame has been captured and is currently stored in the image frame buffer. The image processing system can output a portion of the image frame corresponding to a predetermined amount to a software application, a display buffer, a display controller, and / or a display. For example, if the image processing system knows that at least a first half of the image frame has been captured and is currently stored in the image frame buffer, the image processing system can output the first half of the image frame to a software application, and the software application can process the first half of the image frame.In some examples, processing the first half of the image frame can introduce virtual content into the partial image frame by compositing the first half of the image frame with the virtual content. The image processing system can send the processed half of the image frame to a display buffer. In some cases, the image processing system can output a portion of the image frame corresponding to a predetermined amount of the image frame to an image capture setting determination controller that can use that portion of the image frame for auto-exposure, auto-focus, auto-zoom, or a combination thereof.

[0040]

[0051] By reading directly from the image buffer, much of the negotiation process between the hardware driver, the operating system, and the software application can be bypassed. Because checking how much of the image frame is currently stored in the image buffer can be done very quickly (e.g., through a quick memory read at the watermark location for one or more watermarks), a partial image frame can be sent reliably with certainty that it contains data from a newly captured image frame and does not contain any existing data in the image frame buffer. The processor receiving the partial image frame can process the partial image frame while the remainder of the image frame is still being captured, reducing wasted time. In examples where image frame data is composited with virtual content, the camera frame rate can be synchronized in rate and / or phase with the virtual content.

[0041]

[0052] In some examples, the display can immediately display the processed half of the image frame from the display buffer. For example, an organic light emitting diode (OLED) display can be used in this manner. In some examples, once the remainder of the image frame (e.g., the second half of the image frame) is received in the image frame buffer, a software application can process the remainder of the image frame and send the processed remainder of the image frame to the display buffer. The display buffer can display the image frame. For example, a liquid crystal display (LCD) display can be used in this manner. In an illustrative example, the use of partial image frame access through the image frame buffer watermark can reduce the delay from capture to display of the image frame from 67 ms to 14.2 ms.

[0042]

[0053] 1 is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components used to capture and process an image of a scene (e.g., an image of a scene 110). The image capture and processing system 100 can capture a standalone image (or photograph) and / or a video that includes multiple images (or video frames) in a particular sequence. A lens 115 of the system 100 faces the scene 110 and receives light from the scene 110. The lens 115 bends the light toward an image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130.

[0043]

[0054] The one or more controls 120 may control exposure, focus, and / or zoom based on information from image sensor 130 and / or based on information from image processor 150. The one or more controls 120 may include multiple mechanisms and components, for example, the control 120 may include one or more exposure controls 125A, one or more focus controls 125B, and / or one or more zoom controls 125C. The one or more controls 120 may also include additional controls beyond those shown, such as controls to control analog gain, flash, HDR, depth of field, and / or other image capture properties.

[0044]

[0055] The focus control mechanism 125B of the control mechanism 120 can obtain the focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B can adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B can move the lens 115 closer to or farther from the image sensor 130 by actuating a motor or servo (or other lens mechanism), thereby adjusting the focus. In some cases, additional lenses can be included in the system 100, such as one or more microlenses over each photodiode of the image sensor 130, each of which bends light received from the lens 115 toward a corresponding photodiode before the light reaches the photodiode. The focus setting can be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The focus settings may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus settings may be referred to as image capture settings and / or image processing settings.

[0045]

[0056] The exposure control mechanism 125A of the control mechanism 120 can obtain an exposure setting. In some cases, the exposure control mechanism 125A stores the exposure setting in a memory register. Based on this exposure setting, the exposure control mechanism 125A can control the size of the aperture (e.g., aperture size or f / stop), the duration the aperture is open (e.g., exposure time or shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.

[0046]

[0057] The zoom control 125C of the control mechanism 120 can obtain a zoom setting value. In some examples, the zoom control 125C stores the zoom setting value in a memory register. Based on the zoom setting value, the zoom control 125C can control the focal length of an assembly of lens elements (lens assembly) including the lens 115 and one or more additional lenses. For example, the zoom control 125C can control the focal length of the lens assembly by actuating one or more motors or servos (or other lens mechanisms) to move one or more of the lenses relative to one another. The zoom setting value can be referred to as an image capture setting value and / or an image processing setting value. In some examples, the lens assembly can include a parfocal zoom lens or a variable focus zoom lens. In some examples, the lens assembly can include a focusing lens (which may be the lens 115 in some cases) that first receives light from the scene 110, and then the light passes through an afocal zoom system between the focusing lens (e.g., the lens 115) and the image sensor 130 before the light reaches the image sensor 130. An afocal zoom system may include two positive (e.g., convergent, convex) lenses of equal or similar focal lengths (e.g., within a threshold difference of each other), possibly with a negative (e.g., diverging, concave) lens between them. In some cases, the zoom control 125C moves one or more of the lenses in the afocal zoom system, such as one or both of the negative and positive lenses.

[0047]

[0058] The image sensor 130 includes one or more arrays of photodiodes or other light-sensitive elements. Each photodiode measures an amount of light that ultimately corresponds to a particular pixel in the image generated by the image sensor 130. In some cases, different photodiodes may be covered by different color filters and may therefore measure light that matches the color of the filter that covers the photodiode. For example, a Bayer color filter includes a red filter, a blue filter, and a green filter, and each pixel of the image is generated based on red light data from at least one photodiode covered with a red filter, blue light data from at least one photodiode covered with a blue filter, and green light data from at least one photodiode covered with a green filter. Other types of color filters may use yellow, magenta, and / or cyan (also called "emerald") color filters instead of or in addition to red, blue, and / or green color filters. Some image sensors (e.g., the image sensor 130) may not have color filters at all and may instead use different photodiodes across the (in some cases vertically stacked) pixel array. Different photodiodes across the pixel array have different spectral sensitivity curves and can therefore respond to different wavelengths of light. Monochrome image sensors also lack color filters and therefore may lack color depth.

[0048]

[0059] In some cases, the image sensor 130 may alternatively or additionally include an opaque mask and / or a reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles, which may be used for phase detection autofocus (PDAF). The image sensor 130 may also include an analog gain amplifier for amplifying an analog signal output by the photodiode, and / or an analog-to-digital converter (ADC) for converting the analog signal output of the photodiode (and / or amplified by the analog gain amplifier) ​​to a digital signal. In some cases, some components or functions discussed with respect to one or more of the control mechanisms 120 may instead or additionally be included in the image sensor 130. The image sensor 130 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal-oxide semiconductor (CMOS), an N-type metal-oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0049]

[0060] Image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISPs 154), one or more host processors (including host processor 152), and / or one or more of any other types of processors 1010 discussed with respect to computing system 1000. Host processor 152 may be a digital signal processor (DSP) and / or other types of processors. In some implementations, image processor 150 is a single integrated circuit or chip (e.g., referred to as a system-on-chip or SoC) that includes host processor 152 and ISPs 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) ports 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G, or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth, Global Positioning System (GPS), etc.), any combination thereof, and / or other components. The I / O ports 156 may include any suitable input / output ports or interfaces according to one or more protocols or specifications, such as an Inter Integrated Circuit 2 (I2C) interface, an Inter Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial general purpose input / output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface), an Advanced High Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In one illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port and the ISP 154 may communicate with the image sensor 130 using a MIPI port.

[0050]

[0061] Image processor 150 may perform several tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, auto white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. Image processor 150 may store image frames and / or processed images in random access memory (RAM) 140 and / or 1020, read only memory (ROM) 145 and / or 1025, a cache, a memory unit, another storage device, or some combination thereof.

[0051]

[0062] Various input / output (I / O) devices 160 may be connected to image processor 150. I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a track pad, a touch-sensitive surface, a printer, any other output device 1035, any other input device 1045, or some combination thereof. In some cases, captions may be entered into image processing device 105B through a physical keyboard or keypad of I / O device 160 or through a virtual keyboard or keypad of a touch screen of I / O device 160. I / O 160 may include one or more ports, jacks, or other connectors that enable a wired connection between system 100 and one or more peripheral devices, through which system 100 may receive data from and / or transmit data to one or more peripheral devices. I / O 160 may include one or more wireless transceivers that enable a wireless connection between system 100 and one or more peripheral devices, through which system 100 may receive data from and / or transmit data to one or more peripheral devices. The peripheral devices may include any of the types of I / O devices 160 previously discussed, and may themselves be considered I / O devices 160 when coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.

[0052]

[0063] In some cases, the image capture and processing system 100 may be a single device. In some cases, the image capture and processing system 100 may be two or more separate devices including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing system coupled to a camera). In some implementations, the image capture device 105A and the image processing device 105B may be coupled to each other, for example, via one or more wires, cables, or other electrical connectors and / or wirelessly via one or more wireless transceivers. In some implementations, the image capture device 105A and the image processing device 105B may be separate from each other.

[0053]

[0064] 1, a vertical dashed line divides the image capture and processing system 100 of FIG. 1 into two portions, which respectively represent an image capture device 105A and an image processing device 105B. Image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. Image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), a RAM 140, a ROM 145, and an I / O 160. In some cases, some components shown in image capture device 105A, such as ISP 154 and / or host processor 152, may be included in image capture device 105A.

[0054]

[0065] The image capture and processing system 100 of FIG. 1 includes one or more image buffers 170. The one or more image buffers 170 may each include one or more image frame buffers. Each image frame buffer may temporarily store a single image frame captured by the image sensor 130 and / or processed by the image processor 150. In some examples, the image frame buffer may be or may include a circular buffer, a circular queue, a circular buffer, a ring buffer, or a combination thereof. In some examples, the one or more image buffers 170 may receive and store partial image frame data from an image frame before the entirety of the image has completed capture. In some examples, image frame data from a new image frame overwrites older existing image buffer data in the image frame buffer. The one or more image buffers 170 are shown as being coupled to both the image sensor 130 and the image processor 150. In some examples, the image processor 150 receives image frames from the image sensor 130 through the image buffer 170. In some examples, the image buffer may be coupled only to the image sensor 130 and not to the image processor 150. In some examples, the image buffer may be coupled only to the image processor 150 and not to the image sensor 130.

[0055]

[0066] The image capture and processing system 100 may include an electronic device such as a mobile or landline telephone handset (e.g., a smartphone, a cellular telephone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 Wi-Fi® communication, wireless local area network (WLAN) communication, or some combination thereof. In some implementations, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing system, such as a mobile handset, a desktop computer, or another computing system.

[0056]

[0067] Although image capture and processing system 100 is shown to include several components, one skilled in the art will appreciate that image capture and processing system 100 may include many more components than those shown in FIG. 1. The components of image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of image capture and processing system 100 may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits) and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein. The software and / or firmware may include one or more instructions stored in a computer-readable storage medium and executable by one or more processors of an electronic device implementing image capture and processing system 100.

[0057]

[0068] FIG. 2 is a block diagram illustrating an architecture of an extended reality (XR) system 200. The XR system 200 of FIG. 2 includes one or more cameras 205. Each camera of the cameras 205 may be an example of an image capture device 105A, an image processing device 105B, an image capture and processing device 100, or a combination thereof. For example, four cameras 205 are shown in FIG. 2, each having an image sensor. Two of the cameras 205 shown in FIG. 2 are visible light cameras with image sensors that are sensitive to light from the visible light spectrum and each output a color image. These two cameras are labeled in FIG. 2 as red-green-blue (RGB) cameras. Two of the cameras 205 shown in FIG. 2 each output a monochrome image. These two cameras are labeled in FIG. 2 as monochrome (mono) cameras. The cameras 205 each capture image frame data corresponding to at least a portion of one or more image frames. Image frame data from each of the cameras 205 is passed along one or more interfaces, such as a camera serial interface (CSI), a mobile industry processor interface (MIPI), or a combination thereof, to one or more image front ends (IFEs). The IFEs may be examples of at least a portion of the image processing device 105B. The IFEs may perform some of the image processing tasks described as being performed by the ISP 154, the DSP 152, or the image processor 150. For example, the IFEs may perform demosaicing, color space conversion, color format conversion, bad pixel correction, or a combination thereof. For example, the IFEs may convert image frame data from an RGB camera to a YUV color space. The IFEs may convert image frame data from a monochrome camera to a raw16 color space. The IFEs may send the image frame data, possibly with these color space conversions and / or other image processing operations, to one or more image buffers 210.In some examples, the IFE may send image frame data to an image processing engine (IPE). In some examples, the IPE may perform heavier processing, such as skin tone correction, sky color correction, face detection, red-eye removal, etc. In some examples, the IPE may introduce more scheduling and processing latency (and thus jitter and / or stuttering). The IPE may be bypassed in some examples, as indicated by the "X" mark in FIG. 2. In some examples, at least some of the operations described herein as being performed by the IPE may be performed by the IFE. In some examples, at least some of the operations described herein as being performed by the IFE may be performed by the IPE.

[0058]

[0069] The one or more image buffers 210 may be examples of the one or more image buffers 170 of the image capture and processing device 100 of FIG. 1. For example, each image buffer of the one or more image buffers 210 may include one or more image frame buffers that may store image frame data corresponding to particular image frames from one or more of the cameras 205. In some examples, the one or more image buffers 210 are stored as graphics processor unit (GPU) texture buffers. In some examples, the one or more image buffers 210 are stored as XR camera buffers.

[0059]

[0070] Image frame data corresponding to partial or full image frames may be transferred from the image buffer 210 to the XR application 215. The XR application 215 may generate virtual content to be overlaid on top of, merged with, or otherwise composited with the image frame data from the image buffer 210 using a compositor 225 of the XR application 215. The virtual content may be generated, rendered, and stored in one or more virtual content rendering buffers 220 before being composited with the image frame data from the image buffer 210 using the compositor 225. In some examples, the virtual content rendering buffers 220 may be referred to as a virtual rendering buffer, a virtual content buffer, a game rendering buffer, a video game rendering buffer, an in-game content rendering buffer, or combinations thereof. Video games are one exemplary use case of an XR device in which virtual content is generated and / or rendered, representing, for example, in-game characters, in-game items, in-game costumes, in-game areas, other in-game objects, or combinations thereof. The virtual content rendering buffer 220 may also be referred to as a virtual content buffer. The XR application 215 may include an XR software development kit (SDK), which may include a compositor 225 and / or other image processing functions. A developer of an XR application 215, such as a game, may call the compositor 225 and / or other image processing functions of the XR SDK. In some examples, the XR SDK may be an XR API. The compositing performed by the compositor 225 may be referred to as warping.

[0060]

[0071] The compositor 225 generates composited image frame data by compositing image frame data from the image frame buffer of the image buffer 210 with virtual content from the virtual content rendering buffer 220. If the image frame data is only a partial image frame (e.g., includes only a portion of an image frame), the composited image frame data is composited based on the partial image frame. The composited image frame data may be referred to as composited image data, composited frame data, composited image, composited image frame, composited partial image frame data, composited partial image data, composited partial frame data, composited partial image, composited partial image frame, display image frame data, display image data, display frame data, display image, display image frame, display partial image frame data, display partial image data, display partial frame data, display partial image, display partial image frame, or combinations thereof. The XR application 215 sends the composited image frame data to the display buffer 230. The display buffer 230 may be referred to as a screen buffer. In some examples, when display buffer 230 receives the composited image frame data, display buffer 230 sends the composited image frame data to display 235 to be displayed on display 235. In some examples, when display buffer 230 receives composited image frame data corresponding to an entire image frame, display buffer 230 sends the composited image frame data corresponding to the entire image frame to display 235 to be displayed on display 235. For example, compositor 225 may receive multiple portions of the same image frame from image buffer 210 at different times, may generate multiple sets of composited image frame data based on these portions, and may send the set of composited image frame data to display buffer 230 upon generating each of the sets of composited image frame data.Once the display buffer 230 has received all of the set of composited image frame data corresponding to an image frame from the compositor 225, the display 235 can display the entire image frame stored in the display buffer 230 (along with the virtual content composited by the compositor 225).

[0061]

[0072] In some examples, compositor 225 may perform other image processing tasks besides compositing image frame data from image buffer 210 with virtual content from virtual content rendering buffer 220. Compositor 225 may perform any image processing task described as being performed by ISP 154, DSP 152, and / or image processor 150. Compositor 225 may perform distortion compensation, processing related to camera in-house calibration, processing related to camera ex-house calibration, processing related to consistency between left and right frames in a stereoscopic system, color correction, brightness correction, contrast correction, white balance correction, black balance correction, saturation correction, or a combination thereof. Some of these processing tasks may utilize additional data received by the XR application 215, such as sensor data from the camera 205, an inertial measurement unit (IMU), calibration data from an internal calibration of the camera 205, calibration data from an external calibration of the cameras 205 (e.g., relative to each other), characteristics of the display 235, characteristics of one or more lenses of the camera 205, characteristics of one or more lenses located between the display 235 and the viewer, or a combination thereof. The composited image frame data from the display buffer 230 may be passed to the display 235 along one or more interfaces, such as a Display Serial Interface (DSI), a Mobile Industry Processor Interface (MIPI), or a combination thereof.

[0062]

[0073] Distortion compensation may compensate or correct for distortions caused by the camera 205, for example, by reducing radial distortions caused by a wide-angle lens. Distortion compensation may compensate or correct for distortions caused by the display 235 to make proportions in the environment appear accurately to a viewer of the display 235 based on characteristics of the display 235 and the viewing of the display 235, such as, for example, the curvature of the display 235, the flatness of the display 235, the distance between the display 235 and the viewer's eye or eyes, the viewing angle between the display 235 and the viewer's eye or eyes, the optical characteristics of one or more lenses between the display 235 and the viewer's eye or eyes, the effects of glare and / or shadows and / or transparency and / or other lighting effects on the display 235, or a combination thereof. Distortion compensation may compensate for perspective distortion.

[0063]

[0074] In some examples, image buffer 210 may be shared between the IFE and the GPU and tightly synchronized without API calls throughout all layers. Thus, GPU warping (e.g., using compositor engine 225) can access pixels of an image frame and output a processed (warped) frame based on metadata faster than receiving a fully processed frame from an image processing engine.

[0064]

[0075] In some examples, the camera 205 may include a pair of stereoscopic cameras for generating a stereoscopic view in the display 235. For example, the exemplary image 240 shown in FIG. 2 displayed on the display 235 includes a left image and a right image. The processing may include processing related to camera internal calibration, where each of the cameras may undergo internal calibration so that the processing can correct for problems due to the internal characteristics of the camera 205, such as slight manufacturing defects or differences, or slight differences in the coupling between the camera 205 and the housing of the XR device. The processing may include processing related to external calibration of the cameras, as the two cameras may undergo external calibration to account for slight differences between the cameras and slight changes in the placement between the cameras due to the internal characteristics of the cameras and / or the XR device. The processing may relate to consistency between the left and right frames in the stereoscopic system, for example, to ensure that brightness levels, contrast levels, color levels, white balance, black balance, saturation, and other image characteristics are similar in the left image compared to the right image.

[0065]

[0076] In some examples, the camera 205 may be synchronized with data from the XR application 215 via an XR camera synchronizer. The XR camera synchronizer may, for example, synchronize virtual data from the virtual content render buffer 220 with the camera data so that the image buffer 210 and the virtual content render buffer 220 contain data that corresponds to the same or similar times.

[0066]

[0077] In some examples, the camera frame rate of the camera 205 can be controlled by the XR camera synchronizer in small or large increments. By controlling the camera frame rate, the XR camera synchronizer can synchronize the camera 205 with the virtual content in rate and / or phase. For example, the XR camera synchronizer can synchronize the camera 205 with the virtual content such that a certain fill level is completed when the image processing needs that data, thus avoiding latency on the image processing side. Instead of polling for a fill level to be achieved, the image processor of the XR system 200 simply needs to prove that the fill level has been established and therefore that the camera is still synchronized. Using strict synchronization can eliminate the latency between when data is received in a buffer and when that data can be consumed.

[0067]

[0078] FIG. 3 is a conceptual diagram illustrating an imaging system 300 having a graphics processing unit (GPU) 325 that determines how full the image frame buffer 315 is with image frame data based on a watermark 320 in the image frame buffer 315. One or more cameras 305 are shown in FIG. 3. Each camera of the cameras 305 may be an example of one of the image capture device 105A, the image processing device 105B, the image capture and processing device 100, the one or more cameras 205, or a combination thereof. Image frame data is captured by the one or more cameras 305 and stored in an image buffer 310. The image buffer 310 of FIG. 3 may be an example of the image buffer 210 of FIG. 2. The image buffer 310 of FIG. 3 is a ring buffer, also known as a circular buffer. The image buffer 310 of FIG. 3 includes 16 image frame buffers, each image frame buffer shown as a circle with a black outline and a numeric label within the circle. The 15 image frame buffers of image buffer 310 of Figure 3 are labeled "0" through "15." The numeric label of each image frame buffer ("0" through "15") is shown within a circle representing the image frame buffer. Each image frame buffer of image buffer 310 of Figure 3 is capable of storing an entire image frame.In the example of FIG. 3, image frame buffer 15 stores frame 511, image frame buffer 0 stores frame 512, image frame buffer 1 stores frame 513, image frame buffer 2 stores frame 514, image frame buffer 3 stores frame 515, image frame buffer 4 stores frame 516, image frame buffer 5 stores frame 517, image frame buffer 6 stores frame 518, image frame buffer 7 stores frame 519, image frame buffer 8 stores frame 520, image frame buffer 9 partially stores frame 521, image frame buffer 10 is scheduled to store frame 522, image frame buffer 11 is scheduled to store frame 523, image frame buffer 12 is scheduled to store frame 524, image frame buffer 13 is scheduled to store frame 523, and image frame buffer 14 is unused. Image frames 511-520 are stored in image frame buffers 15 and 0-8 and are available for use by an application (e.g., XR application 215 in FIG. 2). Image frame buffers 15 and 0-8 are shown as open circles to indicate their availability to an application. Image frames 522-525 have not yet been captured by camera 305 and are scheduled to be stored in image frame buffers 10-13. Thus, image frames 522-525 and image frame buffers 10-13 are not available for use by an application. Image frame buffers 10-13 are shown with a cross-hatch pattern to indicate that they are not available to an application. Image frame buffer 14 is not used to prevent race conditions and is shown as a solid circle to indicate this unused nature.

[0068]

[0079] Image frame 521 is in the process of being captured by camera 305. Image frame data corresponding to a portion of image frame 521 is stored in image frame buffer 9. Image frame buffer 9 is shown in a dithered pattern to indicate that image frame buffer 9 already stores a portion of image frame 521, but not the entire image frame 521. Using the systems and techniques described herein, at least a portion of image frame buffer 9 may be accessed by an application.

[0069]

[0080] Image frame buffer 315 is a graphical representation of image frame buffer 9 according to some examples. Image frame buffer 315 is shown as a rectangle. The rectangle representing image frame buffer 315 includes a dithered shaded pattern that extends from the top of image frame buffer 315 to approximately two-thirds of the way into image frame buffer 315, leaving the bottom third of image frame buffer 315 white and unshaded. The dithered shaded pattern in image frame buffer 315 represents image frame data that corresponds to a portion of image frame 521. According to the shaded pattern in image frame buffer 315, then, the image sensor of camera 305 has already captured (and stored in image frame buffer 315) approximately two-thirds of image frame 521.

[0070]

[0081] The image frame buffer includes watermarks 320 in several locations. The watermarks 320 are shown in FIG. 3 as rectangles with diagonal lines throughout. The image processing system adds the watermarks 320 to the existing data in the image frame buffer 315 before image frame data from image frames 521 begin to be captured by the camera 305 and stored in the image frame buffer 315. The watermarks 320 may be inserted periodically, for example, once every 5% of the size of the image frame, once every 10% of the size of the image frame, once every 15% of the size of the image frame, once every 20% of the size of the image frame, once every 25% of the size of the image frame, once every N rows of pixels of the image frame, once every M columns of pixels of the image frame, or some combination thereof. Although the watermarks 320 are all shown to the right of the image frame buffer 315 in FIG. 3, the watermarks 320 may be placed anywhere in the image frame buffer 315. As image frame data from image frame 521 begins to be captured by camera 305 and stored in image frame buffer 315, the image frame data from image frame 521 overwrites watermark 320. Thus, although all of watermark 320 is still shown in FIG. 3 to show how watermark 320 may be laid out, the new image frame data from image frame 521 (represented by the dithered shaded area) overwrites watermark 320 as the new image frame data overwrites the existing image buffer data stored in image frame buffer 315 before the new image frame data is stored in image frame buffer 315.

[0071]

[0082] The GPU 325 performs a memory read in one or more regions of the image frame buffer 315 that correspond to one or more of the watermarks 320. For example, based on the dashed arrow, the GPU 325 performs a memory read of a region of the image frame buffer 315 that corresponds to one of the watermarks 320 in approximately 66% of the image frame buffer 315. The watermark 320 that was previously in this region has just been overwritten by new image frame data (represented by the dithered shaded region) from the image frame 521. The GPU 315 identifies, based on the memory read, that the region will no longer store the watermark 320 and therefore must be (and currently does) overwritten with new image frame data from the image frame 521. Thus, the GPU 315 knows that at least 66% of the image frame 521 has been captured and stored in the image frame buffer 315. In some cases, the GPU 315 may perform memory reads to other regions of the image frame buffer 315 corresponding to other watermarks 320 (e.g., after the watermark at 66%) to more accurately determine how much of the image frame 521 has been captured and stored in the image frame buffer 315. However, in the example shown in FIG. 3, 66% is accurate, which is indicated to the GPU 315 based on memory reads of further regions of the image frame buffer 315 corresponding to further watermarks 320 that still store their respective watermarks 320. The GPU 315 may retrieve the image frame data corresponding to the first 66% of the image frame 521. The GPU 315 may process the image frame data corresponding to the first 66% of the image frame 521 on behalf of the application (e.g., the XR application 215). The GPU may send a processed version of the image frame data corresponding to the first 66% of the image frame 521 to the display buffer 330.The rectangle representing display buffer 330 includes a dithered shaded pattern that extends from the top of display buffer 330 to approximately two-thirds of the way into display buffer 330, leaving the bottom third of display buffer 330 white and unshaded. The dithered shaded pattern represents the processed partial image frame data (e.g., the first 66% of image frame 521) from GPU 325. In some examples, display 335 may display the processed partial image frame data (e.g., the first 66% of image frame 521). In some examples, display 335 may wait until display buffer 330 includes the entirety of the processed version of image frame 521 before displaying the processed version of image frame 521.

[0072]

[0083] In some examples, the watermark 320 includes a visual pattern that is unlikely to be found in an image. In some examples, the watermark 320 includes a pattern of stripes of two alternating colors. For example, the two alternating colors may include black, white, dark gray that is nearly black, light gray that is nearly white, blue, red, green, cyan, magenta, yellow, any combination of primary colors, secondary colors, tertiary colors, or other colors. In some examples, the watermark 320 includes any combination of visual patterns, such as a checkerboard pattern, a dithering pattern, horizontal stripes, diagonal stripes, vertical stripes, a crosshatch pattern, a rainbow pattern, a gradient pattern, or combinations thereof. In some examples, the pattern of the watermark 320 may be selected from one of a set of predefined patterns. In some examples, the pattern of the watermark 320 may be selected by looping through a set of predefined patterns. In some examples, the pattern of the watermark 320 may be selected by selecting a pattern from a set of predefined patterns that is most likely to be different from the new image frame data from the new image frame. For example, if the camera 305 on which the image data is being captured is a monochrome camera that captures only monochrome (grayscale) image data, colorful patterns are unlikely to appear in the image data captured by the camera 305 and therefore may be optimal for use in the watermark 320. If the camera 305 on which the image data is being captured is a full-color camera, true grayscale or black / white patterns (e.g., with zero saturation) are unlikely to appear in the image data captured by the camera 305 and therefore may be optimal for use in the watermark 320. In some examples, the watermark 320 may include data such as a frame identifier (e.g., frame number 521 in FIG. 3), an image frame buffer identifier (e.g., image frame buffer number 9 in FIG. 3), or a combination thereof.In some examples, each watermark includes the lead-in pattern, frame identifier, image frame buffer identifier, and lead-out pattern described above. In one illustrative example, the lead-in pattern may be 0x1FE01FE01FE01FEull. In one illustrative example, the lead-out pattern may be 0xFE01FE01FE01FE01ull. In some examples, each watermark includes an alignment mask to align the watermark to an area of ​​the image frame buffer that can be easily read by GPU 325.

[0073]

[0084] FIG. 4 is a conceptual diagram 400 illustrating image frame buffer data 420-425 overwriting existing image frame buffer data 450-455 with watermarks 430-435. Shown are a left image frame buffer 410 and a right image frame buffer 415. The left image frame buffer 410 contains, towards its bottom, existing image frame buffer data 450 including a watermark 430. The watermark 430 is an example of a watermark 320 and is shown as a white horizontal line with a black stripe in the middle. The watermark 430 is surrounded by a dashed white circular outline in FIG. 4 so that it may be more easily identified. The right image frame buffer 415 contains, towards its bottom, existing image frame buffer data 455 including a watermark 435. The watermark 435 is an example of a watermark 320 and is shown as a white horizontal line with a black stripe in the middle. The watermark 435 is surrounded by a dashed white circular outline in FIG. 4 so that it may be more easily identified.

[0074]

[0085] The left image frame buffer 410 includes new image frame data 420 coming in from the top of the left image frame buffer 410 and overwriting the existing image frame buffer data 450 as the new image frame data 420 fills further down. The new image frame data 420 does not include a watermark. The seam 440 between the new image frame data 420 and the existing image frame buffer data 450 is highlighted in FIG. 4 with a dashed white line. The right image frame buffer 415 includes new image frame data 425 coming in from the top of the right image frame buffer 415 and overwriting the existing image frame buffer data 455 as the new image frame data 425 fills further down. The new image frame data 425 does not include a watermark. The seam 445 between the new image frame data 425 and the existing image frame buffer data 455 is highlighted in FIG. 4 with a dashed white line.

[0075]

[0086] 5 is a conceptual diagram 500 illustrating a timeline from the start of capture of an image frame to the display of the image frame, which is processed in four separate portions and sent to a display buffer. In conceptual diagram 500, the horizontal axis is a time axis 505, with time progressing from left to right along the horizontal axis. In conceptual diagram 500, the vertical axis indicates buffer addresses corresponding to the capture and storage of an image frame, expressed as a percentage. The percentage indicates the percentage of image frames captured and / or stored in the image frame buffer at a given time. Image frames are captured and stored in four quarters: the first quarter (marked 1 / 4, corresponding to a range of 0% to 25%), the second quarter (marked 2 / 4, corresponding to a range of 25% to 50%), the third quarter (marked 3 / 4, corresponding to a range of 50% to 75%), and the fourth quarter (marked 4 / 4, corresponding to a range of 75% to 100%).

[0076]

[0087] In the timeline, exposure of the image sensor to light is identified by a box marked "Exp." Analog-to-digital conversion is identified by a box marked "A / D." The start of data transfer from the image sensor 130 of the camera 100 / 105A / 205 / 305 to the image buffer and / or image processor (e.g., via MIPI / CSI) is marked as "SOF" for "start of frame." The end of data transfer from the image sensor 130 of the camera 100 / 105A / 205 / 305 to the image buffer and / or image processor (e.g., via MIPI / CSI) is marked as "EOF" for "end of frame." The transfer of an entire frame from the image sensor 130 to the image buffer and / or image processor from the start of the frame to the end of the frame is identified as a full frame transfer 540 to the image buffer and / or image processor. The frame data may be transferred via MIPI, CSI, or a combination thereof. The boxes marked "Cam" for each of the four quarters correspond to an image sensor capturing those quarters of an image frame and transferring the quarters of the image frame to an image frame buffer and / or image processor. The boxes marked "Warping" for each of the four quarters correspond to image processing described herein as being performed by an image processor. The image processor may include the compositor 225, the XR application 215, the image processor 150, the host processor 152, the ISP 154, the GPU 325, or a combination thereof. The image processing operations corresponding to the "Warping" boxes may include compositing (e.g., rendering virtual content and merging the rendered virtual content with an image) as described with respect to the compositor 225, the XR application 215, and the GPU 325. The image processing operations corresponding to the "warping" box may include other image processing operations described with respect to the compositor 225, the XR application 215, the image processor 150, the host processor 152, the ISP 154, the GPU 325, or combinations thereof.The diagonally marked display interface pointer 530 corresponds to the transfer of the processed partial image frame data (corresponding to one-quarter of an image frame) to a display buffer and / or a display. The display buffer may be, for example, display buffer 230, display buffer 330, or a combination thereof. The display may be, for example, display 235, display 335, or a combination thereof. The display interface pointer 530 may point to the display buffer and / or a display using a display interface such as a display serial interface (DSI), a MIPI interface, another type of interface, or a combination thereof.

[0077]

[0088] The shaded blocks to the right of the line and the diagonal arrow pointing up and to the right of the display interface pointer 530 in FIG. 5 may correspond to the transfer of processed partial image frame data from the display buffer to the display and / or the display of at least a portion of the processed image by the display. In some examples, the display may automatically display the processed portion of the image frame once the processed portion of the image frame is received by the display buffer. For example, an organic light emitting diode (OLED) display may be used in this manner. In some examples, the display may wait to display the processed image frame once all of the processed portion of the image frame is received by the display buffer. For example, a liquid crystal display (LCD) display may be used in this manner.

[0078]

[0089] As shown above the timeline of FIG. 5, the first half of the display may be populated in approximately 12 milliseconds (ms) from exposure to arrival at the display buffer and / or display. The entire display may be populated in approximately 20.5 ms from exposure to arrival at the display buffer and / or display. Image processing to be distributed over time, performed in parallel. Thus, dividing the image frame transfer and processing into portions, such as the quarters of FIG. 5, may allow the processed image portions to arrive at the display buffer and / or display more quickly. Dividing the image frame transfer and processing into portions, such as the quarters of FIG. 5, may allow the image processing to be distributed over time, giving the image processor dedicated time to process different portions of the image as they come in, and allowing the image processor less work at the end of the frame because earlier portions of the image have already been at least partially processed by that time. Dividing the image frame transfer and processing into portions, such as the quarters of FIG. 5, may allow the image processing to be performed at least partially in parallel, such as allowing the image processor to begin processing a second portion of the image while a first portion of the image is still undergoing processing.

[0079]

[0090] FIG. 6 is a conceptual diagram 600 illustrating a reduction in latency between capture and display of an image frame, according to some examples. The conceptual diagram 600 shows three graphs 610, 620, and 630. Each of the graphs has a horizontal axis with a frame number. Each of the graphs has a horizontal axis with latency in milliseconds. The conceptual diagram 600 includes a first graph 610 illustrating conventional camera stack latency for a conventional camera stack that does not include the partial image frame transfer technique described herein (e.g., shown in FIG. 3 and FIG. 5) and does not include the synchronization of cameras and virtual content latency described herein with respect to the XR camera synchronizer of FIG. 2. In the first graph 610 illustrating conventional camera stack latency, the latency is high (typically around 50 ms) and has high variability.

[0080]

[0091] The conceptual diagram 600 includes a second graph 620 illustrating XR application synchronized camera latency for a camera stack that performs synchronization of cameras and virtual content latency as described herein with respect to the XR camera synchronizer of Figure 2. In the second graph 620 illustrating XR application synchronized camera latency, the latency is high (generally above 50 ms) but has reduced variability compared to the conventional camera stack latency of the first graph 610.

[0081]

[0092] The conceptual diagram 600 includes a third graph 630 showing latency with an XR application that performs synchronization of the camera with the virtual content latency as described herein with respect to the XR camera synchronizer of Figure 2 and uses partial frame delivery as described herein (e.g., shown in Figures 3 and 5). In the third graph 640 showing latency with an XR application that is synchronized with the camera and uses partial frame delivery, the latency is low (typically around 10-15 ms) and also has low variability.

[0082]

[0093] In some examples, the use of camera and virtual content latency synchronization described herein (e.g., with respect to the XR camera synchronizer of FIG. 2) and partial frame delivery described herein (e.g., shown in FIGS. 3 and 5) can provide significant improvements in latency and consistency. In one illustrative example, the average availability of the first ½ frame in a set of frames is reduced from 47 ms to 14 ms. In one illustrative example, the standard deviation of frame availability is reduced from 10 ms to 0.1 ms. In one illustrative example, the timing of processing (e.g., as determined by latency+2×stdev) (by compositor 225 of FIG. 2 or “warping” of FIG. 5) is reduced from 67 ms to 14.2 ms.

[0083]

[0094] 7 is a flow diagram 700 illustrating an image capture, processing, and display technique according to some examples. In some examples, the image capture, processing, and display technique illustrated by the flow diagram 700 is performed by an image processing system. In one illustrative example, the image processing system is an XR system 200. In some examples, the image processing system includes at least a portion of an image capture and processing system 100, an image capture device 105A, an image processing device 105B, an XR system 200, an XR application 215, a compositor 225, an imaging system 300, a GPU 325, a system for capturing and processing images according to a timeline 500, an imaging system 800, an imaging system 880, an image processing system performing an operation 900, an image processing system performing an operation 950, one or more network servers of a cloud service, a computing system 1000, a mobile device, a mobile handset, a wireless communication device, a head mounted display (HMD), a camera, or a combination thereof.

[0084]

[0095] In operation 705, the image processing system stores existing data in an image frame buffer. In operation 710, the image processing system modifies the existing data in the image frame buffer by overwriting at least a predetermined area of ​​the image frame buffer with a watermark. The watermark is sometimes referred to as metadata.

[0085]

[0096] In operation 715, the image processing system receives new image frame data corresponding to a new image frame, and the new image frame data is captured by the image sensor. In some examples, the image processing system may include a connector coupled to the image sensor 130, and the image may be received using the connector. The connector may include a port, a jack, a wire, an input / output (IO) pin, a conductive trace on a printed circuit board (PCB), any other type of connector described herein, or some combination thereof. In some examples, the image processing system may include the image sensor 130 that captures the image.

[0086]

[0097] In operation 720, the image processing system stores the new image frame data in the image frame buffer by at least overwriting the existing data in the image frame buffer with the new image frame data. In operation 725, the image processing system reads a predetermined area of ​​the image frame buffer.

[0087]

[0098] At operation 730, the image processing system identifies that a watermark in a predetermined region of the image frame buffer has been overwritten based on reading the predetermined region of the image frame buffer. At operation 735, the image processing system determines that the new image frame data includes at least a predetermined percentage of the new image frame based on identifying that the predetermined region of the image frame buffer has been overwritten.

[0088]

[0099] In operation 740, the image processing system receives a first portion of new image frame data from the image frame buffer, the first portion corresponding to a predetermined percentage of the new image frame. In operation 745, the image processing system processes the first portion of the new image frame data. In operation 750, the image processing system sends the first portion of the new image frame data to the display buffer.

[0089]

[0100] In operation 755, the image processing system receives from the image frame buffer a second portion of new image frame data corresponding to the new image frame. In some examples, the second watermark (second metadata) may overwrite the second predetermined region of the existing data in operation 710. In some examples, the image processing system may read the second predetermined region of the image frame buffer. In some examples, the image processing system may identify that the second watermark in the second predetermined region of the image frame buffer has been overwritten based on reading the predetermined region of the image frame buffer. In some examples, the image processing system may determine that the new image frame data includes at least a second predetermined percentage of the new image frame based on identifying that the second predetermined region of the image frame buffer has been overwritten. These operations may occur prior to operation 755. In some examples, operation 755 may occur once the remainder of the new image frame (other than the first portion of the new image frame data) is received, in which case the second portion of the new image frame data may include the remainder of the new image frame.

[0090]

[0101] In operation 760, the image processing system processes the second portion of the new image frame data. In operation 765, the image processing system sends the second portion of the new image frame data to the display buffer.

[0091]

[0102] At operation 770, the image processing system determines, in response to sending the first portion of the new image frame data and the second portion of the new image frame data to the display buffer, that the display buffer contains a display image based on the new image frame. At operation 775, the image processing system displays the display image on the display.

[0092]

[0103] FIG. 8A is a block diagram illustrating an imaging system 800 for auto-exposure, according to some examples. An auto-exposure system automatically adjusts exposure settings for the image sensor 130 and / or control mechanism 120 for capturing one or more images, generally based on ambient lighting (e.g., determined based on a previous image captured by the same image sensor 130 and / or control mechanism 120). An auto-exposure system automatically adjusts exposure settings using hardware, software, or a combination thereof. An auto-exposure system can add latency to an imaging system while automatically adjusting exposure settings. In certain applications, such as extended reality (XR), the latency added by an auto-exposure system can degrade performance, for example, by desynchronizing the view that the XR device displays to the user with the view that the user should have based on the pose of the XR device and / or the user. The latency can have a particularly performance-degrading effect when combined with demanding features such as six degrees of freedom (6DOF) and / or video pass-through (VPT).

[0093]

[0104] In some examples, an auto-exposure system that automatically adjusts exposure settings using hardware may add less latency (e.g., 2-3 frames) than an auto-exposure system that automatically adjusts exposure settings using software (e.g., 8-10 frames). However, some imaging systems may still automatically adjust exposure settings using software due to limitations imposed by, for example, hardware settings, operating systems, and / or software applications within or with which the imaging system operates.

[0094]

[0105] The imaging system 800 includes a frame capture queue 805 that may include one or more image frames captured by the image sensor 130. The imaging system 800 includes an image statistics engine 810 that may determine (e.g., calculate) in hardware and / or software one or more image statistics for one or more image frames in the frame capture queue 805. The one or more image statistics may include one or more average brightness levels of the image frames in the frame capture queue 805, one or more ambient brightness levels of the image frames in the frame capture queue 805, one or more average luminosity levels of the image frames in the frame capture queue 805, one or more minimum brightness levels of the image frames in the frame capture queue 805, one or more minimum luminosity levels of the image frames in the frame capture queue 805, one or more maximum brightness levels of the image frames in the frame capture queue 805, one or more maximum luminosity levels of the image frames in the frame capture queue 805, one or more brightness curves of the image frames in the frame capture queue 805, one or more luminosity curves of the image frames in the frame capture queue 805, or a combination thereof. In some examples, an image frame may be in the frame capture queue 805 for a period of time before the image statistics engine receives and / or determines image statistics for the image frame. In some examples, the image sensor 130 may capture one or more image frames (e.g., 1-2 frames) during that period. In some examples, the period of time may contribute to auto-exposure latency.

[0095]

[0106] The imaging system 800 includes an auto-exposure setting engine 815 that can determine (e.g., calculate) exposure settings for the image sensor 130 and / or the control mechanism 120 based on one or more image statistics determined by the image statistics engine 810. For example, if the image statistics determined by the image statistics engine 810 indicate that the images in the frame capture queue 805 are very dim or dark, the auto-exposure setting engine 815 can increase the exposure for the exposure setting, for example, by increasing the exposure time, decreasing the shutter speed, increasing the aperture size, and / or increasing the ISO. If the image statistics determined by the image statistics engine 810 indicate that the images in the frame capture queue 805 are very bright or bright, the auto-exposure setting engine 815 can decrease the exposure for the exposure setting, for example, by decreasing the exposure time, increasing the shutter speed, decreasing the aperture size, and / or decreasing the ISO. If the image statistics determined by the image statistics engine 810 indicate that the images in the frame capture queue 805 are adequately exposed, the auto exposure settings engine 815 may maintain the previously set exposure settings.

[0096]

[0107] The imaging system 800 includes an exposure setting queue 820. The exposure setting queue 820 can include settings to be applied to image frames to be captured by the image sensor 130, such as exposure settings determined by the auto exposure setting engine 815. The exposure setting queue 820 can hold settings before the settings are applied in hardware to the image sensor 130 and / or the control mechanism 120 via a request loop 825. The exposure setting queue 820 can be referred to as an exposure setting buffer, a setting queue, a setting buffer, or some combination thereof. In some examples, settings including exposure settings determined by the auto exposure setting engine 815 can be in the exposure setting queue 820 for a period of time. In some examples, the image sensor 130 can capture one or more image frames (e.g., 1-2 frames) during that period. In some examples, the time period can contribute to auto exposure latency.

[0097]

[0108] Request loop 825 can submit requests, including hardware process capture request 830, to capture hardware, such as image sensor 130, control mechanism 120, and / or other elements of image capture device 105A. The hardware, represented on the left side of FIG. 8A by blocks 835-860 between hardware process capture request 830 and hardware process capture results 865, requests the capture of a sequence of image frames. The frame requests for each of the sequence of image frames are identified as Frame Request N, Frame Request N+1, Frame Request N+2, Frame Request N+3, Frame Request N+4, Frame Request N+5, and Request Frame N+6. The hardware process may ultimately capture image frames corresponding to frame requests such as image frame N corresponding to frame request N, image frame N+1 corresponding to frame request N+1, image frame N+2 corresponding to frame request N+2, image frame N+3 corresponding to frame request N+3, image frame N+4 corresponding to frame request N+4, image frame N+5 corresponding to frame request N+5, and / or image frame N+6 corresponding to frame request N+6.

[0098]

[0109] In an illustrative example, image frames N+6 and N+5 may be enqueued in a frame request queue 835. The frame request queue 835 may be referred to as a frame request queue, a frame capture request queue, a capture request queue, a request queue, a frame queue, a capture queue, a frame request buffer, a frame capture request buffer, a capture request buffer, a request buffer, a frame buffer, a capture queue, or combinations thereof. The frame requests (e.g., frame requests N+6 and N+5) may include an auto-exposure setting determined by the auto-exposure setting engine 815. Each of the frame requests (e.g., frame requests N+6 and N+5) may be automatically enqueued in the frame request queue when a slot becomes available in the frame request queue 835. Each of frame requests N, N+1, N+2, N+3, N+4, N+5, and N+6 is either currently in the frame request queue 835 or was previously in the frame request queue 835 as of the instant shown in FIG. 8A.

[0099]

[0110] The capture hardware reads frame request N+4 and obtains the exposure settings determined by the auto exposure setting engine 815. The capture hardware sets the exposure settings to be used and sends the exposure settings to the kernel 840. While the capture hardware reads frame request N+3, the exposure settings determined by the auto exposure setting engine 815 and sent to the kernel (operation 840) are waiting in the kernel 845. While the capture hardware reads frame request N+2, the exposure settings determined by the auto exposure setting engine 815 are being written to the hardware 850, for example in the control mechanism 120. Frame request N+1 is identified as an active sensor frame 855, indicating that the capture hardware is capturing image frame N+1 corresponding to frame request N+1 and that the capture hardware will apply the exposure settings determined by the auto exposure setting engine 815 for the capture of image frame N+1 (e.g., at or before the capture of image frame N+1). Frame request N has already been captured at the time shown in Figures 8A-8B. The capture hardware sends frame request N to the frame capture queue 805. The capture hardware may initiate a return from kernel space to user space in operation 860. The transition from kernel space to user space may take some time. Operation 860 may also include metadata processing on frame N+1 and / or packed data structures for return to user space as part of hardware process capture results 865. The hardware sends the hardware process capture results 865 to a software process represented by blocks 805-825 on the right side of FIG. 8A. In some examples, the software process may execute in user space rather than kernel space. The hardware process capture results 865 may include one or more of image frames N-N+6 to be stored in the frame queue 805.

[0100]

[0111] 8B is a block diagram illustrating an imaging system 880 for auto-exposure with partial frame delivery, according to some examples. The imaging system 880 includes the components and operations of the imaging system 800, but uses a partial frame delivery engine 870 to speed up the auto-exposure process.

[0101]

[0112] The partial frame delivery engine 870 can deliver the portion of the active sensor frame 855 (image frame N+1) directly to the image statistics engine 810. The portion of the active sensor frame 855 (image frame N+1) is sent to the image statistics engine 810 as in FIG. 2, FIG. 3, FIG. 4, FIG. 5, FIG. 7, FIG. 9A, and / or FIG. 9B. For example, the partial frame delivery engine 870 can add a watermark 320 or other marking to an image buffer where image data from the active sensor frame 855 (image frame N+1) should be written. The partial frame delivery engine 870 retrieves the partial frame data for the active sensor frame 855 (image frame N+1) that represents a portion of the active sensor frame 855 (image frame N+1) when some watermark 320 or marking is overwritten with image data from the active sensor frame 855 (image frame N+1). The partial frame delivery engine 870 sends the partial frame data for the active sensor frame 855 (image frame N+1) to the image statistics engine 810, which can begin to determine image statistics for the active sensor frame 855 (image frame N+1) on a frame portion by frame basis. In some examples, the partial frame delivery engine 870 retrieves image data from the active sensor frame 855 (image frame N+1) line by line and sends the lines to the image statistics engine 810. In some examples, each partial frame includes one or more lines of the active sensor frame 855 (image frame N+1).

[0102]

[0113] The imaging system 880 can accelerate the auto-exposure process by bypassing some operations, the corresponding blocks of which in FIG. 8B are shown shaded and outlined with dashed lines, and the corresponding arrows in FIG. 8B are shown with dashed lines. For example, the imaging system 880 can accelerate the auto-exposure process by bypassing the start software process 860 and the frame capture queue 805. Accelerating the auto-exposure process can reduce latency in the auto-exposure process. Reducing latency in the auto-exposure process can improve performance of, for example, an XR system. For example, reducing latency in the auto-exposure process can help synchronize the view that the XR device displays to the user with the view that the user should have based on the pose of the XR device and / or the user. Reducing latency in the auto-exposure process can allow demanding features such as six degrees of freedom (6DOF) and / or video pass-through (VPT) to function smoother and more consistently.

[0103]

[0114] Although imaging system 800 of FIG. 8A and imaging system 880 of FIG. 8B perform automatic exposure, similar procedures can be used for autofocus, automatic zoom, automatic gain adjustment, automatic brightness adjustment, automatic contrast adjustment, automatic tone mapping, automatic sharpness adjustment, automatic saturation adjustment, and / or other automatic image processing operations based on image statistics.

[0104]

[0115] 9A is a flow chart 900 illustrating an image processing technique according to some examples. In some examples, the image processing technique illustrated by the flow chart 900 is performed by an image processing system. In one illustrative example, the image processing system is an XR system 200. In some examples, the image processing system includes at least a portion of an image capture and processing system 100, an image capture device 105A, an image processing device 105B, an XR system 200, an XR application 215, a compositor 225, an imaging system 300, a GPU 325, a system for capturing and processing images according to a timeline 500, an image processing system performing operations 700, an imaging system 800, an imaging system 880, an image processing system performing operations 950, one or more network servers of a cloud service, a computing system 1000, a mobile device, a mobile handset, a wireless communication device, a head mounted display (HMD), a camera, or a combination thereof.

[0105]

[0116] In operation 905, the image processing system stores existing data marked with metadata in a predetermined area of ​​the image frame buffer in the image frame buffer. The metadata of operation 905 may be or may include a watermark. Watermark 320, watermark 430, and watermark 435 may be examples of metadata of operation 905. Watermark 320, watermark 430, and watermark 435 may be examples of watermarks included in the metadata of operation 905. Operation 905 may correspond to at least a subset of operations 705-710 of FIG. 7. The image processing system may include an image frame buffer. The image processing system may include an image buffer including one or more image frame buffers, such as image buffer 210 and / or image buffer 310. Image buffer 210 and / or image buffer 310 may include the image frame buffer of operation 905. Image frame buffer 315 of FIG. 3 may be an example of an image frame buffer of operation 905. The unshaded white area at the bottom of image frame buffer 315 in Figure 3 may be an example of existing data for operation 905. Existing image frame buffer data 450 and existing image frame buffer data 455 in Figure 4 may be examples of existing data for operation 905. The metadata for operation 905 may include a color pattern, a frame identifier associated with the new image frame, an image frame buffer identifier associated with the image frame buffer, or a combination thereof. The metadata for operation 905 may include any pattern or information described herein as being capable of being included in any of watermark 320, watermark 430, and watermark 435.

[0106]

[0117] In some examples, prior to operation 905, the image processing system modifies existing data in a predetermined region of the image frame buffer to mark the existing data with metadata. For example, the existing data may not have been previously marked with metadata in the predetermined region and the image processing system may insert the metadata in the predetermined region and / or overwrite a portion of the existing data in the predetermined region. For example, the image processing system may have inserted watermark 430 into existing image frame buffer data 450 of FIG. 4 and / or inserted watermark 435 into existing image frame buffer data 455 of FIG. 4.

[0107]

[0118] In some examples, the existing data is previous image frame data from a previous image frame captured prior to the capture of the new image frame. For example, existing image frame buffer data 450 and existing image frame buffer data 455 of FIG. 4 include image data depicting a room with a chandelier. Similarly, new image frame data 420 and new image frame data 425 of FIG. 4 appear to include image data depicting the same room with the same chandelier captured from slightly different positions (see, for example, the difference between existing image frame buffer data 450 and new image frame data 420 along seam 440, and the difference between existing image frame buffer data 455 and new image frame data 425 along seam 445). The previous image frame and new image frame can be two different image frames in the same video, with the previous image frame appearing earlier in the video than the new image frame and the new image frame appearing later in the video than the previous image frame.

[0108]

[0119] In operation 910, the image processing system receives new image frame data corresponding to a new image frame, the new image frame data being received by the image sensor. The new image frame data being received by the image sensor may include new image frame data being captured by the image sensor. In some examples, the image processing system may include a connector coupled to the image sensor 130, and the image may be received using the connector. The connector may include a port, a jack, a wire, an input / output (IO) pin, a conductive trace on a printed circuit board (PCB), any other type of connector described herein, or some combination thereof. In some examples, the image processing system may include the image sensor 130 that captures the image. Operation 910 may correspond to operation 715 of FIG. 7. The shaded dithered area of ​​the image frame buffer 315 of FIG. 3 may be an example of new image frame data of operation 910. New image frame data 420 and new image frame data 425 of FIG. 4 may be examples of new image frame data of operation 910. The data in the "Cam" block of FIG. 5 may be an example of new image frame data of operation 910.

[0109]

[0120] In operation 915, the image processing system overwrites at least a portion of the existing data in the image frame buffer with the new image frame data. Operation 915 may correspond to operation 720 of Figure 7. Operation 915 may include storing the new image frame data in the image frame buffer.

[0110]

[0121] In operation 920, the image processing system determines that the new image frame data includes at least a predetermined amount of the new image frame based on a read of the predetermined region of the image frame buffer indicating that metadata will no longer be stored in the predetermined region. Operation 920 may correspond to at least a subset of operations 725-735 of FIG. 7. In some examples, to determine that the new image frame data includes at least a predetermined amount of the new image frame, the image processing system reads the predetermined region of the image frame buffer and identifies that metadata in the predetermined region of the image frame buffer has been overwritten based on reading the predetermined region of the image frame buffer. The memory read in FIG. 3 indicated by the dashed arrow from GPU 325 to one of the watermarks 320 in image frame buffer 315 of FIG. 3 is an example of a memory read that may be the basis for the determination of operation 920. The memory read of the region above seam 440 in left image frame buffer 410 may be an example of a memory read that may be the basis for the determination of operation 920. A memory read of the area above seam 445 in right image frame buffer 415 may be one example of a memory read that may be the basis for the determination of action 920.

[0111]

[0122] The predetermined amount of the new image frame may include a predetermined percentage of the new image frame, a predetermined proportion of the new image frame, a predetermined number of rows of the new image frame, a predetermined percentage of the rows of the new image frame, a predetermined percentage of the rows of the new image frame, a predetermined number of columns of the new image frame, a predetermined percentage of the columns of the new image frame, a predetermined percentage of the columns of the new image frame, a predetermined number of pixels of the new image frame, a predetermined percentage of pixels of the new image frame, a predetermined percentage of pixels of the new image frame, or a combination thereof.

[0112]

[0123] In operation 925, the image processing system outputs at least a first portion of the new image frame data corresponding to a predetermined amount of the new image frame. Outputting the first portion of the new image frame data may include sending the first portion of the new image frame data to an application (e.g., XR application 215), processing the new image frame data using an application (e.g., XR application 215), compositing the new image frame data with virtual data (e.g., using compositor 225), sending the first portion of the new image frame data to a display buffer (e.g., display buffer 230, display buffer 330, DSI read pointer of FIG. 5), displaying the first portion of the new image frame data using a display (e.g., display 235, display 335, DSI read pointer of FIG. 5), sending the first portion of the new image frame data to another device, or combinations thereof. Operation 920 may correspond to at least a subset of operations 740-775 of FIG. 7.

[0113]

[0124] In some examples, to output the first portion of the new image frame data as in operation 925, the image processing system is configured to and can send the first portion of the new image frame data to an extended reality (XR) application (e.g., XR application 215 of FIG. 2) and / or another type of application. In some examples, to output the first portion of the new image frame data as in operation 925, the image processing system is configured to and can process the first portion of the new image frame data using an extended reality (XR) application (e.g., XR application 215 of FIG. 2) and / or another type of application. In some examples, to output the first portion of the new image frame data as in operation 925, the image processing system is configured to and can composite the first portion of the new image frame data with virtual content (e.g., virtual content from virtual content rendering buffer 220 of FIG. 2). In some examples, to output the first portion of the new image frame data as in operation 925, the image processing system is configured to and can perform warping, distortion, and / or distortion correction on at least a portion of the first portion of the new image frame data.

[0114]

[0125] In some examples, to output the first portion of the new image frame data as in operation 925, the one or more processors store the first portion of the new image frame data in a display buffer. In some examples, the image processing system includes a display buffer. Examples of display buffers include I / O 156 of FIG. 1, I / O 160 of FIG. 1, display buffer 230 of FIG. 2, display buffer 330 of FIG. 3, DSI read pointer indicator of FIG. 5, output device 1035 of FIG. 10, or a combination thereof.

[0115]

[0126] In some examples, to output the first portion of the new image frame data as in operation 925, the one or more processors display the first portion of the new image frame data using a display. In some examples, the image processing system includes a display. Examples of displays include I / O 156 of FIG. 1, I / O 160 of FIG. 1, display 235 of FIG. 2, display 335 of FIG. 3, DSI read pointer indicator of FIG. 5, and output device 1035 of FIG. 10, or a combination thereof.

[0116]

[0127] In some examples, to output the first portion of the new image frame data as in operation 925, the one or more processors send the first portion of the new image frame data to a device. The device may be an external device. The device may be a remote computing device, such as a remote server for storing image data and / or servicing stored image data. The device may be a display device, such as a display screen, projector, printer, or another type of display device. The device may be a computing system 1000. In some examples, to output at least the first portion of the new image frame data, the image processing system is configured to and may output the new image frame. For example, if the predetermined area of ​​the image frame buffer is at the end of the image frame buffer (or, in some cases, is expected to be at the end of the new image frame), operation 920 may indicate that the entirety of the new image frame has been received and may be output in operation 925.

[0117]

[0128] In some examples, the existing data is marked with second metadata in a second predetermined area of ​​the image frame buffer. The second metadata may be a second watermark and may include any type of data described as being included in the metadata and / or watermark with respect to operation 905. In some examples, the image processing system may determine that the new image frame data includes at least a second predetermined amount of the new image frame based on a second read of the second predetermined area of ​​the image frame buffer indicating that the second metadata will no longer be stored in the second predetermined area. Any details described herein with respect to operation 920 may apply to determining that the new image frame data includes at least a second predetermined amount of the new image frame. Determining that the new image frame data includes at least a second predetermined amount of the new image frame may correspond to operation 755 of FIG. 7. In some examples, the image processing system may output at least a second portion of the new image frame data corresponding to the second predetermined amount of the new image frame. Any details described herein with respect to operation 925 may apply to outputting at least a second portion of the new image frame data. Outputting at least the second portion of the new image frame data may correspond to any of acts 755-775 in FIG.

[0118]

[0129] 9B is a flow chart 900 illustrating an image processing technique according to some examples. In some examples, the image processing technique illustrated by the flow chart 900 is performed by an image processing system. In one illustrative example, the image processing system is the XR system 200. In some examples, the image processing system includes at least a portion of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the XR system 200, the XR application 215, the compositor 225, the imaging system 300, the GPU 325, the system for capturing and processing images according to the timeline 500, the image processing system performing the operations 700, the imaging system 800, the imaging system 880, one or more network servers of the cloud service, the image processing system performing the operations 900, the computing system 1000, a mobile device, a mobile handset, a wireless communication device, a head mounted display (HMD), a camera, or a combination thereof.

[0119]

[0130] In operation 955, the image processing system stores the data marked with the metadata in a predetermined area of ​​the image frame buffer in the image frame buffer. The data stored in the image frame buffer in operation 955 may be referred to as stored data or pre-existing data (as in operation 905 of FIG. 9A or operations 705-710 of FIG. 7). Operation 955 may correspond to operation 905 of FIG. 9A, operation 705 of FIG. 7, and / or operation 710 of FIG. 7. The metadata may be and / or may include one or more watermarks. Examples of metadata may include watermark 320, watermark 430, and watermark 435. Examples of one or more watermarks included in the metadata of operation 955 include watermark 320, watermark 430, and watermark 435.

[0120]

[0131] The image processing system may include an image frame buffer. The image processing system may include an image buffer including one or more image frame buffers, such as image buffer 210 and / or image buffer 310. Image buffer 210 and / or image buffer 310 may include the image frame buffer of operation 955. Image frame buffer 315 of FIG. 3 may be an example of an image frame buffer of operation 955. The unshaded white area at the bottom of image frame buffer 315 of FIG. 3 may be an example of stored data of operation 955. Examples of stored data may include existing image frame buffer data 450 and existing image frame buffer data 455 of FIG. 4. The metadata of operation 955 may include a pattern of colors, a frame identifier associated with the image frame, an image frame buffer identifier associated with the image frame buffer, or a combination thereof. The metadata of operation 955 may include any pattern or information described herein as being capable of being included in any of watermark 320, watermark 430, and watermark 435.

[0121]

[0132] In some examples, prior to operation 955, the image processing system modifies the stored data in a predetermined region of the image frame buffer to mark the stored data with metadata. For example, the stored data may have been previously marked with metadata in the predetermined region, and the image processing system may insert the metadata in the predetermined region and / or overwrite a portion of the stored data in the predetermined region. For example, the image processing system may insert watermark 430 into existing image frame buffer data 450 of FIG. 4 and / or insert watermark 435 into existing image frame buffer data 455 of FIG. 4 and / or insert watermark 320 into data stored in image frame buffer 315 of FIG. 3.

[0122]

[0133] In operation 960, the image processing system receives image frame data corresponding to an image frame from the image sensor. Operation 960 may correspond to operation 910 of FIG. 9A and / or operation 715 of FIG. 7. The image frame data being received by the image sensor may include image frame data being captured by the image sensor. In some examples, the image processing system may include a connector coupled to the image sensor 130, and the image may be received using the connector. The connector may include a port, a jack, a wire, an input / output (IO) pin, a conductive trace on a printed circuit board (PCB), any other type of connector described herein, or some combination thereof. In some examples, the image processing system may include the image sensor 130 that captures the image. The shaded dithered area of ​​the image frame buffer 315 of FIG. 3 may be an example of image frame data of operation 960. Image frame data 420 and image frame data 425 of FIG. 4 may be examples of image frame data of operation 960. The data in the "Cam" block of FIG. 5 may be an example of image frame data of operation 960.

[0123]

[0134] In some examples, the stored data is (and / or includes) previous image frame data from a previous image frame captured (by the image sensor or by a different second image sensor) and / or prior to the capture and / or storage of the image frame of operation 960. For example, existing image frame buffer data 450 and existing image frame buffer data 455 of FIG. 4 include image data depicting a room with a chandelier. Similarly, image frame data 420 and image frame data 425 of FIG. 4 appear to include image data depicting the same room with the same chandelier, captured from slightly different positions (see, for example, the difference between existing image frame buffer data 450 and image frame data 420 along seam 440, and the difference between existing image frame buffer data 455 and image frame data 425 along seam 445). The previous image frame and image frame may be two different image frames in the same video, with the previous image frame appearing earlier in the video than the image frame and the image frame appearing later in the video than the previous image frame.

[0124]

[0135] In some examples, the image processing system receives a request for a view of the scene prior to operation 960. In response to receiving the request, the image processing system can automatically expose the image sensor to light from the scene, for example, by opening an aperture. The image frame data received from the image sensor in operation 960 can be based on the exposure of the image sensor to light from the scene. For example, an XR device such as an HMD can request a pass-through image of the scene in front of the XR device at a particular time (e.g., when requested by a user of the XR device or when the scene or something based on the scene needs to be shown to the user of the XR device), but may not necessarily need a pass-through image of the scene.

[0125]

[0136] In operation 965, the image processing system overwrites at least a portion of the stored data in the image frame buffer with the image frame data. Operation 965 may correspond to operation 915 of Figure 9A and / or operation 720 of Figure 7. As part of operation 965, the image processing system may store the image frame data in the image frame buffer. Operation 965 may include storing the image frame data in the image frame buffer.

[0126]

[0137] In operation 970, the image processing system determines that the image frame data includes at least a predetermined amount of the image frame based on a reading of a predetermined region of the image frame buffer indicating that the predetermined region stores additional data distinct from the metadata. Operation 970 may correspond to operation 920 of FIG. 9A, operation 725 of FIG. 7, operation 730 of FIG. 7, and / or operation 735 of FIG. 7. In some examples, the additional data is (and / or includes) at least a portion of the image frame data of operation 965. The additional data may be referred to as alternative data, new data, new image data, secondary data, different data, separate data, data, information, or another type of data.

[0127]

[0138] As part of operation 970, the image processing system may read a predetermined region of the image frame buffer, which may be referred to as the "read" in operation 970. Based on reading the predetermined region of the image frame buffer, the image processing system may determine that the predetermined region stores additional data distinct from the metadata. Based on reading the predetermined region of the image frame buffer, the image processing system may identify that metadata in the predetermined region of the image frame buffer has been overwritten (e.g., with additional data and / or at least a portion of the image frame data of operation 965) because, for example, the metadata is no longer found in the predetermined region of the image frame buffer. The memory read in FIG. 3 indicated by the dashed arrow from the GPU 325 to one of the watermarks 320 in the image frame buffer 315 of FIG. 3 is an example of a memory read that may be the basis for the determination of operation 970. The memory read of the region above the seam 440 in the left image frame buffer 410 may be an example of a memory read that may be the basis for the determination of operation 970. A memory read of the area above seam 445 in right image frame buffer 415 may be one example of a memory read that may be the basis for the determination of action 970 .

[0128]

[0139] The predetermined amount of the image frame may include a predetermined percentage of the image frame, a predetermined proportion of the image frame, a predetermined number of rows of the image frame, a predetermined percentage of the rows of the image frame, a predetermined percentage of the rows of the image frame, a predetermined number of columns of the image frame, a predetermined percentage of the columns of the image frame, a predetermined number of pixels of the image frame, a predetermined percentage of the pixels of the image frame, a predetermined percentage of the pixels of the image frame, or a combination thereof.

[0129]

[0140] In operation 975, the image processing system outputs at least a first portion of the image frame data corresponding to a predetermined amount of the image frame. Operation 975 may correspond to operation 925 of Figure 9A, operation 740 of Figure 7, operation 745 of Figure 7, operation 750 of Figure 7, operation 755 of Figure 7, operation 760 of Figure 7, operation 765 of Figure 7, operation 770 of Figure 7, and / or operation 775 of Figure 7.

[0130]

[0141] Outputting the first portion of the image frame data may include sending the first portion of the image frame data to an application (e.g., XR application 215), processing the image frame data using an application (e.g., XR application 215), compositing the image frame data with virtual data (e.g., using compositor 225), sending the first portion of the image frame data to a display buffer (e.g., display buffer 230, display buffer 330, DSI read pointer of FIG. 5), displaying the first portion of the image frame data using a display (e.g., display 235, display 335, DSI read pointer of FIG. 5), sending the first portion of the image frame data to another device, or combinations thereof.

[0131]

[0142] In some examples, to output the first portion of the image frame data as in operation 975, the image processing system is configured to and can send the first portion of the image frame data to an extended reality (XR) application (e.g., XR application 215 of FIG. 2) and / or another type of application. In some examples, to output the first portion of the image frame data as in operation 975, the image processing system is configured to and can process the first portion of the image frame data using an extended reality (XR) application (e.g., XR application 215 of FIG. 2) and / or another type of application. In some examples, to output the first portion of the image frame data as in operation 975, the image processing system is configured to and can composite the first portion of the image frame data with virtual content (e.g., virtual content from virtual content rendering buffer 220 of FIG. 2). In some examples, to output the first portion of the image frame data as in operation 975, the image processing system is configured to and can perform warping, distortion, and / or distortion correction on at least a portion of the first portion of the image frame data.

[0132]

[0143] In some examples, to output the first portion of the image frame data as in operation 975, the image processing system stores the first portion of the image frame data in a display buffer. In some examples, the image processing system includes a display buffer. Examples of display buffers include I / O 156 of FIG. 1, I / O 160 of FIG. 1, display buffer 230 of FIG. 2, display buffer 330 of FIG. 3, DSI read pointer indicator of FIG. 5, output device 1035 of FIG. 10, or a combination thereof.

[0133]

[0144] In some examples, to output the first portion of the image frame data as in operation 975, the image processing system displays the first portion of the image frame data using a display. In some examples, the image processing system includes a display. Examples of displays include I / O 156 of FIG. 1, I / O 160 of FIG. 1, display 235 of FIG. 2, display 335 of FIG. 3, DSI read pointer indicator of FIG. 5, and output device 1035 of FIG. 10, or a combination thereof.

[0134]

[0145] In some examples, to output the first portion of the image frame data as in operation 975, the image processing system sends the first portion of the image frame data to a device. The device may be an external device. The device may be a remote computing device, such as a remote server for storing image data and / or servicing stored image data. The device may be a display device, such as a display screen, projector, printer, or another type of display device. The device may be a computing system 1000. In some examples, to output at least the first portion of the image frame data, the image processing system is configured to and may output an image frame. For example, if a predetermined area of ​​the image frame buffer is at the end of the image frame buffer (or, in some cases, is expected to be at the end of the image frame), operation 920 may indicate that the entire image frame has been received and may be output in operation 975.

[0135]

[0146] In some examples, the stored data is marked with second metadata in a second predetermined area of ​​the image frame buffer. The second metadata may be a second watermark and may include any type of data described with respect to operation 905 as being included in the metadata and / or watermark. In some examples, the image processing system may determine that the image frame data includes at least a second predetermined amount of the image frame based on a second read of the second predetermined area of ​​the image frame buffer indicating that the second metadata will no longer be stored in the second predetermined area. Any details described herein with respect to operation 920 may apply to determining that the image frame data includes at least a second predetermined amount of the image frame. Determining that the image frame data includes at least a second predetermined amount of the image frame may correspond to operation 755 of FIG. 7. In some examples, the image processing system may output at least a second portion of the image frame data corresponding to the second predetermined amount of the image frame. Any details described herein with respect to operation 975 may apply to outputting at least a second portion of the image frame data. Outputting at least the second portion of the image frame data may correspond to any of acts 755-775 in FIG.

[0136]

[0147] In some examples, to output the first portion of the image frame data as in operation 975, the image processing system determines one or more image statistics based on at least the first portion of the image frame data, for example, as described with respect to the image statistics engine 810. The image processing system determines an image capture setting based on the one or more image statistics, for example, as described with respect to the auto exposure setting engine 815. In some examples, the image processing system sends the image capture setting to the image capture hardware including the image sensor, for example, as in the request loop 825, the hardware processes capture requests 830, and / or the frame request queue 835. The image processing system receives secondary image frame data from the image sensor corresponding to the second image frame, and the image sensor captures the second image frame based on the image capture setting. For example, the active sensor frame 855 (image frame N+1) is captured based on the image capture setting determined and set by the auto exposure setting engine 815 and sent to the kernel (operation 840) and written to the hardware for use with the active sensor frame 855 (image frame N+1) (operation 850). The image capture setting can be an exposure setting.

[0137]

[0148] In some aspects, the image processing system may include means for storing data marked with metadata in a predetermined area of ​​the image frame buffer in an image frame buffer; means for receiving image frame data from an image sensor corresponding to an image frame; means for overwriting at least a portion of the stored data in the image frame buffer with the image frame data; means for determining that the image frame data includes at least a predetermined amount of the image frame based on a reading of the predetermined area of ​​the image frame buffer indicating that the predetermined area stores new data that differs from the metadata; and means for outputting at least a first portion of the image frame data corresponding to the predetermined amount of the image frame.

[0138]

[0149] In some examples, the means for storing data may include image buffer 170, image buffer 210, image buffer 310, individual image frame buffers 0-15 of image buffer 310, image frame buffer 315, left image frame buffer 410, right image frame buffer 415, memory 1015, storage device 1030, cache 1012, or a combination thereof. In some examples, the means for receiving image frame data may include image sensor 130, image processor 150, ISP 154, host processor 152, I / O 156, image capture device 105A, image processing device 105B, image capture and processing system 100, camera 205, camera 305. In some examples, the means for overwriting at least a portion of the stored data may include image buffer 170, image buffer 210, image buffer 310, individual image frame buffers 0-15 of image buffer 310, image frame buffer 315, left image frame buffer 410, right image frame buffer 415, memory 1015, storage device 1030, cache 1012, or a combination thereof. In some examples, the means for determining that the image frame data comprises at least a predetermined amount of the image frame may include image processing device 105B, image capture and processing system 100, XRapp 215, GPU 325, processor 1010, or a combination thereof. In some examples, the means for outputting at least a first portion of the image frame data may include I / O 156, I / O 160, XRApp 215, XRSDK, compositor 225, display buffer 230, display 235, GPU 325, display buffer 330, display 335, display interface pointer 530, memory 1015, storage device 1030, cache 1012, output device 1035, communication interface 1040, or combinations thereof.

[0139]

[0150] The image processing technique illustrated in flowchart 900 may also include any operations shown and described in or with respect to any of the conceptual diagrams, block diagrams, and flowcharts 100, 200, 300, 500, 700, 800, 880, and / or 1000.

[0140]

[0151] In some cases, at least a subset of the techniques illustrated by any of the conceptual diagrams, block diagrams, and flow diagrams 100, 200, 300, 500, 700, 800, 880, 900, and / or 1000 may be performed remotely by one or more network servers of a cloud service. In some examples, the processes described herein (e.g., processes including those illustrated by the conceptual diagrams, block diagrams, and flow diagrams 200, 300, 500, 700, 800, 880, 900, and / or 1000, and / or other processes described herein) may be performed by a computing system or apparatus. In one example, the processes illustrated by the conceptual diagrams, block diagrams, and flow diagrams 100, 200, 300, 500, 700, 800, 880, 900, and / or 1000 may be performed by the image capture device 105A of FIG. 1. In another example, the processes illustrated by the conceptual diagrams, block diagrams, and flow charts 100, 200, 300, 500, 700, 800, 880, 900, and / or 1000 may be performed by the image processing device 105B of Figure 1. The processes illustrated by the conceptual diagrams, block diagrams, and flow charts 100, 200, 300, 500, 700, 800, 880, 900, and / or 1000 may also be performed by the image capture and processing system 100 of Figure 1. The processes illustrated by the conceptual diagrams, block diagrams, and flow charts 100, 200, 300, 500, 700, 800, 880, 900, and / or 1000 may be performed by the XR system 200 of Figure 2. The processes illustrated by the conceptual diagrams, block diagrams, and flow diagrams 100, 200, 300, 500, 700, 800, 880, 900, and / or 1000 may be performed by the system shown in Figure 3. The processes illustrated by the conceptual diagrams, block diagrams, and flow diagrams 100, 200, 300, 500, 700, 800, 880, 900, and / or 1000 may be performed by a computing system having the architecture of computing system 1000 shown in Figure 10.The computing system may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected wristwatch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or computing device for an autonomous vehicle, a robotic device, a television, and / or any other computing device having resource capabilities to perform the processes described herein, including the processes illustrated by the conceptual diagrams, block diagrams, and flow diagrams 100, 200, 300, 500, 700, 800, 880, 900, and / or 1000. In some cases, a computing system or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform steps of the processes described herein. In some examples, the computing system may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0141]

[0152] Components of a computing system may be implemented in circuits. For example, components may include and / or be implemented using one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry) and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.

[0142]

[0153] The processes illustrated by the conceptual diagrams, block diagrams, and flow diagrams 100, 200, 300, 500, 700, 800, 880, 900, and / or 1000 are organized as logical flow diagrams whose operations represent sequences of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular data type. The order in which the operations are described is not to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement a process.

[0143]

[0154] In addition, the processes illustrated by the conceptual diagrams, block diagrams, and flow charts 100, 200, 300, 500, 700, 800, 880, 900, and / or 1000 and / or other processes described herein may be executed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively run on one or more processors, by hardware, or by a combination thereof. As mentioned above, the code may be stored in a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0144]

[0155] Fig. 10 is a diagram illustrating an example of a system for implementing some aspects of the present technology. In particular, Fig. 10 illustrates an example of a computing system 1000, which may be, for example, an image capture and processing system 100, an image capture device 105A, an image processing device 105B, an image processor 150, a host processor 152, an ISP 154, an XR system 200, an XR application 215, a compositor 225, an imaging system 300, a GPU 325, a system for capturing and processing images according to a timeline 500, an imaging system 800, an imaging system 880, a CNN 1100, a Cifar-10NN 1200, a remote computing system, a camera, any combination thereof, or any computing device or computing system that constitutes any component thereof, the components of the system communicating with each other using a connection 1005. The connection 1005 may be a physical connection using a bus, such as in a chipset architecture, or a direct connection to a processor 1010. The connection 1005 may also be a virtual connection, a networked connection, or a logical connection.

[0145]

[0156] In some embodiments, computing system 1000 is a distributed system in which the functions described in this disclosure may be distributed across a data center, multiple data centers, a peer network, etc. In some embodiments, one or more of the system components described represent many such components, where the components each perform some or all of the functions described. In some embodiments, the components may be physical or virtual devices.

[0146]

[0157] The exemplary system 1000 includes at least one processing unit (CPU or processor) 1010 and connections 1005 coupling various system components to the processor 1010, including system memory 1015, such as read only memory (ROM) 1020 and random access memory (RAM) 1025. The computing system 1000 may include a cache 1012 of high-speed memory directly connected to the processor 1010, or in close proximity to it, or integrated as part of it.

[0147]

[0158] The processor 1010 may include any general purpose processor, hardware or software services such as services 1032, 1034, and 1036 stored in a storage device 1030 and configured to control the processor 1010, and special purpose processors where software instructions are embedded in the actual processor design. The processor 1010 may essentially be a completely self-contained computing system including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0148]

[0159] To enable user interaction, computing system 1000 includes input devices 1045, which may represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. Computing system 1000 may also include output devices 1035, which may be one or more of several output mechanisms. In some instances, a multimodal system may allow a user to provide multiple types of input and output for communicating with computing system 1000. Computing system 1000 may include a communication interface 1040, which may generally govern and manage user input and system output.The communications interface may be an audio jack / plug, a microphone jack / plug, a Universal Serial Bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet® port / plug, an optical fiber port / plug, a proprietary wired port / plug, BLUETOOTH® wireless signal transmission, BLUETOOTH Low Energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, Radio Frequency Identification (RFID) wireless signal transmission, Near Field Communication (NFC) wireless signal transmission, Dedicated Short Range Communications (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, Wireless Local Area Network (WLAN) wireless signal transmission, The communication interface 1040 may implement or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers including transceivers utilizing wireless LAN (WAN), visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communications wireless signal transfer, public switched telephone network (PSTN) signal transfer, integrated services digital network (ISDN) signal transfer, 3G / 4G / 5G / LTE cellular data network wireless signal transfer, ad-hoc network signal transfer, radio wave signal transfer, microwave signal transfer, infrared signal transfer, visible light signal transfer, ultraviolet light signal transfer, wireless signal transfer along the electromagnetic spectrum, or any combination thereof. The communication interface 1040 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine a position of the computing system 1000 based on reception of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the Chinese-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS.There is no restriction to operating on any particular hardware configuration, and therefore the basic features herein may be easily replaced with improved hardware or firmware configurations as they are developed.

[0149]

[0160] The storage device 1030 may be a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid state memory, a compact disk read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disk (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory Stick card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof, may be a hard disk or other type of computer readable medium capable of storing data that is accessible by a computer.

[0150]

[0161] The storage devices 1030 may include software services, servers, services, etc., which cause the system to perform functions when code defining such software is executed by the processor 1010. In some embodiments, a hardware service that performs a particular function may include software components stored in a computer-readable medium in relation to the necessary hardware components, such as the processor 1010, connections 1005, output devices 1035, etc., to perform that function.

[0151]

[0162] The term "computer-readable medium" as used herein includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or transporting instructions and / or data. Computer-readable media may include non-transitory media that can store data and do not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory, or memory devices. A computer-readable medium may have stored thereon code and / or machine-executable instructions, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted using any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0152]

[0163] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when stated, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0153]

[0164] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that the embodiments may be practiced without these specific details. For clarity of explanation, in some cases, the technology may be presented as including individual functional blocks, including devices, device components, steps or routines in a method implemented in software, or functional blocks comprising a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0154]

[0165] Particular embodiments may be described above as a process or method depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Moreover, the order of operations may be rearranged. A process is terminated when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or a main function.

[0155]

[0166] The processes and methods according to the examples described above may be implemented using computer-executable instructions stored or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general purpose computer, a special purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, and the like.

[0156]

[0167] A device implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, and the like. The functionality described herein may also be embodied in peripheral devices or add-in cards. Such functionality may also be implemented on a circuit board among different chips or different processes executing within a single device, as further examples.

[0157]

[0168] The instructions, media for carrying such instructions, computing resources for executing them, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0158]

[0169] In the above description, aspects of the present application have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concept may be embodied and employed in various ways, and the appended claims are to be construed to include such variations, except as limited by the prior art. Various features and aspects of the applications described above may be used individually or together. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be regarded as illustrative and not restrictive. For purposes of explanation, the methods have been described in a particular order. It should be appreciated that in alternative embodiments, the methods may be performed in an order different from that described.

[0159]

[0170] Those skilled in the art will appreciate that the less than ("<") and greater than (">") symbols or terms used herein may be interchanged with the less than ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of the present specification.

[0160]

[0171] When a component is described as being "configured to" perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform that operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform that operation, or any combination thereof.

[0161]

[0172] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component, and / or any component that is in communication, either directly or indirectly, with another component (e.g., connected to the other component via a wired or wireless connection and / or other suitable communications interface).

[0162]

[0173] Claim language or other language reciting "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, a claim language reciting "at least one of A and B" means A, B, or A and B. In another example, a claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, a claim language reciting "at least one of A and B" can mean A, B, or A and B, and can further include items not listed in the set of A and B.

[0163]

[0174] The various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether to implement such functionality as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0164]

[0175] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. When implemented in software, the techniques may be realized by a computer-readable data storage medium comprising program code including instructions that, when executed, at least in part, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a FLASH memory, a magnetic or optical data storage medium, or the like. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or radio wave, that carries or conveys program code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer.

[0165]

[0176] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated software or hardware modules configured for encoding and decoding, or incorporated into a combined video encoder / decoder (CODEC).

[0166]

[0177] Exemplary aspects of the present disclosure include the following.

[0167]

[0178] Aspect 1: An apparatus for image processing comprising: a memory; and one or more processors coupled to the memory, the one or more processors configured to: store existing data marked with metadata in a predetermined area of ​​the image frame buffer in the image frame buffer; receive new image frame data corresponding to a new image frame received by the image sensor; overwrite at least a portion of the existing data in the image frame buffer with the new image frame data; determine that the new image frame data includes at least a predetermined amount of the new image frame based on a reading of the predetermined area of ​​the image frame buffer indicating that metadata will no longer be stored in the predetermined area; and output at least a first portion of the new image frame data corresponding to the predetermined amount of the new image frame.

[0168]

[0179] Aspect 2: The device of aspect 1, wherein to output the first portion of the new image frame data, the one or more processors are configured to send the first portion of the new image frame data to an extended reality (XR) application.

[0169]

[0180] Aspect 3: The device of aspect 1 or 2, wherein to output the first portion of the new image frame data, the one or more processors are configured to process the first portion of the new image frame data using an extended reality (XR) application.

[0170]

[0181] Aspect 4: The apparatus of any one of aspects 1 to 3, wherein to output the first portion of the new image frame data, the one or more processors are configured to composite the first portion of the new image frame data with virtual content.

[0171]

[0182] Aspect 5: The apparatus of any one of aspects 1 to 4, wherein to output the first portion of the new image frame data, the one or more processors are configured to warp at least a portion of the first portion of the new image frame data.

[0172]

[0183] Aspect 6: The apparatus of any one of aspects 1 to 5, wherein to output the first portion of the new image frame data, the one or more processors are configured to distort at least a portion of the first portion of the new image frame data.

[0173]

[0184] Aspect 7: The apparatus of any one of aspects 1 to 6, wherein to output the first portion of the new image frame data, the one or more processors are configured to perform distortion compensation on at least a portion of the first portion of the new image frame data.

[0174]

[0185] Aspect 8: The device of any one of aspects 1 to 7, wherein to output the first portion of the new image frame data, the one or more processors store the first portion of the new image frame data in a display buffer.

[0175]

[0186] Example 9: The apparatus of any one of examples 1 to 8, further comprising a display buffer.

[0176]

[0187] Aspect 10: The apparatus of any one of aspects 1 to 9, wherein to output the first portion of the new image frame data, the one or more processors display the first portion of the new image frame data using a display.

[0177]

[0188] Example 11: The device of any one of examples 1 to 10, further comprising a display.

[0178]

[0189] Aspect 12: The apparatus of any one of aspects 1 to 11, wherein to output the first portion of the new image frame data, the one or more processors send the first portion of the new image frame data to the device.

[0179]

[0190] Aspect 13: The apparatus of any one of aspects 1 to 12, wherein to determine that the new image frame data includes at least a predetermined amount of new image frames, the one or more processors are configured to read a predetermined region of the image frame buffer and, based on reading the predetermined region of the image frame buffer, identify that metadata in the predetermined region of the image frame buffer has been overwritten.

[0180]

[0191] Aspect 14: The apparatus of any one of aspects 1 to 13, wherein the metadata includes a color pattern.

[0181]

[0192] Aspect 15: The apparatus of any one of aspects 1 to 14, wherein the metadata includes a frame identifier associated with the new image frame.

[0182]

[0193] Aspect 16: The apparatus of any one of aspects 1 to 15, wherein the metadata includes an image frame buffer identifier associated with the image frame buffer.

[0183]

[0194] Aspect 17: The apparatus of any one of aspects 1 to 16, wherein the one or more processors are configured to modify existing data in a predetermined area of ​​the image frame buffer to mark the existing data with metadata.

[0184]

[0195] Aspect 18: The apparatus of any one of aspects 1 to 17, wherein the apparatus is one of a mobile device, a wireless communication device, a head-mounted display, and a camera.

[0185]

[0196] Example 19: An apparatus described in any one of examples 1 to 18, further comprising an image sensor.

[0186]

[0197] Example 20: The apparatus of any one of examples 1 to 19, further comprising an image frame buffer.

[0187]

[0198] Aspect 21: The apparatus of any one of aspects 1 to 20, wherein the existing data is previous image frame data from a previous image frame, the previous image frame being captured prior to capture of the new image frame.

[0188]

[0199] Aspect 22: The apparatus of any one of aspects 1 to 21, wherein the metadata includes a watermark.

[0189]

[0200] Aspect 23: An apparatus described in any one of aspects 1 to 22, wherein the predetermined amount of new image frames comprises a predetermined percentage of new image frames.

[0190]

[0201] Aspect 24: The apparatus of any one of aspects 1 to 23, wherein the predetermined amount of new image frames comprises a predetermined percentage of new image frames.

[0191]

[0202] Aspect 25: The apparatus of any one of aspects 1 to 24, wherein the predetermined amount of new image frames includes a predetermined number of rows of the new image frames.

[0192]

[0203] Aspect 26: An apparatus described in any one of aspects 1 to 25, wherein the predetermined amount of new image frames comprises a predetermined number of columns of new image frames.

[0193]

[0204] Aspect 27: The apparatus of any one of aspects 1 to 26, wherein the predetermined amount of the new image frame includes a predetermined number of pixels of the new image frame.

[0194]

[0205] Aspect 28: The apparatus of any one of aspects 1 to 27, wherein to output at least a first portion of the new image frame data, the one or more processors are configured to output the new image frame.

[0195]

[0206] Aspect 29: The apparatus of any one of aspects 1 to 28, wherein the existing data is marked with the second metadata in a second predetermined area of ​​the image frame buffer.

[0196]

[0207] Aspect 30: An apparatus described in any one of aspects 1 to 29, wherein the one or more processors are configured to determine that the new image frame data includes at least a second predetermined amount of the new image frame based on a second read of the second predetermined area of ​​the image frame buffer, indicating that the second metadata will no longer be stored in the second predetermined area, and to output at least a second portion of the new image frame data corresponding to the second predetermined amount of the new image frame.

[0197]

[0208] Aspect 31: A method of image processing comprising: storing existing data marked with a watermark in a predetermined area of ​​an image frame buffer in an image frame buffer; receiving new image frame data corresponding to a new image frame; the new image frame data is received by an image sensor, overwriting at least a portion of the existing data in the image frame buffer with the new image frame data; determining that the new image frame data includes at least a predetermined amount of new image frames based on a reading of the predetermined area of ​​the image frame buffer indicating that metadata is no longer stored in the predetermined area; and outputting at least a first portion of the new image frame data corresponding to the predetermined amount of new image frames.

[0198]

[0209] Embodiment 32: The method of embodiment 31, further comprising any one of embodiments 2 to 30.

[0199]

[0210] Aspect 33: A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: store existing data marked with metadata in a predetermined area of ​​the image frame buffer in the image frame buffer; receive new image frame data corresponding to a new image frame; overwrite at least a portion of the existing data in the image frame buffer with the new image frame data, where the new image frame data is received by the image sensor; determine that the new image frame data includes at least a predetermined amount of new image frames based on a reading of the predetermined area of ​​the image frame buffer indicating that metadata is no longer stored in the predetermined area; and output at least a first portion of the new image frame data corresponding to the predetermined amount of new image frames.

[0200]

[0211] Example 34: The non-transitory computer-readable medium of example 33, further comprising any one of examples 2 to 30.

[0201]

[0212] Aspect 35: An apparatus for image processing comprising: means for storing existing data marked with a watermark in a predetermined area of ​​an image frame buffer in an image frame buffer; means for receiving new image frame data corresponding to a new image frame; means for overwriting at least a portion of the existing data in the image frame buffer with the new image frame data, the new image frame data being received by an image sensor; means for determining that the new image frame data includes at least a predetermined amount of the new image frame based on a reading of the predetermined area of ​​the image frame buffer indicating that metadata is no longer stored in the predetermined area; and means for outputting at least a first portion of the new image frame data corresponding to the predetermined amount of the new image frame.

[0202]

[0213] Embodiment 36: The apparatus of embodiment 35, further comprising any one of embodiments 2 to 30.

[0203]

[0214] Aspect 37: An apparatus for image processing comprising a memory and one or more processors coupled to the memory, the one or more processors configured to: store data marked with metadata in a predetermined area of ​​the image frame buffer in the image frame buffer; receive image frame data corresponding to the image frames from an image sensor; overwrite at least a portion of the stored data in the image frame buffer with the image frame data; determine that the image frame data includes at least a predetermined amount of the image frames based on a reading of the predetermined area of ​​the image frame buffer indicating that the predetermined area stores additional data different from the metadata; and output at least a first portion of the image frame data corresponding to the predetermined amount of the image frames.

[0204]

[0215] Aspect 38: The apparatus of aspect 37, wherein the additional data is at least a portion of the image frame data.

[0205]

[0216] Aspect 39: The apparatus of aspect 37 or 38, wherein the one or more processors are configured to process the first portion of the image frame data to output the first portion of the image frame data.

[0206]

[0217] Aspect 40: The apparatus of aspect 39, wherein to process the first portion of the image frame data, the one or more processors are configured to modify at least a portion of the first portion of the image frame data using at least one of distortion, distortion compensation, and warping.

[0207]

[0218] Aspect 41: The apparatus of aspect 39 or 40, wherein to process the first portion of the image frame data, the one or more processors are configured to composite the first portion of the image frame data with virtual content.

[0208]

[0219] Aspect 42: The apparatus of aspect 41, wherein the one or more processors are further configured to identify a pose of the image sensor and render virtual content based on the pose of the image sensor.

[0209]

[0220] Aspect 43: An apparatus described in any one of aspects 37 to 42, wherein to output the first portion of the image frame data, the one or more processors are configured to display the first portion of the image frame data using a display.

[0210]

[0221] Example 44: The device of example 43, further comprising a display.

[0211]

[0222] Aspect 45: The apparatus of any one of aspects 37 to 44, wherein to output the first portion of the image frame data, the one or more processors are configured to store the first portion of the image frame data in a display buffer.

[0212]

[0223] Example 46: The device of example 45, further comprising a display buffer.

[0213]

[0224] Aspect 47: The apparatus of any one of aspects 37 to 46, wherein to output the first portion of the image frame data, the one or more processors are configured to send the first portion of the image frame data to a receiving device using a communications transceiver.

[0214]

[0225] Aspect 48: The apparatus of aspect 47, further comprising a communications transceiver.

[0215]

[0226] Aspect 49: An apparatus described in any one of aspects 37 to 48, wherein to output the first portion of the image frame data, the one or more processors are configured to determine one or more image statistics based on at least the first portion of the image frame data, and to determine image capture settings based on the one or more image statistics.

[0216]

[0227] Aspect 50: The device described in aspect 49, wherein the one or more processors are further configured to: send image capture settings to image capture hardware, the image capture hardware including an image sensor; receive secondary image frame data corresponding to the second image frame from the image sensor; and the image sensor captures the second image frame based on the image capture settings.

[0217]

[0228] Embodiment 51: An apparatus described in embodiment 49 or 50, wherein the image capture setting is an exposure setting.

[0218]

[0229] Aspect 52: An apparatus described in any one of aspects 37 to 51, wherein the one or more processors are further configured to mark stored data with metadata in a predetermined area of ​​the image frame buffer.

[0219]

[0230] Aspect 53: An apparatus described in any one of aspects 37 to 52, wherein the metadata includes a color pattern.

[0220]

[0231] Aspect 54: An apparatus described in any one of aspects 37 to 53, wherein the metadata includes a frame identifier associated with the image frame.

[0221]

[0232] Aspect 55: An apparatus described in any one of aspects 37 to 54, wherein to determine that the image frame data includes at least a predetermined amount of the image frame, the one or more processors are configured to read a predetermined region of the image frame buffer and, based on the reading of the predetermined region of the image frame buffer, determine that the reading indicates that the predetermined region stores additional data other than the metadata.

[0222]

[0233] Aspect 56: An apparatus described in any one of aspects 37 to 55, wherein the stored data includes previous image frame data from a previous image frame, the previous image frame being captured by the image sensor prior to capture of the image frame.

[0223]

[0234] Aspect 57: The apparatus of any one of aspects 37 to 56, wherein to output at least a first portion of the image frame data, the one or more processors are configured to output the image frame.

[0224]

[0235] Aspect 58: The apparatus of any one of aspects 37 to 57, wherein the stored data is marked with second metadata in a second predetermined area of ​​the image frame buffer.

[0225]

[0236] Aspect 59: The apparatus described in aspect 58, wherein the one or more processors are configured to determine that the image frame data includes at least a second predetermined amount of the image frame based on a second reading of a second predetermined area of ​​the image frame buffer indicating that the second predetermined area stores secondary additional data different from the second metadata, and to output at least a second portion of the image frame data corresponding to the second predetermined amount of the image frame.

[0226]

[0237] Aspect 60: An apparatus described in any one of aspects 37 to 59, wherein the one or more processors are configured to receive a request for a view of the scene and, in response to receiving the request, automatically expose the image sensor to light from the scene, and the image frame data received from the image sensor is based on the exposure of the image sensor to light from the scene.

[0227]

[0238] Embodiment 61: An apparatus described in any one of embodiments 37 to 60, further comprising an image sensor.

[0228]

[0239] Aspect 62: The apparatus of any one of aspects 37 to 61, further comprising an image frame buffer.

[0229]

[0240] Embodiment 63: An apparatus described in any one of embodiments 37 to 62, which is a head-mounted display.

[0230]

[0241] Aspect 64: A method of image processing comprising: storing data marked with metadata in an image frame buffer in a predetermined area of ​​the image frame buffer; receiving image frame data corresponding to an image frame from an image sensor; overwriting at least a portion of the stored data in the image frame buffer with the image frame data; determining that the image frame data includes at least a predetermined amount of the image frame based on a reading of a predetermined area of ​​the image frame buffer indicating that the predetermined area stores additional data different from the metadata; and outputting at least a first portion of the image frame data corresponding to the predetermined amount of the image frame.

[0231]

[0242] Aspect 65: The method described in aspect 64, wherein the additional data is at least a portion of the image frame data.

[0232]

[0243] Aspect 67: The method of aspect 64 or 65, wherein outputting the first portion of the image frame data includes processing the first portion of the image frame data.

[0233]

[0244] Aspect 68: The method of aspect 67, wherein processing the first portion of the image frame data includes modifying at least a portion of the first portion of the image frame data using at least one of distortion, distortion compensation, and warping.

[0234]

[0245] Aspect 69: The method of aspect 67 or 68, wherein processing the first portion of the image frame data includes combining the first portion of the image frame data with virtual content.

[0235]

[0246] Aspect 70: The method of aspect 69, further comprising identifying a posture of the image sensor and rendering virtual content based on the posture of the image sensor.

[0236]

[0247] Aspect 71: The method of any one of aspects 64 to 70, wherein outputting the first portion of the image frame data includes displaying the first portion of the image frame data using a display.

[0237]

[0248] Aspect 72: The method of aspect 71, performed by a system including a display.

[0238]

[0249] Aspect 73: A method according to any one of aspects 64 to 72, wherein outputting the first portion of the image frame data includes storing the first portion of the image frame data in a display buffer.

[0239]

[0250] Aspect 74: The method of aspect 73, performed by a system including a display buffer.

[0240]

[0251] Aspect 75: The method of any one of aspects 64 to 74, wherein outputting the first portion of the image frame data includes sending the first portion of the image frame data to a receiving device using a communications transceiver.

[0241]

[0252] Aspect 76: The method of aspect 75, performed by a system including a communications transceiver.

[0242]

[0253] Aspect 77: A method described in any one of aspects 64 to 76, wherein outputting a first portion of the image frame data includes determining one or more image statistics based on at least the first portion of the image frame data, and determining image capture settings based on the one or more image statistics.

[0243]

[0254] Aspect 78: The method of aspect 77, further comprising sending image capture settings to image capture hardware, where the image capture hardware includes an image sensor, receiving secondary image frame data corresponding to the second image frame from the image sensor, where the image sensor captures the second image frame based on the image capture settings.

[0244]

[0255] Aspect 79: The method described in aspect 77 or 78, wherein the image capture setting is an exposure setting.

[0245]

[0256] Aspect 80: The method of any one of aspects 64 to 79, further comprising marking stored data with metadata in a predetermined area of ​​the image frame buffer.

[0246]

[0257] Aspect 81: A method described in any one of aspects 64 to 80, wherein the metadata includes a color pattern.

[0247]

[0258] Aspect 82: A method according to any one of aspects 64 to 81, wherein the metadata includes a frame identifier associated with the image frame.

[0248]

[0259] Aspect 83: A method according to any one of aspects 64 to 82, wherein determining that the image frame data includes at least a predetermined amount of the image frame includes reading a predetermined area of ​​the image frame buffer, and determining, based on the reading of the predetermined area of ​​the image frame buffer, that the reading indicates that the predetermined area stores additional data different from the metadata.

[0249]

[0260] Aspect 84: A method according to any one of aspects 64 to 83, wherein the stored data includes previous image frame data from a previous image frame, the previous image frame being captured by the image sensor prior to capture of the image frame.

[0250]

[0261] Aspect 85: The method of any one of aspects 64 to 84, wherein outputting at least a first portion of the image frame data includes outputting the image frame.

[0251]

[0262] Aspect 86: A method according to any one of aspects 64 to 85, wherein the stored data is marked with second metadata in a second predetermined area of ​​the image frame buffer.

[0252]

[0263] Aspect 87: The method described in aspect 86, further comprising determining that the image frame data includes at least a second predetermined amount of the image frame based on a second reading of a second predetermined area of ​​the image frame buffer indicating that the second predetermined area stores secondary additional data different from the second metadata, and outputting at least a second portion of the image frame data corresponding to the second predetermined amount of the image frame.

[0253]

[0264] Aspect 88: A method as described in any one of aspects 64 to 87, further comprising receiving a request for a view of the scene and, in response to receiving the request, automatically exposing the image sensor to light from the scene, wherein the image frame data received from the image sensor is based on the exposure of the image sensor to light from the scene.

[0254]

[0265] Embodiment 89: A method according to any one of embodiments 64 to 88, performed by a system including an image sensor.

[0255]

[0266] Aspect 90: A method according to any one of aspects 64 to 89, performed by a system including an image frame buffer.

[0256]

[0267] Aspect 91: A method described in any one of aspects 64 to 90, performed by a head-mounted display.

[0257]

[0268] Aspect 92: A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: store data marked with metadata in a predetermined area of ​​the image frame buffer in an image frame buffer; receive image frame data from an image sensor corresponding to an image frame; overwrite at least a portion of the stored data in the image frame buffer with the image frame data; determine that the image frame data includes at least a predetermined amount of the image frame based on a reading of a predetermined area of ​​the image frame buffer indicating that the predetermined area stores additional data different from the metadata; and output at least a first portion of the image frame data corresponding to the predetermined amount of the image frame.

[0258]

[0269] Embodiment 93: The non-transitory computer-readable medium of embodiment 92, further comprising any one of embodiments 38 to 63 and / or any one of embodiments 65 to 91.

[0259]

[0270] Aspect 94: An apparatus for image processing comprising: means for storing data marked with metadata in a predetermined area of ​​an image frame buffer in an image frame buffer; means for receiving image frame data corresponding to an image frame from an image sensor; means for overwriting at least a portion of the stored data in the image frame buffer with the image frame data; means for determining that the image frame data includes at least a predetermined amount of the image frame based on a reading of a predetermined area of ​​the image frame buffer indicating that the predetermined area stores additional data different from the metadata; and means for outputting at least a first portion of the image frame data corresponding to the predetermined amount of the image frame.

[0260]

[0271] Embodiment 95: The non-transitory computer-readable medium of embodiment 94, further comprising any one of embodiments 38 to 63 and / or any one of embodiments 65 to 91.

[0261]

[0272] Aspect 96: An apparatus for image processing comprising: a memory; and one or more processors coupled to the memory, the one or more processors configured to: store data marked with metadata in a predetermined area of ​​an image frame buffer in the image frame buffer; receive a request for a view of the scene; automatically expose an image sensor to light from the scene in response to receiving the request; receive image frame data from the image sensor corresponding to an image frame; overwrite at least a portion of the stored data in the image frame buffer with the image frame data, where the image frame data received from the image sensor is based on the exposure of the image sensor to light from the scene; determine that the image frame data includes at least a predetermined amount of the image frame based on a reading of the predetermined area indicating that the predetermined area stores additional data different from the metadata; and output at least a first portion of the image frame data corresponding to the predetermined amount of the image frames.

[0262]

[0273] Embodiment 97: The apparatus described in embodiment 96, further comprising any one of embodiments 38 to 63.

[0263]

[0274] Aspect 98: A method of image processing comprising: storing data marked with metadata in an image frame buffer in a predetermined area of ​​the image frame buffer; receiving a request for a view of the scene; automatically exposing an image sensor to light from the scene in response to receiving the request; receiving image frame data from the image sensor corresponding to an image frame; the image frame data received from the image sensor is based on the exposure of the image sensor to light from the scene; overwriting at least a portion of the stored data in the image frame buffer with the image frame data; determining that the image frame data includes at least a predetermined amount of the image frame based on reading a predetermined area of ​​the image frame buffer indicating that the predetermined area stores additional data different from the metadata; and outputting at least a first portion of the image frame data corresponding to the predetermined amount of the image frame.

[0264]

[0275] Embodiment 99: The method according to embodiment 98, further comprising any one of embodiments 65 to 91.

[0265]

[0276] Aspect 100: A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: store data marked with metadata in a predetermined region of the image frame buffer; receive a request for a view of the scene; automatically expose an image sensor to light from the scene in response to receiving the request; receive image frame data from the image sensor corresponding to an image frame; overwrite at least a portion of the stored data in the image frame buffer with the image frame data, where the image frame data received from the image sensor is based on the exposure of the image sensor to light from the scene; determine that the image frame data includes at least a predetermined amount of the image frame based on a reading of a predetermined region of the image frame buffer indicating that the predetermined region stores additional data different from the metadata; and output at least a first portion of the image frame data corresponding to the predetermined amount of the image frame.

[0266]

[0277] Embodiment 101: The non-transitory computer-readable medium of embodiment 100, further comprising any one of embodiments 38 to 63 and / or any one of embodiments 65 to 91.

[0267]

[0278] Aspect 102: An apparatus for image processing comprising: means for storing data marked with metadata in an image frame buffer in a predetermined area of ​​the image frame buffer; means for receiving a request for a view of the scene; means for automatically exposing the image sensor to light from the scene in response to receiving the request; means for receiving image frame data from the image sensor corresponding to an image frame; means for overwriting at least a portion of the stored data in the image frame buffer with the image frame data, the image frame data received from the image sensor being based on the exposure of the image sensor to light from the scene; means for determining that the image frame data includes at least a predetermined amount of the image frame based on reading a predetermined area of ​​the image frame buffer indicating that the predetermined area stores additional data different from the metadata; and means for outputting at least a first portion of the image frame data corresponding to the predetermined amount of the image frame.

[0268]

[0279] Embodiment 103: The apparatus described in embodiment 102, further comprising any one of embodiments 38 to 63 and / or any one of embodiments 65 to 91.

Claims

1. 1. An apparatus for image processing, comprising: Memory, one or more processors coupled to the memory; wherein the one or more processors: storing in the image frame buffer data marked with metadata in a predetermined region of the image frame buffer, wherein the metadata includes a frame identifier associated with the image frame; receiving image frame data from an image sensor corresponding to an image frame; overwriting at least a portion of the stored data in the image frame buffer with the image frame data; determining that the image frame data comprises at least a predetermined amount of the image frame based on a reading of the predetermined area of ​​the image frame buffer indicating that the predetermined area stores additional data distinct from the metadata, wherein the additional data is at least a portion of the image frame data; outputting at least a first portion of the image frame data corresponding to the predetermined amount of the image frame; An apparatus configured to:

2. The apparatus of claim 1 , wherein the one or more processors are configured to process the first portion of the image frame data to output the first portion of the image frame data.

3. 3. The apparatus of claim 2, wherein to process the first portion of the image frame data, the one or more processors are configured to modify at least a portion of the first portion of the image frame data using at least one of distortion, distortion compensation, and warping.

4. To process the first portion of the image frame data, the one or more processors are configured to combine the first portion of the image frame data with virtual content; The one or more processors: identifying a posture of the image sensor; Rendering the virtual content based on a pose of the image sensor; and The apparatus of claim 2 , further configured to:

5. 2. The apparatus of claim 1, further comprising a display, wherein to output the first portion of the image frame data, the one or more processors are configured to display the first portion of the image frame data using the display.

6. 2. The apparatus of claim 1, further comprising a display buffer, wherein to output the first portion of the image frame data, the one or more processors are configured to store the first portion of the image frame data in the display buffer.

7. To output the first portion of the image frame data, the one or more processors: determining one or more image statistics based on at least the first portion of the image frame data; determining image capture settings based on the one or more image statistics; configured to: The one or more processors: sending the image capture settings to image capture hardware, wherein the image capture hardware includes the image sensor; receiving secondary image frame data from the image sensor corresponding to a second image frame, wherein the image sensor captures the second image frame based on the image capture settings; and further configured to: The device of claim 1 , wherein the image capture setting is an exposure setting.

8. The one or more processors: The apparatus of claim 1 , further configured to mark the stored data in the predetermined area of ​​the image frame buffer with the metadata.

9. To determine that the image frame data comprises at least the predetermined amount of the image frame, the one or more processors: reading the predetermined area of ​​the image frame buffer; determining, based on the reading of the predetermined region of the image frame buffer, that the reading indicates that the predetermined region stores the additional data distinct from the metadata; The apparatus of claim 1 configured to:

10. The apparatus of claim 1 , wherein the stored data includes previous image frame data from a previous image frame, the previous image frame being captured by the image sensor prior to capture of the image frame.

11. The apparatus of claim 1 , wherein to output at least the first portion of the image frame data, the one or more processors are configured to output the image frame.

12. the stored data is marked with second metadata in a second predetermined area of ​​the image frame buffer; The one or more processors: determining that the image frame data comprises at least a second predetermined amount of the image frame based on a second read of the second predetermined area of ​​the image frame buffer indicating that the second predetermined area stores secondary additional data distinct from the second metadata; outputting at least a second portion of the image frame data corresponding to the second predetermined amount of the image frame; The apparatus of claim 1 configured to:

13. 1. A method of image processing, comprising the steps of: storing in the image frame buffer data marked with metadata in a predetermined region of the image frame buffer, wherein the metadata includes a frame identifier associated with the image frame; receiving image frame data from an image sensor corresponding to an image frame; overwriting at least a portion of the stored data in the image frame buffer with the image frame data; determining that the image frame data comprises at least a predetermined amount of the image frame based on a reading of the predetermined area of ​​the image frame buffer indicating that the predetermined area stores additional data distinct from the metadata, wherein the additional data is at least a portion of the image frame data; outputting at least a first portion of the image frame data corresponding to the predetermined amount of the image frame; A method comprising:

14. 14. The method of claim 13, further comprising marking the stored data in the predetermined area of ​​the image frame buffer with the metadata.

15. Receiving a request for a view of a scene; automatically exposing the image sensor to light from the scene in response to receiving the request, wherein the image frame data received from the image sensor is based on exposure of the image sensor to the light from the scene. The method of claim 13 further comprising:

Images