Dynamic capture time

Abstract

Techniques for processing image data are described herein. For example, a process may include obtaining a respective exposure time for each exposure in a set of exposures of an output image, where each respective exposure in the set of exposures contributes to the output image; determining a respective start time for each exposure in the set of exposures based on each respective exposure time of the set of exposures and an end time of an exposure period; determining an original start time based on each respective start time for each exposure in the set of exposures; adjusting the original start time based on an amount of contribution of each respective exposure in the set of exposures to the output image to obtain an adjusted start time; and determining a timestamp of the output image based on the adjusted start time and the end time of the exposure period.

Description

Technical Field

[0001] The present disclosure generally relates to image processing. For example, aspects of the present disclosure relate to systems and techniques for determining a dynamic capture time of captured images, such as high dynamic range (HDR) images and / or other images. Background Art

[0002] A camera is a device that uses an image sensor to receive light and capture image frames, such as still images or video frames. A camera may include one or more processors, such as an image signal processor (ISP), that can process the one or more image frames captured by the image sensor. For example, the raw image frames captured by the camera sensor may be processed by the ISP to generate the final image. A camera can be configured with various image capture and image processing settings to alter the appearance of the image. Some camera settings, such as ISO, exposure time (also known as duration), aperture size, f / stop, shutter speed, focus, and gain, are determined and applied before or during image capture. Additionally, some camera settings can be configured for post-processing of the image, such as changes to contrast, brightness, saturation, sharpness, levels, curves, or color.

[0003] In some cases, images generated by a camera can be used by a computer vision system that can detect objects or otherwise analyze the scene imaged by the camera. As an example, a computer vision system can detect and track objects in an environment by detecting the object in an image taken at one time and in another image taken at a later time. Based on the changes in the position of the object in the image, the computer vision system may be able to determine information about the object, such as the speed of other objects, direction of travel, etc. Therefore, being able to determine the precise time when an image was taken can be useful for a computer vision system. Summary of the Invention

[0004] The following presents a simplified summary of one or more aspects disclosed herein. Therefore, the following summary should neither be considered an exhaustive overview of all contemplated aspects nor be considered to identify key or critical elements related to all contemplated aspects or to delineate the scope associated with any particular aspect. Accordingly, the following summary presents certain concepts related to one or more aspects of the mechanisms disclosed herein in a simplified form prior to the detailed description presented below.

[0005] In one illustrative example, an apparatus for processing image data is provided. The apparatus includes at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to: obtain a corresponding exposure time for each exposure in an exposure set for an output image, wherein each corresponding exposure in the exposure set contributes to the output image; determine a corresponding start time for each exposure in the exposure set based on each corresponding exposure time in the exposure set and an end time of an exposure period; determine an original start time based on each corresponding start time of each exposure in the exposure set; adjust the original start time based on an amount by which each corresponding exposure in the exposure set contributes to the output image to obtain an adjusted start time; and determine a timestamp for the output image based on the adjusted start time and end time of the exposure period.

[0006] As another example, a method for processing image data is provided. The method includes obtaining a corresponding exposure time for each exposure in an exposure set for an output image, wherein each corresponding exposure in the exposure set contributes to the output image; determining a corresponding start time for each exposure in the exposure set based on each corresponding exposure time in the exposure set and an end time of an exposure period; determining an original start time based on each corresponding start time of each exposure in the exposure set; adjusting the original start time based on an amount by which each corresponding exposure in the exposure set contributes to the output image to obtain an adjusted start time; and determining a timestamp for the output image based on the adjusted start time and the end time of the exposure period.

[0007] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided. The instructions, when executed by at least one processor, cause the at least one processor to: obtain a corresponding exposure time for each exposure in an exposure set for outputting an image, wherein each corresponding exposure in the exposure set contributes to the output image; determine a corresponding start time for each exposure in the exposure set based on each corresponding exposure time in the exposure set and an end time of an exposure period; determine an original start time based on each corresponding start time of each exposure in the exposure set; adjust the original start time based on an amount by which each corresponding exposure in the exposure set contributes to the output image to obtain an adjusted start time; and determine a timestamp for the output image based on the adjusted start time and end time of the exposure period.

[0008] As another example, an apparatus for processing image data is provided. The apparatus includes: means for obtaining a corresponding exposure time for each exposure in an exposure set for outputting an image, wherein each corresponding exposure in the exposure set contributes to the output image; means for determining a corresponding start time for each exposure in the exposure set based on each corresponding exposure time in the exposure set and an end time of an exposure period; means for determining an original start time based on each corresponding start time of each exposure in the exposure set; means for adjusting the original start time based on an amount by which each corresponding exposure in the exposure set contributes to the output image to obtain an adjusted start time; and means for determining a timestamp for the output image based on the adjusted start time and end time of the exposure period.

[0009] In some aspects, one or more of the devices described herein are, may be part of, or may include a mobile device, smart or connected device, camera system, and / or extended reality (XR) device (e.g., a virtual reality (VR) device, augmented reality (AR) device, or mixed reality (MR) device). In some examples, the device may include, or be part of, a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wearable device, a personal computer, a laptop computer, a tablet computer, a server computer, a robotic device or system, a vehicle, or other device. In some aspects, the device includes an image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, the device includes one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device includes one or more speakers, one or more light-emitting devices, and / or one or more microphones. In some aspects, the devices described above may include one or more sensors. In some cases, the one or more sensors may be used to determine the device's location, the device's status (e.g., tracking status, operational status, temperature, humidity level, and / or other status), and / or for other purposes.

[0010] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all of the drawings, and each claim.

[0011] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Illustrative examples of the present application are described in detail below with reference to the following drawings:

[0013] Figure 1 is a block diagram illustrating an example architecture of an image processing system according to some examples of the present disclosure;

[0014] Figure 2 illustrates multiple images with different exposures for creating a fused high dynamic range image according to some examples of the present disclosure;

[0015] Figure 3 Illustrate multiple images with different exposures used to create a fused HDR image;

[0016] Figure 4 is a conceptual diagram illustrating differences between images captured by an image capture system with different exposure times according to some aspects of the present disclosure;

[0017] Figure 5A and Figure 5B is a timeline illustrating obtaining a timestamp for an HDR image according to aspects of the present disclosure;

[0018] Figure 6 is a timeline illustrating exposure of a row of pixels during an exposure period according to aspects of the present disclosure;

[0019] Figure 7 is a flow diagram of a process for processing image data according to aspects of the present disclosure;

[0020] Figure 8 An example computing device architecture is illustrated that illustrates an example computing device that may implement the various techniques described herein. DETAILED DESCRIPTION

[0021] Certain aspects of the present disclosure are provided below. Some of these aspects can be applied independently, and some of them can be applied in combination, which will be apparent to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it is apparent that various aspects can be practiced without these specific details. Each drawing and description is not intended to be restrictive.

[0022] The following description provides only exemplary aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of exemplary aspects will provide a description that can be used to implement the exemplary aspects to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0023] Electronic devices (e.g., mobile phones, wearable devices (e.g., smartwatches, smart glasses, etc.), tablet computers, extended reality (XR) devices (e.g., virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, etc.), connected devices, laptops, vehicles, etc.) are increasingly equipped with camera hardware to capture image frames, such as still images and / or video frames, for consumption. For example, an electronic device may include a camera to allow the electronic device to capture video or images of scenes, people, objects, etc. A camera is a device that uses an image sensor to receive light and capture image frames (e.g., still images or video frames). In some examples, a camera may include one or more processors, such as an image signal processor (ISP), that may process one or more image frames captured by the image sensor. For example, raw image frames captured by a camera sensor may be processed by the camera's image signal processor (ISP) to generate a final image. In some cases, the electronic device implementing the camera may further process the captured images or videos for certain effects (e.g., compression, image enhancement, image restoration, scaling, frame rate conversion, etc.) and / or certain applications (such as computer vision, extended reality (e.g., augmented reality, virtual reality, etc.), object detection, image recognition (e.g., face recognition, object recognition, scene recognition, etc.), feature extraction, authentication and automation, etc.).

[0024] Additionally, cameras can be configured with various image capture and image processing settings to alter the appearance of an image. Some camera settings, such as ISO, exposure time (also known as exposure duration), aperture size, f / stop, shutter speed, focus, and gain, can be determined and applied before or during image capture. Some camera settings can be configured for post-processing of an image, such as changes to contrast, brightness, saturation, sharpness, levels, curves, or color. In some examples, a camera can be configured with certain settings to adjust the exposure of an image captured by the camera.

[0025] In photography, the exposure of an image captured by a camera refers to the amount of light per unit area that reaches the photographic film or, in modern cameras, the electronic image sensor. Exposure is based on certain camera settings, such as shutter speed, exposure time, and / or lens aperture, as well as the brightness of the scene being photographed. Many cameras are equipped with an automatic exposure or "autoexposure" mode, in which the camera's exposure settings (e.g., shutter speed, exposure time, lens aperture, etc.) are automatically adjusted to match the brightness of the scene or subject being photographed as closely as possible. In some cases, an automatic exposure control (AEC) engine may perform AEC to determine the exposure settings of the image sensor (e.g., exposure time, gain, aperture, etc.).

[0026] In photography and videography, a technology known as high dynamic range (HDR) allows the dynamic range of image frames captured by a camera to be increased beyond the camera's inherent capabilities. In this context, dynamic range refers to the range of brightness between the brightest and darkest areas of a scene or image frame. For example, high dynamic range means that there is a lot of variation in light levels within a scene or image frame. HDR may involve capturing multiple image frames of a scene with different exposures and combining these captured image frames into a single image frame. The combination of image frames with different exposures can produce an image with a higher dynamic range than the dynamic range of each individual image frame captured and combined to form the HDR image frame. For example, an electronic device can create a high dynamic range scene by fusing two or more exposed frames into a single frame. HDR is a feature frequently used by electronic devices such as smartphones and mobile devices for various purposes. For example, in some cases, smartphones can use HDR to achieve better image quality or image quality similar to that achieved by digital single-lens reflex (DSLR) cameras.

[0027] In some examples, an electronic device can create an HDR image using multiple image frames with different exposures. For example, an electronic device can create an HDR image using a short exposure (SE) image, a medium exposure (ME) image, and a long exposure (LE) image. In another example, an electronic device can create an HDR image using SE and LE images. In some cases, the electronic device can write the different image frames from the camera front end to a memory device, such as double data rate (DDR) synchronous dynamic random access memory (SDRAM) or any other memory device. A processing engine can then retrieve these image frames to fuse them into a single image.

[0028] Generally speaking, overexposed pixels in the long-exposure image and underexposed pixels in the short-exposure image do not contribute to the final fused image (e.g., HDR image) produced by the HDR algorithm. However, the overexposed pixels in the long-exposure image and underexposed pixels in the short-exposure image are still written to the memory device by the camera front end and read back from the memory device by the processing engine. Therefore, the operations of reading and writing the overexposed pixels in the long-exposure image and the underexposed pixels in the short-exposure image contribute to power and bandwidth consumption in the electronic device, even though such pixels do not contribute to the final fused image.

[0029] In some cases, determining the precise time an image was captured can be used for a variety of applications, such as image analysis, computer vision, object tracking, object detection, and object speed detection. For example, being able to accurately locate an object or determine its speed can be useful for applications such as automatic emergency braking or lane change assistance in vehicles. The accuracy of an object's location and speed in world coordinates can depend on the accuracy of when the object was present in the environment. Accurate timestamps for when an object was observed in an image can help improve this accuracy.

[0030] This document describes systems, apparatus, methods (also referred to as processes), and computer-readable media (collectively referred to herein as "systems and techniques") for determining the dynamic capture time of images, such as high dynamic range (HDR) images and / or other images. In some cases, determining an accurate timestamp for an HDR (e.g., stacked) image can be problematic because HDR images may be composed of multiple images, each with a different exposure time. While the timestamp for an image can be determined based on the median of the time periods during which different exposures were captured, the subject may have moved between the different exposures. A more accurate timestamp can take into account the contribution of a particular exposure to the output image (e.g., final image, final merged image). In some cases, the start time of capture can be adjusted to provide a more accurate timestamp. For example, the start time of capture can be adjusted based on the contribution of a particular exposure to the output image. In some cases, the contribution of a particular exposure to the output image can be determined based on any indication of how many of the exposures in the set of exposures were used in the final merged image. Examples of indications include the total weight of the exposures, the exposure ratio, the exposure gain weight, and so on.

[0031] Various aspects of the application will be described with respect to the accompanying drawings.

[0032] Figure 1 is a block diagram illustrating an example architecture of an image processing system 100. Image processing system 100 includes various components for capturing and processing images, such as an image of scene 110. Image processing system 100 can capture image frames (e.g., still images or video frames). In some cases, lens 115 and image sensor 130 can be associated with an optical axis. In one illustrative example, both the photosensitive area (e.g., photodiode) of image sensor 130 and lens 115 can be centered on the optical axis.

[0033] In some examples, lens 115 of image processing system 100 faces scene 110 and receives light from scene 110. Lens 115 bends the incoming light from the scene toward image sensor 130. The light received by lens 115 then passes through an aperture of image processing system 100. In some cases, the aperture (e.g., the aperture size) is controlled by one or more control mechanisms 120. In other cases, the aperture may have a fixed size.

[0034] The one or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or information from the image processor 150. In some cases, the one or more control mechanisms 120 may include multiple mechanisms and components. For example, the control mechanism 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. The one or more control mechanisms 120 may also include, in addition to Figure 1 Additional control mechanisms beyond those illustrated in For example, in some cases, one or more control mechanisms 120 may include controls for controlling analog gain, flash, HDR, depth of field, and / or other image capture characteristics.

[0035] Focus control mechanism 125B of control mechanism 120 may obtain a focus setting. In some examples, focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, focus control mechanism 125B may adjust the position of lens 115 relative to the position of image sensor 130. For example, based on the focus setting, focus control mechanism 125B may actuate a motor or servo system (or other lens mechanism) to move lens 115 closer to or further away from image sensor 130, thereby adjusting the focus. In some cases, image processing system 100 may include additional lenses. For example, image processing system 100 may include one or more microlenses on each photodiode of image sensor 130. Each of these microlenses may bend light received from lens 115 toward its corresponding photodiode before the light reaches the photodiode.

[0036] In some examples, the focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The focus setting may be determined using control mechanism 120, image sensor 130, and / or image processor 150. The focus setting may be referred to as an image capture setting and / or an image processing setting. In some cases, lens 115 may be fixed relative to the image sensor and focus control mechanism 125B.

[0037] Exposure control mechanism 125A of control mechanism 120 may obtain an exposure setting. In some cases, exposure control mechanism 125A stores the exposure setting in a memory register. Based on the exposure setting, exposure control mechanism 125A may control the size of the aperture (e.g., aperture size or f / stop), the duration that the aperture is open (e.g., exposure time or shutter speed), the duration that the sensor collects light (e.g., exposure time or electronic shutter speed), the sensitivity of image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by image sensor 130, or any combination thereof. Exposure settings may be referred to as image capture settings and / or image processing settings.

[0038] Zoom control mechanism 125C of control mechanism 120 may obtain a zoom setting. In some examples, zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, zoom control mechanism 125C may control the focal length of an assembly of lens elements (lens assembly) including lens 115 and one or more additional lenses. For example, zoom control mechanism 125C may control the focal length of the lens assembly by actuating one or more motors or servos (or other lens mechanisms) to move one or more lenses relative to one another. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly may include a focusing lens (in some cases, this focusing lens may be lens 115) that first receives light from scene 110, where the light then passes through an afocal zoom system between the focusing lens (e.g., lens 115) and image sensor 130 before reaching image sensor 130. In some cases, an afocal zoom system may include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference of each other), with a negative (e.g., diverging, concave) lens between them. In some cases, zoom control mechanism 125C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses. In some cases, zoom control mechanism 125C may control zoom by capturing images from an image sensor (e.g., including image sensor 130) from a plurality of image sensors at a zoom corresponding to a zoom setting. For example, image processing system 100 may include a wide-angle image sensor with a relatively low zoom and a telephoto image sensor with a greater zoom. In some cases, based on the selected zoom setting, zoom control mechanism 125C may capture images from the corresponding sensor.

[0039] Image sensor 130 includes one or more arrays of photodiodes or other light-sensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in the image generated by image sensor 130. In some cases, different photodiodes may be covered by different filters. In some cases, different photodiodes may be covered in color filters and may therefore measure light that matches the color of the filter covering the photodiode. Various color filter arrays may be used, such as, for example and without limitation, a Bayer color filter array, a quad color filter array (QCFA), and / or any other color filter array.

[0040] In some cases, image sensor 130 may alternatively or additionally include opaque and / or reflective masks that block light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles. In some cases, opaque and / or reflective masks may be used for phase detection autofocus (PDAF). In some cases, opaque and / or reflective masks may be used to block portions of the electromagnetic spectrum from reaching the image sensor's photodiodes (e.g., IR cutoff filters, UV cutoff filters, bandpass filters, low-pass filters, high-pass filters, etc.). Image sensor 130 may also include analog gain amplifiers for amplifying analog signals output by the photodiodes and / or analog-to-digital converters (ADCs) for converting the analog signals output by the photodiodes (and / or the analog signals amplified by the analog gain amplifiers) into digital signals. In some cases, certain components or functionality discussed with respect to one or more of control mechanisms 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal oxide semiconductor (CMOS), an N-type metal oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0041] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including the ISP 154), one or more host processors (including the host processor 152), and / or related Figure 8Host processor 152 may be a digital signal processor (DSP) and / or other types of processors discussed in connection with computing device architecture 800. In some implementations, image processor 150 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes host processor 152 and ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth™, Global Positioning System (GPS), etc.), any combination thereof, and / or other components. The I / O ports 156 may include any suitable input / output ports or interfaces according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial General Purpose Input / Output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface), an Advanced High-Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In one illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port, and the ISP 154 may communicate with the image sensor 130 using a MIPI port.

[0042] The image processor 150 may perform a number of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The image processor 150 may store the image frames and / or processed images in a random access memory (RAM) 140, a read-only memory (ROM) 145, a cache, a memory unit, another storage device, or some combination thereof.

[0043] Various input / output (I / O) devices 160 may be connected to the image processor 150. I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a touchpad, a touch-sensitive surface, a printer, any other output device, any other input device, or any combination thereof. In some cases, subtitles may be entered into the image processing device 105B via a physical keyboard or keypad of the I / O device 160, or via a virtual keyboard or keypad of the touch screen of the I / O device 160. The I / O devices 160 may include one or more ports, jacks, or other connectors that enable wired connections between the image processing system 100 and one or more peripheral devices, via which the image processing system 100 can receive data from and / or send data to the one or more peripheral devices. The I / O devices 160 may also include one or more wireless transceivers that enable wireless connections between the image processing system 100 and one or more peripheral devices, via which the image processing system 100 can receive data from and / or send data to the one or more peripheral devices. Peripheral devices may include any of the types of I / O devices 160 discussed previously, and may themselves be considered I / O devices 160 once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.

[0044] In some cases, the image processing system 100 can be a single device. In some cases, the image processing system 100 can be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 105A and the image processing device 105B can be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some implementations, the image capture device 105A and the image processing device 105B can be disconnected from each other.

[0045] like Figure 1 As shown, the vertical dotted line will Figure 1Image processing system 100 is divided into two parts, representing image capture device 105A and image processing device 105B. Image capture device 105A includes lens 115, control mechanism 120, and image sensor 130. Image processing device 105B includes image processor 150 (including ISP 154 and host processor 152), RAM 140, ROM 145, and I / O device 160. In some cases, some components illustrated in image capture device 105A (such as ISP 154 and / or host processor 152) may be included in image capture device 105A. In some examples, image processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, or some combination thereof.

[0046] The image processing system 100 can be part of or implemented by a single computing device or multiple computing devices. In some examples, the image processing system 100 can be part of an electronic device (or multiple electronic devices), such as a camera system (e.g., a digital camera, an IP camera, a video camera, a security camera, etc.), a phone system (e.g., a smartphone, a cellular phone, a conferencing system, etc.), a laptop or notebook computer, a tablet computer, a set-top box, a smart TV, a display device, a game console, an XR device (e.g., an HMD, smart glasses, etc.), an IoT (Internet of Things) device, a smart wearable device, a video streaming device, an Internet Protocol (IP) camera, a vehicle, or any other suitable electronic device.

[0047] The image capture device 105A and the image processing device 105B can be part of the same electronic device or different electronic devices. In some implementations, the image capture device 105A and the image processing device 105B can be different devices. For example, the image capture device 105A can include a camera device, and the image processing device 105B can include a computing device, such as a mobile device, a desktop computer, a smartphone, a smart TV, a game console, a vehicle, or other computing device.

[0048] Although the image processing system 100 is shown as including certain components, one of ordinary skill will understand that the image processing system 100 may include more than Figure 1Components of the image processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, components of the image processing system 100 may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the image processing system 100.

[0049] Host processor 152 can configure image sensor 130 with new parameter settings (e.g., via an external control interface such as I2C, I3C, SPI, GPIO, and / or other interfaces). In one illustrative example, host processor 152 can update the exposure settings used by image sensor 130 based on internal processing results of an exposure control algorithm from past image frames. Host processor 152 can also dynamically configure parameter settings of ISP 154's internal pipeline or modules to match the settings of one or more input image frames from image sensor 130, so that the image data is properly processed by ISP 154. Processing (or pipeline) blocks or modules of ISP 154 may include modules for lens (or sensor) noise correction, demosaicing, color conversion, correction or enhancement / suppression of image properties, denoising filters, sharpening filters, and the like. Each module of ISP 154 may include a large number of tunable parameter settings. Furthermore, because different modules may affect similar aspects of an image, modules may be interdependent. For example, denoising and texture correction or enhancement may both affect high-frequency aspects of an image. As a result, a large number of parameters are used by the ISP to generate the final image from the captured raw image.

[0050] In some examples, Figure 8 The computing device architecture 800 shown in FIG. 8 and described further below may include the image processing system 100 , the image capture device 105A, the image processing device 105B, or a combination thereof.

[0051] Figure 2 is a block diagram illustrating an example of an image capture and processing system 200 that includes an image processor 250 (including a host processor 252 and an ISP 254 ) in communication with an image sensor 230 . Figure 2The illustrated configuration illustrates a conventional synchronization technique used in camera systems. Generally, the host processor 252 attempts to synchronize the image sensor 230 and the ISP 254 using fixed time periods by communicating with each separately. For example, in conventional camera systems, the host processor 252 communicates with the image sensor 230 (e.g., via an I2C port) and programs image sensor 230 parameters for a first fixed time period (e.g., two frames before the image frame is processed by the ISP 254). The host processor 252 then communicates with the ISP 254 (e.g., via an internal AHB bus or other interface) and provides the ISP 254 with parameter settings for capturing an image during a second fixed time period (e.g., one frame before the image frame is processed by the ISP 254). The image sensor 230 may transmit the image frames to the ISP 254, such as via a MIPI CSI-2 PHY port or interface or other suitable interface.

[0052] In some examples, the image processing system 100 may use multiple image frames with different exposures to create an HDR image. For example, the image processing system 100 may use a short exposure (SE) image, a medium exposure (ME) image, and a long exposure (LE) image to create an HDR image. In another example, the image processing system 100 may use SE images and LE images to create an HDR image. In some cases, the image processing system 100 may write different image frames from one or more camera front-end engines to a memory device, such as a DDR memory device or any other memory device. A post-processing engine may then retrieve these image frames and fuse (e.g., merge, combine) them into a single image. As previously explained, the different write and read operations used to create an HDR image can result in significant power and bandwidth consumption.

[0053] In some cases, ISP 254 may receive multiple exposures (e.g., SE images, ME images, and / or LE images) from image sensor 230 and process the multiple exposures into a single HDR image. In other cases, image sensor 230 may include an integrated processor and memory (not shown) for storing and processing the multiple exposures (e.g., SE images, ME images, and / or LE images) into a single HDR image. This HDR image may be passed to ISP 254 for additional processing. In some cases, the HDR processing used to generate the HDR image may also be performed by a system processor (such as host processor 252).

[0054] As previously explained, when creating an HDR image, overexposed pixels of the long exposure image and underexposed pixels of the short exposure image generally do not contribute to the final HDR image produced by the image processing system 100. For example, Figure 3Multiple images with different exposures are illustrated for creating a fused HDR image (eg, HDR image 330). Specifically, Figure 3 3. A short exposure image 300, a medium exposure image 310, a long exposure image 320, and an HDR image 330 generated by combining or fusing the short exposure image 300, the medium exposure image 310, and the long exposure image 320 are shown. In some cases, the HDR processing used to generate the HDR image 330 may be performed by a processor such as a processor. Figure 2 The image processing routines may be executed by a host processor 252 of the image sensor 230, or by a processor that is a component of the ISP 254, or by a processor integrated with the image sensor 230.

[0055] In some cases, different exposures (e.g., short-exposure image 300, medium-exposure image 310, and long-exposure image 320) may be fused via an HDR algorithm. In some cases, the HDR algorithm may determine how much influence a particular exposure of a scene has on the final pixel value of the fused image. For example, for a particular pixel in the fused image, the corresponding short-exposure image 300 version of the pixel may be underexposed and may not include useful information for the fused image, while the corresponding medium-exposure image 310 and long-exposure image 320 versions of the pixel may include more useful information for the fused image. Therefore, it may be useful to reduce the influence of the corresponding short-exposure image 300 version of the pixel or to adjust the influence of the corresponding medium-exposure image 310 and long-exposure image 320 versions of the pixel. In some cases, this adjustment can be performed by adjusting weights applied to pixels of different exposures to help control the degree to which the different exposures influence the final fused version of the pixel. This weighting can be performed on a pixel-by-pixel basis for the entire exposed image or a portion of an exposed image. As an example of how weights may be determined, in some cases, weights may be determined based on the luminance value of a pixel or group of pixels and applied based on a lookup table. In some cases, the total weight of the exposures (eg, short exposure image 300 , medium exposure image 310 , and long exposure image 320 ) may be determined by summing the weight values applied to the pixels of the exposures and dividing by the total possible weight.

[0056] Figure 44 is a conceptual diagram 400 illustrating the difference between images captured by an image capture system with different exposure times, according to some aspects of the present disclosure. Diagram 400 illustrates a short-exposure image 402, a medium-exposure image 404, and a long-exposure image 406 captured by an image processing system. In a conventional HDR image synthesis process, short-exposure image 402, medium-exposure image 404, and long-exposure image 406 are provided from an image sensor to an ISP (e.g., ISP 254) for processing. The ISP may store short-exposure image 402, medium-exposure image 404, and long-exposure image 406 in memory and then process short-exposure image 402, medium-exposure image 404, and long-exposure image 406 into a single HDR image.

[0057] As described above, each of short-exposure image 402, medium-exposure image 404, and long-exposure image 406 is read out from the sensor array by the image sensor and has a different exposure time, but may be directed at the same scene. In some cases, the different exposures may or may not overlap. Diagram 400 illustrates multiplying short-exposure image 402 by a short-to-long exposure ratio to normalize the intensity of short-exposure image 402 relative to long-exposure image 406. For example, if the short exposure time is 10 milliseconds (ms) and the long exposure time is 40 ms, the brightness of the pixels in short-exposure image 402 is multiplied by the exposure ratio of 4.0. The normalized short-exposure image 402 and long-exposure image 406 are then compared in pixel differentiator 410. In one illustrative example, pixel differentiator 410 calculates the difference between each pixel to generate a differential pixel bitmap, and graph 420 illustrates the difference in pixel values between the normalized short-exposure image 402 and long-exposure image 406. In some cases, the exposure ratio can be used to represent the effect of exposure on the final HDR image.

[0058] In some cases, the exposure time and exposure gain weight can be dynamically determined by the exposure control algorithm. In some cases, the exposure time and exposure gain weight can be selected based on the lighting conditions of the environment. As discussed above, there can be multiple exposures per pixel with multiple exposure times, which can be used to generate a single HDR image. The exposure gain weight of the exposure can also be used to represent the impact of the exposure on the final HDR image.

[0059] As discussed above, determining the precise time at which an image was captured can be used in a variety of applications, such as image analysis, computer vision, object tracking, object detection, and the like. It is worth noting that for the purposes of this disclosure, time can refer to a relative time with respect to the camera device. That is, relative time can be independent of world time. For example, when attempting to determine the precise speed of an object, the precise timestamp at the time the image was captured can be used. However, while those timestamps can be accurate relative to each other and possibly other sensors and / or systems included and / or coupled to the camera device, those timestamps may not be accurate relative to the time in the environment (e.g., as measured by a separate clock).

[0060] In some cases, determining an accurate timestamp for an HDR (e.g., stacked) image can be problematic because an HDR image may be composed of multiple images, each with a different exposure time. In some cases, the timestamp for an HDR image may be based on a median of the time periods over which the different exposures were taken (e.g., the exposure period of the HDR image).

[0061] Figure 5A and Figure 5B is a timeline illustrating obtaining timestamps of images captured with a rolling shutter for use in generating an HDR image according to aspects of the present disclosure. Figure 5A A timeline 500 is illustrated for a camera device with a global shutter. In timeline 500, the capture of different portions of image data for an HDR image by an imaging system is illustrated relative to a horizontal time axis 520, along which time flows from left to right. In some cases, the HDR image may be based on three images with different exposure periods: a long exposure image 502, a medium exposure image 504, and a short exposure image 506. A frame exposure period 508 may be the length of time used to obtain the exposed image. The timestamp for the HDR image may be a median value 510 of the frame exposure period 508, which may be found by subtracting the start time 512 of the frame exposure period 508 from the end time 514 of the frame exposure period and dividing by two.

[0062] Figure 5BA timeline 550 illustrating a camera device with a rolling shutter is shown. In timeline 550, the imaging system's capture of different portions of image data for an HDR image is illustrated relative to a horizontal time axis 580, along which time flows from left to right, and relative to a vertical row axis 590, which indicates different rows of the image sensor. In some cases, a rolling shutter camera can capture each frame of a still image or video by rapidly scanning across a scene from one side of the image sensor to the other. In some cases, the rolling shutter may scan horizontally across the scene. In other cases, the rolling shutter may scan vertically across the scene. In some examples, using a rolling shutter, different portions of an image of a scene are recorded at different moments in time. Images captured using a rolling shutter may depict different portions of a scene because they appear at slightly different times (e.g., instants) in the scene. A rolling shutter camera may capture pixel data row by row (e.g., row by row) from one side of the image sensor to the other.

[0063] In timeline 550, a first row can be exposed and captured during a first exposure period 552, wherein the photodetectors of the first row can be exposed to light from a scene. A second exposure period 556 for a second row of photodetectors can begin 558 before the end 554 of the first exposure period 552. The second row can be exposed and captured during the second exposure period 556. This pattern repeats for the rows of the image sensor until the last row is exposed and captured at the end 562 of the last exposure period 564. In some cases, the time period of each exposure period (e.g., first exposure period 552, second exposure period 556, ..., and last exposure period 564) is the same, and multiple exposures can be generated for a row during an exposure period.

[0064] Figure 66 is a timeline 600 illustrating the exposure of a row of pixels during an exposure period in accordance with aspects of the present disclosure. In timeline 600, the capture of data for pixels of an HDR image by an imaging system is illustrated relative to a horizontal time axis 620, along which time flows from left to right. In some cases, pixel data for three exposures may be captured during an exposure period. In some cases, pixel data may be captured as sub-pixels by multiple photodetectors and combined, such as with a QCFA. In some cases, these multiple sub-pixels may be used to concurrently capture different exposures of the HDR image. For example, pixel data for a long exposure may be captured by a first sub-pixel of a pixel starting at the beginning 602 of a long exposure capture period 604, and a medium exposure may be captured by a second sub-pixel of the pixel starting at the beginning 606 of a medium exposure capture period 608. At the end 610 of the long exposure capture period 604, pixel data for a short exposure may be captured by the first sub-pixel of the pixel. Capturing pixel data for the short and medium exposures may conclude at the end 612 of the exposure period. In some cases, the end 612 of the exposure period may be predetermined. In some cases, the exposure times of the long exposure capture period 604 , the medium exposure capture period 608 , and the short exposure capture period 614 may be determined by an exposure control algorithm, and the start of the exposure may be set based on a predetermined end 612 of the exposure period.

[0065] Return to Figure 5B , the frame exposure period 566 can be the length of time from the start of the first exposure period 552 (e.g., the start time of capture 572) to the end 562 of the last exposure period. The timestamp of the HDR image can be the middle value 568 of the frame exposure period 566 and can be the same as above with respect to Figure 5A The intermediate value 568 is found in a substantially similar manner as discussed above. The transmission period 570 may be a period of time for transmitting and / or processing pixel data from a row.

[0066] However, depending on how the exposures are incorporated into the final HDR image, the timestamp may not be very accurate. For example, if an HDR image captured in an environment with very bright lighting includes a long exposure captured before a medium exposure, followed by a short exposure, the long exposure may contribute very little or nothing to the final HDR image. In this case, it may be more accurate to base the timestamp of the HDR image more on the points in time when the medium and short exposures were taken.

[0067] In some cases, dynamically adjusting the capture time of an image can result in a more accurate timestamp. For example, as indicated above, the timestamp of an HDR image can be based on the median value of a frame exposure period (such as frame exposure periods 508 and 566), where the median value of the frame exposure period is based on a start time and an end time. In some cases, the median value of the frame exposure period can be dynamically adjusted by adjusting the start time of the frame exposure period. In some cases, the start time of the frame exposure period can be adjusted based on the start time of the exposure of the merged image and the contribution of the exposure of the merged image to the final merged image. For example, the start time of the frame exposure period can be adjusted based on the contribution of a particular exposure to the final merged image. The contribution of a particular exposure to the final merged image can be determined by any indication of the extent to which a particular exposure from the set of exposures is used in the final merged image. Examples of indications can include a total weight of the exposures, an exposure ratio, an exposure gain weight, and the like.

[0068] As a more detailed example, because the capture time of a fused image can be adjusted based on exposure gain weights, the earliest individual exposure in the exposure set of the output image can initially be determined as the original start time of capture. For example, the exposure control algorithm can determine the duration of the individual exposures in the exposure set. In some cases, the exposure set may include three individual exposures: a short exposure, a medium exposure, and a long exposure. The exposure control algorithm can determine the duration of each of these individual exposures based on environmental conditions (such as lighting conditions, camera device speed, camera device acceleration, etc.). Based on the determined duration of the individual exposures and the end of the exposure period, an original start time can be determined for each individual exposure. By obtaining an exposure gain weight for each exposure and applying the exposure gain weight to the individual exposure time of each exposure, the original start time of capture can be adjusted based on the exposure weight gain. From the adjusted start times, the earliest adjusted start time can be identified as the adjusted start time of capture.

[0069] As a first example of applying exposure gain weights to exposure times that can be captured at least partially concurrently, if an exposure, such as a medium exposure, has an exposure time (e.g., exposure length) of 4 ms and the exposure has an exposure gain weight of 0.5, the exposure can have a weighted exposure time of 2 ms. An adjusted capture start time can be determined for each exposure based on the weighted exposure time. From the adjusted capture start times, the earliest adjusted capture start time can be identified as the adjusted capture start time. In some cases, such as where multiple exposures can be captured concurrently, the exposure associated with the original capture start time (e.g., the exposure with the earliest original capture start time) may differ from the exposure associated with the adjusted capture start time. A timestamp for the fused image can then be determined based on the adjusted capture start time and the end of the frame exposure period. In some cases, the timestamp for the fused image can be determined before, concurrently with, or after the exposure that captured the fused image.

[0070] As a second example of applying exposure gain weights to the exposure times of sequentially captured individual exposures, if an individual exposure (such as a medium exposure) has an exposure time (e.g., exposure length) of 4 ms and the individual exposure has an exposure gain weight of 0.5, the individual exposure may have a weighted exposure time of 2 ms. A weighted exposure time may be determined for each individual exposure, and the weighted exposure times may be summed. The original start time of the capture may be adjusted based on the difference between the sum of the original individual exposure times and the summed weighted exposure time. The timestamp of the fused image may then be determined based on the adjusted start time of the capture and the end of the frame exposure period.

[0071] Figure 7 is a flow diagram of a process 700 for dynamically adjusting image capture time according to aspects of the present disclosure. Process 700 may be performed by a computing device (or apparatus) or a component of a computing device (e.g., a chipset, a codec, etc.), such as a processor. Figure 1 image processor 150, Figure 1 Image sensor 130, Figure 2 ISP 254, Figure 2 Image sensor 230, Figure 2 The host processor 252 and / or Figure 8 The process 700 is executed by a processor 810. The computing device can be a mobile device (e.g., a mobile phone), a network-connected wearable device such as a watch, an extended reality (XR) device such as a virtual reality (VR) device or an augmented reality (AR) device, a vehicle or a component or system of a vehicle, or other type of computing device. The operations of process 700 can be implemented as software components that execute and run on one or more processors.

[0072] At block 702, a computing device (or a component thereof) may obtain a respective exposure time for each exposure in a set of exposures for an output image. In some cases, each respective exposure in the set of exposures contributes to the output image.

[0073] At block 704, the computing device (or a component thereof) may determine a respective start time for each exposure in the exposure set based on each respective exposure time of the exposure set and an end time of the exposure period. In some cases, the end time of the exposure period is predetermined.

[0074] At block 706 , the computing device (or a component thereof) may determine an original start time based on each respective start time of each exposure in the set of exposures.

[0075] At block 708, the computing device (or a component thereof) may adjust the original start time based on the contribution of each corresponding exposure in the exposure set to the output image to obtain an adjusted start time. The computing device (or a component thereof) may obtain a corresponding exposure gain weight for each exposure in the exposure set. The computing device (or a component thereof) may determine the contribution of each corresponding exposure in the exposure set to the output image based on the corresponding exposure gain weight for each exposure in the exposure set. The computing device (or a component thereof) may determine the contribution of each corresponding exposure in the exposure set to the output image by applying the corresponding exposure gain weight associated with each exposure to each corresponding start time of each exposure. The computing device (or a component thereof) may determine a corresponding adjusted start time for each exposure in the exposure set based on the original start time of each exposure in the exposure set and the corresponding exposure gain weight associated with each exposure in the exposure set. In some cases, to adjust the original start time, the computing device (or a component thereof) may select the earliest start time from the adjusted start times as the adjusted start time for each exposure in the exposure set. In some cases, the respective exposure time of each exposure in the exposure set and the respective exposure gain weight of each exposure in the exposure set are based on environmental conditions surrounding the device.

[0076] At block 710, the computing device (or a component thereof) may determine a timestamp for the output image based on the adjusted start and end times of the exposure period. To determine the timestamp for the output image, the computing device (or a component thereof) may determine a midpoint between the adjusted start and end times of the exposure period. Figure 8An example computing device architecture 800 illustrates an example computing device that can implement the various techniques described herein. In some examples, the computing device can include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device of a vehicle), or other device. For example, the computing device architecture 800 can include Figure 1 8. The components of computing device architecture 800 are shown in electrical communication with each other using a connection 805, such as a bus. The example computing device architecture 800 includes a processing unit (CPU or processor) 810 and computing device connections 805 that couple various computing device components, including computing device memory 815, such as read-only memory (ROM) 820 and random access memory (RAM) 825, to the processor 810.

[0077] The computing device architecture 800 may include a cache of high-speed memory directly connected to, in close proximity to, or integrated as part of the processor 810. The computing device architecture 800 may copy data from memory 815 and / or storage device 830 to cache 812 for rapid access by the processor 810. In this way, the cache can provide a performance boost by preventing processor 810 from experiencing delays while waiting for data. These and other modules may control or be configured to control the processor 810 to perform various actions. Other computing device memory 815 may also be available for use. Memory 815 may include a variety of different types of memory with varying performance characteristics. The processor 810 may include any general-purpose processor and hardware or software services configured to control the processor 810 (such as Service 1 832, Service 2 834, and Service 3 836 stored in storage device 830), as well as specialized processors where software instructions are incorporated into the processor design. The processor 810 may be a self-contained system containing multiple cores or processors, a bus, a memory controller, a cache, and the like. Multi-core processors may be symmetric or asymmetric.

[0078] To enable user interaction with the computing device architecture 800, the input device 845 may represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The output device 835 may also be one or more of a variety of output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, and the like. In some instances, a multimodal computing device may enable a user to provide multiple types of input to communicate with the computing device architecture 800. The communication interface 840 may generally control and manage user input and computing device output. There is no restriction on operating on any particular hardware arrangement, and thus the underlying features herein may be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0079] The storage device 830 is a non-volatile memory and can be a hard disk or other type of computer-readable medium that can store computer-accessible data, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a magnetic tape cartridge, random access memory (RAM) 825, read-only memory (ROM) 820, or hybrids thereof. The storage device 830 may include services 832, 834, and 836 for controlling the processor 810. Other hardware or software modules are contemplated. The storage device 830 may be connected to the computing device connection 805. In one aspect, a hardware module that performs a particular function may include a software component stored in a computer-readable medium connected to the necessary hardware components (such as the processor 810, the connection 805, the output device 835, etc.) to perform the function.

[0080] Aspects of the present disclosure are applicable to any suitable electronic device (such as a security system, smartphone, tablet, laptop, vehicle, drone, or other device) that includes or is coupled to one or more active depth sensing systems. Although described below with respect to a device having or coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to a particular device.

[0081] The term "device" is not limited to one or a specific number of physical objects (such as a smartphone, a controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that can implement at least some portions of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. Additionally, the term "system" is not limited to multiple components or specific implementations. For example, a system can be implemented on one or more printed circuit boards or other substrates and can have movable or static components. Although the following description and examples use the term "system" to describe various aspects of the present disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

[0082] Specific details are provided in the description above to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that embodiments can be put into practice without these specific details. For clarity of explanation, in some cases, the present technology can be presented as comprising separate functional blocks, including functional blocks comprising devices, device components, steps in the method embodied in software or a combination of hardware and software or routines. Additional components other than those components shown in the accompanying drawings and / or described herein can be used. For example, circuits, systems, networks, processes and other components can be shown as components in block diagram form to avoid these embodiments becoming difficult to understand in unnecessary details. In other cases, known circuits, processes, algorithms, structures and techniques can be shown in order to avoid making each embodiment difficult to understand without necessary details.

[0083] Individual embodiments may be described above as processes or methods depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or concurrently. Furthermore, the order of the operations may be rearranged. A process is terminated when its operations are completed, but a process may have additional steps not included in the accompanying figures. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, termination of the process may correspond to the function returning to the calling function or the main function.

[0084] The processes and methods according to the examples described above can be implemented using computer-executable instructions stored or otherwise obtained from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessed over a network. The computer-executable instructions may be, for example, binary files, intermediate format instructions (such as assembly language), firmware, source code, etc.

[0085] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media that can store data and does not include carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, memory or other electronic storage devices, magnetic or optical disks, USB devices provided with non-volatile memory, network storage devices, and / or any suitable combination thereof. A computer-readable medium may have stored thereon code and / or machine-executable instructions, which may represent a procedure, function, subroutine, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0086] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0087] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., a computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of further example, such functionality may also be implemented on circuit boards in different chips or different processes executed on a single device.

[0088] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0089] In the foregoing description, various aspects of the present application have been described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary embodiments of the present application have been described in detail herein, it is to be understood that the inventive concept can be embodied and adopted in various other ways, and the appended claims are intended to be interpreted as including such variations, unless limited by the prior art. The various features and aspects of the application described above can be used individually or in combination. In addition, without departing from the broader essence and scope of this specification, the embodiments can be used in any number of environments and applications beyond the environment and application described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For illustrative purposes, each method is described in a specific order. It should be understood that in an alternative embodiment, each method can be performed in a different order than described.

[0090] Those of ordinary skill in the art will appreciate that the less than ("<") and greater than (">") symbols or terms used herein can be replaced by less than or equal to (" ") and greater than or equal to (" )” symbol without departing from the scope of this description.

[0091] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0092] The phrase “coupled to” refers to any component being directly or indirectly physically connected to another component, and / or any component being in direct or indirect communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0093] Claim language or other language reciting "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A, B, and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0094] The various exemplary logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. Technicians can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0095] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium containing program code, including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0096] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0097] Illustrative aspects of the present disclosure include:

[0098] Aspect 1. A device for processing image data, the device comprising: at least one memory; and at least one processor, the at least one processor being coupled to the at least one memory and configured to: obtain a corresponding exposure time for each exposure in an exposure set for an output image, wherein each corresponding exposure in the exposure set contributes to the output image; determine a corresponding start time for each exposure in the exposure set based on each corresponding exposure time of the exposure set and an end time of an exposure cycle; determine an original start time based on each corresponding start time of each exposure in the exposure set; adjust the original start time based on the amount of contribution of each corresponding exposure in the exposure set to the output image to obtain an adjusted start time; and determine a timestamp for the output image based on the adjusted start time and the end time of the exposure cycle.

[0099] Aspect 2. The apparatus according to aspect 1, wherein the end time of the exposure period is predetermined.

[0100] Aspect 3. An apparatus according to any one of Aspects 1 to 2, wherein the at least one processor is further configured to: obtain a corresponding exposure gain weight for each exposure in the exposure set; and determine the contribution of each corresponding exposure in the exposure set to the output image based on the corresponding exposure gain weight for each exposure in the exposure set.

[0101] Aspect 4. An apparatus according to Aspect 3, wherein the at least one processor is configured to determine the contribution of each corresponding exposure in the exposure set to the output image by applying the corresponding exposure gain weight associated with each exposure to each corresponding start time of each exposure.

[0102] Aspect 5. An apparatus according to Aspect 3, wherein the at least one processor is further configured to: determine the corresponding adjusted start time of each exposure in the exposure set based on the original start time of each exposure in the exposure set and the corresponding exposure gain weight associated with each exposure in the exposure set.

[0103] Aspect 6. An apparatus according to aspect 5, wherein, in order to adjust the original start time, the at least one processor is further configured to select the earliest start time from the adjusted start times for each exposure in the exposure set as the adjusted start time.

[0104] Aspect 7. The apparatus of aspect 3, wherein the respective exposure time of each exposure in the exposure set and the respective exposure gain weight of each exposure in the exposure set are based on environmental conditions surrounding the apparatus.

[0105] Aspect 8. An apparatus according to any one of Aspects 1 to 7, wherein, in order to determine the timestamp of the output image, the at least one processor is configured to determine an intermediate value between the adjusted start time and the end time of the exposure period.

[0106] Aspect 9. A method for processing image data, the method comprising: obtaining a corresponding exposure time for each exposure in an exposure set for an output image, wherein each corresponding exposure in the exposure set contributes to the output image; determining a corresponding start time for each exposure in the exposure set based on each corresponding exposure time of the exposure set and an end time of an exposure cycle; determining an original start time based on each corresponding start time of each exposure in the exposure set; adjusting the original start time based on the amount of contribution of each corresponding exposure in the exposure set to the output image to obtain an adjusted start time; and determining a timestamp for the output image based on the adjusted start time and the end time of the exposure cycle.

[0107] Aspect 10. The method according to aspect 9, wherein the end time of the exposure period is predetermined.

[0108] Aspect 11. The method according to any one of Aspects 9 to 10, further comprising: obtaining a corresponding exposure gain weight for each exposure in the exposure set; and determining the contribution of each corresponding exposure in the exposure set to the output image based on the corresponding exposure gain weight for each exposure in the exposure set.

[0109] Aspect 12. The method according to aspect 11 further comprises determining the contribution of each respective exposure in the exposure set to the output image by applying the respective exposure gain weight associated with each exposure to each respective start time of each exposure.

[0110] Aspect 13. The method according to Aspect 11 further includes: determining the corresponding adjusted start time of each exposure in the exposure set based on the original start time of each exposure in the exposure set and the corresponding exposure gain weight associated with each exposure in the exposure set.

[0111] Aspect 14. The method of aspect 13, wherein adjusting the original start time comprises selecting an earliest start time from the adjusted start times as the adjusted start time for each exposure in the exposure set.

[0112] Aspect 15. The method of aspect 11, wherein the respective exposure time of each exposure in the exposure set and the respective exposure gain weight of each exposure in the exposure set are based on environmental conditions.

[0113] Clause 16. The method of any one of clauses 9 to 15, wherein determining the timestamp of the output image comprises determining a midpoint between the adjusted start time and the end time of the exposure period.

[0114] Aspect 17. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by at least one processor, causes the at least one processor to: obtain a corresponding exposure time for each exposure in an exposure set for an output image, wherein each corresponding exposure in the exposure set contributes to the output image; determine a corresponding start time for each exposure in the exposure set based on each corresponding exposure time of the exposure set and an end time of an exposure cycle; determine an original start time based on each corresponding start time of each exposure in the exposure set; adjust the original start time based on the amount by which each corresponding exposure in the exposure set contributes to the output image to obtain an adjusted start time; and determine a timestamp for the output image based on the adjusted start time and the end time of the exposure cycle.

[0115] Aspect 18. The non-transitory computer-readable medium of aspect 17, wherein the end time of the exposure period is predetermined.

[0116] Aspect 19. A non-transitory computer-readable medium according to any one of Aspects 17 to 18, wherein the instructions further cause the one or more processors to: obtain a corresponding exposure gain weight for each exposure in the exposure set; and determine the contribution of each corresponding exposure in the exposure set to the output image based on the corresponding exposure gain weight for each exposure in the exposure set.

[0117] Aspect 20. A non-transitory computer-readable medium according to Aspect 19, wherein the instructions further cause the one or more processors to determine the contribution of each corresponding exposure in the exposure set to the output image by applying the corresponding exposure gain weight associated with each exposure to each corresponding start time of each exposure.

[0118] Aspect 21. A non-transitory computer-readable medium according to Aspect 19, wherein the instructions further cause the one or more processors to: determine the corresponding adjusted start time of each exposure in the exposure set based on the original start time of each exposure in the exposure set and the corresponding exposure gain weight associated with each exposure in the exposure set.

[0119] Aspect 22. A non-transitory computer-readable medium according to Aspect 21, wherein, in order to adjust the original start time, the instructions cause the at least one processor to select the earliest start time from the adjusted start times for each exposure in the exposure set as the adjusted start time.

[0120] Aspect 23. The non-transitory computer-readable medium of aspect 19, wherein the respective exposure time of each exposure in the exposure set and the respective exposure gain weight of each exposure in the exposure set are based on environmental conditions.

[0121] Aspect 24. A non-transitory computer-readable medium according to any one of Aspects 17 to 23, wherein, to determine the timestamp of the output image, the instructions cause the at least one processor to determine an intermediate value between the adjusted start time and the end time of the exposure period.

[0122] Aspect 25. A device for processing image data, the device comprising: a component for obtaining a corresponding exposure time for each exposure in an exposure set for an output image, wherein each corresponding exposure in the exposure set contributes to the output image; a component for determining a corresponding start time for each exposure in the exposure set based on each corresponding exposure time of the exposure set and an end time of an exposure cycle; a component for determining an original start time based on each corresponding start time of each exposure in the exposure set; a component for adjusting the original start time to obtain an adjusted start time based on the amount of contribution of each corresponding exposure in the exposure set to the output image; and a component for determining a timestamp for the output image based on the adjusted start time and end time of the exposure cycle.

[0123] Aspect 26. The apparatus of aspect 25, wherein the end time of the exposure period is predetermined.

[0124] Aspect 27. An apparatus according to any one of Aspects 25 to 26, wherein the apparatus further comprises: a component for obtaining a corresponding exposure gain weight for each exposure in the exposure set; and a component for determining the contribution of each corresponding exposure in the exposure set to the output image based on the corresponding exposure gain weight for each exposure in the exposure set.

[0125] Aspect 28. An apparatus according to Aspect 27, further comprising a component for determining the contribution of each corresponding exposure in the exposure set to the output image by applying the corresponding exposure gain weight associated with each exposure to each corresponding start time of each exposure.

[0126] Aspect 29. An apparatus according to Aspect 27, further comprising: a component for determining the corresponding adjusted start time of each exposure in the exposure set based on the original start time of each exposure in the exposure set and the corresponding exposure gain weight associated with each exposure in the exposure set.

[0127] Aspect 30. The apparatus of aspect 29, wherein the means for adjusting the original start time comprises means for selecting an earliest start time from the adjusted start times as the adjusted start time for each exposure in the exposure set.

[0128] Aspect 31. The apparatus of aspect 27, wherein the respective exposure time of each exposure in the exposure set and the respective exposure gain weight of each exposure in the exposure set are based on environmental conditions surrounding the apparatus.

[0129] Aspect 32. An apparatus according to any one of aspects 25 to 31, wherein the means for determining the timestamp of the output image comprises means for determining an intermediate value between the adjusted start time and the end time of the exposure period.

Claims

1. A device for processing image data, the device comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: obtaining a respective exposure time for each exposure in a set of exposures for an output image, wherein each respective exposure in the set of exposures contributes to the output image; determining a corresponding start time for each exposure in the exposure set based on each corresponding exposure time of the exposure set and an end time of an exposure period; determining an original start time based on each corresponding start time of each exposure in the set of exposures; adjusting the original start time based on an amount that each corresponding exposure in the set of exposures contributes to the output image to obtain an adjusted start time; as well as A timestamp for the output image is determined based on the adjusted start time and the end time of the exposure period.

2. The apparatus of claim 1, wherein the end time of the exposure period is predetermined.

3. The apparatus of claim 1 , wherein the at least one processor is further configured to: Obtaining a corresponding exposure gain weight for each exposure in the exposure set; and The contribution amount of each respective exposure in the exposure set to the output image is determined based on the respective exposure gain weight of each exposure in the exposure set.

4. An apparatus according to claim 3, wherein the at least one processor is configured to determine the contribution of each respective exposure in the exposure set to the output image by applying the respective exposure gain weight associated with each exposure to each respective start time of each exposure.

5. The apparatus of claim 3 , wherein the at least one processor is further configured to: A respective adjusted start time for each exposure in the exposure set is determined based on the original start time of each exposure in the exposure set and the respective exposure gain weight associated with each exposure in the exposure set.

6. The apparatus of claim 5, wherein to adjust the original start time, the at least one processor is further configured to select an earliest start time from the adjusted start times for each exposure in the exposure set as the adjusted start time.

7. The device of claim 3, wherein the respective exposure time of each exposure in the exposure set and the respective exposure gain weight of each exposure in the exposure set are based on environmental conditions surrounding the device. 8 . The apparatus of claim 1 , wherein to determine the timestamp of the output image, the at least one processor is configured to determine a midpoint between the adjusted start time and the end time of the exposure period.

9. A method for processing image data, the method comprising: obtaining a respective exposure time for each exposure in a set of exposures for an output image, wherein each respective exposure in the set of exposures contributes to the output image; determining a corresponding start time for each exposure in the exposure set based on each corresponding exposure time of the exposure set and an end time of an exposure period; determining an original start time based on each corresponding start time of each exposure in the set of exposures; adjusting the original start time based on an amount that each corresponding exposure in the set of exposures contributes to the output image to obtain an adjusted start time; as well as A timestamp for the output image is determined based on the adjusted start time and the end time of the exposure period.

10. The method of claim 9, wherein the end time of the exposure period is predetermined.

11. The method according to claim 9, further comprising: Obtaining a corresponding exposure gain weight for each exposure in the exposure set; as well as The contribution amount of each respective exposure in the exposure set to the output image is determined based on the respective exposure gain weight of each exposure in the exposure set.

12. The method of claim 11, further comprising determining the contribution of each respective exposure in the set of exposures to the output image by applying the respective exposure gain weight associated with each exposure to each respective start time of each exposure.

13. The method according to claim 11, further comprising: A respective adjusted start time for each exposure in the exposure set is determined based on the original start time of each exposure in the exposure set and the respective exposure gain weight associated with each exposure in the exposure set. 14 . The method of claim 13 , wherein adjusting the original start time comprises selecting an earliest start time from the adjusted start times as the adjusted start time for each exposure in the exposure set.

15. The method of claim 11, wherein the respective exposure time of each exposure in the exposure set and the respective exposure gain weight of each exposure in the exposure set are based on environmental conditions.

16. The method of claim 9, wherein determining the timestamp of the output image comprises determining a midpoint between the adjusted start time and the end time of the exposure period.

17. A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to: obtaining a respective exposure time for each exposure in a set of exposures for an output image, wherein each respective exposure in the set of exposures contributes to the output image; determining a corresponding start time for each exposure in the exposure set based on each corresponding exposure time of the exposure set and an end time of an exposure period; determining an original start time based on each corresponding start time of each exposure in the set of exposures; adjusting the original start time based on an amount that each corresponding exposure in the set of exposures contributes to the output image to obtain an adjusted start time; as well as A timestamp for the output image is determined based on the adjusted start time and the end time of the exposure period.

18. The non-transitory computer readable medium of claim 17, wherein the end time of the exposure period is predetermined.

19. The non-transitory computer-readable medium of claim 17, wherein the instructions further cause the at least one processor to: Obtaining a corresponding exposure gain weight for each exposure in the exposure set; and The contribution amount of each respective exposure in the exposure set to the output image is determined based on the respective exposure gain weight of each exposure in the exposure set.

20. The non-transitory computer-readable medium of claim 19, wherein the instructions further cause the at least one processor to determine the contribution of each respective exposure in the exposure set to the output image by applying the respective exposure gain weight associated with each exposure to each respective start time of each exposure.

21. The non-transitory computer-readable medium of claim 19, wherein the instructions further cause the at least one processor to: A respective adjusted start time for each exposure in the exposure set is determined based on the original start time of each exposure in the exposure set and the respective exposure gain weight associated with each exposure in the exposure set.

22. The non-transitory computer-readable medium of claim 21, wherein to adjust the original start time, the instructions cause the at least one processor to select an earliest start time from the adjusted start times for each exposure in the exposure set as the adjusted start time.

23. The non-transitory computer-readable medium of claim 19, wherein the respective exposure time of each exposure in the exposure set and the respective exposure gain weight of each exposure in the exposure set are based on environmental conditions.

24. The non-transitory computer-readable medium of claim 17, wherein to determine the timestamp of the output image, the instructions cause the at least one processor to determine a midpoint between the adjusted start time and the end time of the exposure period.

25. An apparatus for processing image data, the apparatus comprising: means for obtaining a respective exposure time for each exposure in a set of exposures for an output image, wherein each respective exposure in the set of exposures contributes to the output image; means for determining a respective start time for each exposure in the exposure set based on each respective exposure time and an end time of an exposure period of the exposure set; means for determining an original start time based on each respective start time of each exposure in the set of exposures; means for adjusting the original start time based on an amount by which each respective exposure in the set of exposures contributes to the output image to obtain an adjusted start time; as well as Means for determining a timestamp for the output image based on the adjusted start time and the end time of the exposure period.

26. The apparatus of claim 25, wherein the end time of the exposure period is predetermined.

27. The apparatus according to claim 25, further comprising: means for obtaining a corresponding exposure gain weight for each exposure in the exposure set; as well as means for determining the contribution of each respective exposure in the set of exposures to the output image based on the respective exposure gain weight of each exposure in the set of exposures.

28. An apparatus according to claim 27, further comprising means for determining the contribution of each respective exposure in the exposure set to the output image by applying the respective exposure gain weight associated with each exposure to each respective start time of each exposure.

29. The apparatus according to claim 27, further comprising: Means for determining a respective adjusted start time for each exposure in the exposure set based on the original start time of each exposure in the exposure set and the respective exposure gain weight associated with each exposure in the exposure set.

30. The apparatus of claim 29, wherein means for adjusting the original start time comprises means for selecting an earliest start time from the adjusted start times as the adjusted start time for each exposure in the set of exposures.

Images