Image compression for generating combined image

Abstract

Systems, devices, processes, and computer-readable media for capturing images having subjects at different depths of field are disclosed. For example, a method of generating an image includes obtaining a first image captured using an image sensor at a first exposure. The method may also include obtaining a second image captured using the image sensor at a second exposure. The method may include compressing the second image based on a comparison of the second image and the first image; and storing the compressed second image in a memory. The method may also include obtaining a compressed second image from the memory; and decompressing the compressed second image based on a difference between the compressed second image and the first image. The method may also include generating a combined image at least in part by combining the first image and the second image.

Description

Technical Field

[0001] In general, the present application relates to processing image data. For example, the systems and techniques described herein relate to compressing images to reduce the bandwidth and power required to synthesize a combined image (eg, an HDR image) from multiple images. Background Art

[0002] A camera is a device that receives light and uses an image sensor to capture image frames (such as still images or video frames). A camera may include a processor (such as an image signal processor (ISP)) that can receive one or more image frames and process one or more image frames. For example, the raw image frames captured by the camera sensor can be processed by the ISP to generate the final image. The camera can be configured with various image capture and image processing settings to change the appearance of the image. Some camera settings are determined and applied before or during the capture of the photo, such as ISO, exposure time, aperture size, f / stop, shutter speed, focus, and gain. Other camera settings can configure post-processing of the photo, such as changes to contrast, brightness, saturation, sharpness, levels, curves, or color.

[0003] The camera can be configured with various image capture and image processing settings. Application of different settings may result in frames or images with different appearances. Some camera settings are determined and applied before or during the capture of a photo, such as ISO, exposure time (also known as exposure duration), aperture size, f / stop, shutter speed, focus, and gain. Other camera settings may configure post-processing of the photo, such as changes to contrast, brightness, saturation, sharpness, levels, curves, or color. Summary of the invention

[0004] In some examples, systems and techniques for compressing images to create high dynamic range (HDR) images are described. The systems and techniques can reduce the bandwidth and power required to synthesize HDR images from multiple images captured in sequence.

[0005] According to at least one example, a method for generating one or more images is provided, the method comprising: obtaining a first image captured using an image sensor, the first image being associated with a first exposure; obtaining a second image captured using the image sensor, the second image being associated with a second exposure; compressing the second image based on a comparison of the second image with the first image; storing the compressed second image in a memory; obtaining the compressed second image from the memory; decompressing the compressed second image based on a difference between the compressed second image and the first image; and generating a combined image at least in part by combining the first image and the second image.

[0006] In another example, an apparatus for generating one or more images is provided, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory. The at least one processor is configured to: obtain a first image captured using an image sensor, the first image associated with a first exposure; obtain a second image captured using the image sensor, the second image associated with a second exposure; compress the second image based on a comparison of the second image with the first image; store the compressed second image in a memory; obtain the compressed second image from the memory; decompress the compressed second image based on a difference between the compressed second image and the first image; and generate a combined image at least in part by combining the first image and the second image.

[0007] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which, when executed by one or more processors, causes the one or more processors to: obtain a first image captured using an image sensor, the first image being associated with a first exposure; obtain a second image captured using the image sensor, the second image being associated with a second exposure; compress the second image based on a comparison of the second image with the first image; store the compressed second image in a memory; obtain the compressed second image from the memory; decompress the compressed second image based on a difference between the compressed second image and the first image; and generate a combined image at least in part by combining the first image and the second image.

[0008] In another example, an apparatus for generating one or more images is provided. The apparatus includes: obtaining a first image captured using an image sensor, the first image being associated with a first exposure; means for obtaining a second image captured using the image sensor, the second image being associated with a second exposure; means for compressing the second image based on a comparison of the second image with the first image; means for storing the compressed second image in a memory; means for obtaining the compressed second image from the memory; means for decompressing the compressed second image based on a difference between the compressed second image and the first image; and means for generating a combined image at least in part by combining the first image and the second image.

[0009] According to at least one other example, a method for processing one or more images is provided, the method comprising: reading a first image from a pixel array of an image sensor; reading a second image from the pixel array; compressing the second image based on the first image; and sending the first image and the compressed second image to an image sensor processor (ISP).

[0010] In another example, a device for processing one or more images is provided, the device comprising: at least one memory; and at least one processor coupled to the at least one memory. The at least one processor is configured to: read a first image from a pixel array of an image sensor; read a second image from the pixel array; compress the second image based on the first image; and send the first image and the compressed second image to an image sensor processor (ISP).

[0011] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which, when executed by one or more processors, causes the one or more processors to: read a first image from a pixel array of an image sensor; read a second image from the pixel array; compress the second image based on the first image; and send the first image and the compressed second image to an image sensor processor (ISP).

[0012] In another example, an apparatus for processing one or more images is provided. The apparatus includes: a unit for reading a first image from a pixel array of an image sensor; a unit for reading a second image from the pixel array; a unit for compressing the second image based on the first image; and a unit for sending the first image and the compressed second image to an image sensor processor (ISP).

[0013] In some aspects, one or more of the devices described herein are, are part of, and / or include a wearable device, an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head mounted device (HMD) device, a wireless communication device, a mobile device (e.g., a mobile phone and / or a mobile handset and / or a so-called "smart phone" or other mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or component of a vehicle, another device, or a combination thereof. In some aspects, the device includes one or more cameras for capturing one or more images. In some aspects, the device also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the above-mentioned device may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyroscopes, one or more accelerometers, any combination thereof, and / or other sensors).

[0014] This summary is neither intended to identify key or essential features of the claimed subject matter nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0015] The foregoing and other features and aspects will become more fully apparent after reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Illustrative aspects of the present application are described in detail below with reference to the following figures:

[0017] Figure 1A , Figure 1B and Figure 1C is a schematic diagram illustrating an example configuration of an image sensor of an image capture device according to aspects of the present disclosure.

[0018] Figure 2 is a block diagram illustrating the architecture of an image capture and processing device in accordance with aspects of the present disclosure.

[0019] Figure 3 is a block diagram illustrating an example of an image capture system in accordance with aspects of the present disclosure.

[0020] Figure 4 is a schematic diagram illustrating generation of a fused frame from a short exposure frame and a long exposure frame according to aspects of the present disclosure.

[0021] Figure 5is a schematic diagram illustrating long exposure flow and short exposure flow from an image sensor in accordance with certain aspects of the present disclosure.

[0022] Figure 6 is a schematic diagram illustrating an example of online fusion of one or more short exposure frames and one or more long exposure frames according to aspects of the present disclosure.

[0023] Figure 7 is a conceptual diagram illustrating the difference between images captured by image capture systems having different exposure times according to some aspects of the present disclosure.

[0024] Fig. 8A is a schematic diagram illustrating an image capture system that compresses at least one image to reduce bandwidth and power during high dynamic range (HDR) image synthesis according to some aspects of the present disclosure.

[0025] Figure 8B is a schematic diagram illustrating another image capture system that compresses at least one image to reduce bandwidth and power during HDR image synthesis according to some aspects of the present disclosure.

[0026] Fig. 9 is a block diagram of a compression engine configured to losslessly compress at least one image during HDR image synthesis according to some aspects of the present disclosure.

[0027] Fig.10 is a block diagram of a decompression engine configured to losslessly decompress at least one compressed image during HDR image synthesis according to some aspects of the present disclosure.

[0028] Fig.11 is a schematic diagram illustrating another image capture system that compresses at least one image to reduce bandwidth and power during HDR image synthesis according to some aspects of the present disclosure.

[0029] Fig.12 is a flow chart illustrating an example of a method for compressing an image during HDR image synthesis according to aspects of the present disclosure.

[0030] Fig.13 is a schematic diagram illustrating another image capture system that compresses at least one image in an image sensor to reduce bandwidth and power of an HDR image synthesis process according to some aspects of the present disclosure.

[0031] Fig.14 is a flow chart illustrating another example of a method for compressing an image during HDR image synthesis according to aspects of the present disclosure.

[0032] Fig.15 is a schematic diagram illustrating an example of a system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0033] Some aspects of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects may be applied independently, and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are set forth in order to provide a comprehensive understanding of aspects of the application. However, it will be apparent that various aspects may be practiced without these specific details. The accompanying drawings and description are not intended to be restrictive.

[0034] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. Specifically, the following description of the example aspects will provide a feasible description for implementing the example aspects to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0035] The following description provides only exemplary aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. Specifically, the following description of the exemplary aspects will provide a feasible description of the aspects for implementing the present disclosure to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0036] The terms "exemplary" and / or "example" are used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" and / or "example" is not necessarily to be construed as preferred or advantageous over other aspects. Likewise, the term "aspects of the disclosure" does not require that all aspects of the disclosure include the discussed feature, advantage, or mode of operation.

[0037] A camera is a device that uses an image sensor to receive light and capture an image frame (such as a still image or a video frame). The terms "image", "image frame" and "frame" are used interchangeably herein. A camera may be configured with various image capture and image processing settings. Different settings result in images with different appearances. Some camera settings are determined and applied before or during the capture of one or more image frames, such as ISO, exposure time, aperture size, f / stop, shutter speed, focus and gain. For example, settings or parameters may be applied to an image sensor to capture one or more image frames. Other camera settings may configure post-processing of one or more image frames, such as changes in contrast, brightness, saturation, sharpness, level, curve or color. For example, settings or parameters may be applied to a processor (e.g., an image signal processor (ISP)) to process one or more image frames captured by an image sensor.

[0038] The dynamic range of a digital imaging device (such as a digital camera) is the ratio between the maximum amount of light that the device can capture without light saturation and the lowest amount of light that the device can accurately measure and distinguish from inherent image noise (electrical noise, thermal noise, etc.). Traditionally, digital cameras are able to capture only a small fraction of the natural illumination range of real-world scenes. For example, the dynamic range of a scene might be 100,000:1, while the dynamic range of a digital camera's image sensor might be 100:1. When the dynamic range of a scene exceeds the dynamic range of the sensor, detail is lost in areas of the highest light levels and / or the lowest light levels.

[0039] An imaging device may generate a high dynamic range (HDR) image by merging multiple images captured using different exposure settings. For example, an imaging device may generate an HDR image by merging a short exposure image captured using a short exposure time, a medium exposure image captured using a medium exposure time longer than the short exposure time, and a long exposure image captured using a long exposure time longer than the medium exposure time. Because short exposure images are generally dark, they generally retain most of the details in the highlights (bright areas) of the captured scene. Medium exposure images and long exposure images are generally brighter than short exposure images, and may be overexposed (e.g., too bright to discern details) in the highlights (bright areas) of the scene. Because long exposure images generally include bright parts, they may retain details in the shadows (dark areas) of the captured scene. Medium exposure images and short exposure images are generally darker than long exposure images, and may be underexposed (e.g., too dark to discern details therein) in the shadows (dark areas) of the scene, making their depiction of the shadows too dark to observe details. To generate an HDR image, the imaging device may, for example, use portions of the short exposure image to depict highlights (bright areas) of the captured scene, portions of the long exposure image to depict shadows (dark areas) of the scene, and portions of the medium exposure image to depict other areas of the scene (other than highlights and shadows).

[0040] Imaging devices continue to improve in resolution and frame rate to improve visual fidelity, which increases the bandwidth and power requirements of image capture systems. For example, ultra-high-definition (FHD) video (e.g., with a resolution of 3840x2160 at 30 frames per second (FPS)) requires capturing at least 60 images and sending them from the image sensor to an image signal processor (ISP) device, which processes and outputs the images for recording or display. Adding additional exposure images (such as medium exposure) to improve image quality may exceed the bandwidth capacity between the image sensor and the image processing circuit. Increasing the frame rate may also exceed the bandwidth capacity of the bus. In addition, the processing required for outputting, sending, and storing images in memory consumes a lot of power and may exceed power consumption limits. For example, an example chipset can support 8K video (e.g., with a resolution of 7680x4320) at 30FPS with standard HDR (8K30-SHDR) and consume 3.2 watts of power. However, such an example chipset is thermally limited to 3.1 watts of power. Due to a combination of power consumption and bandwidth limitations, device manufacturers may not be able to support these features, while display manufacturers may support images with higher frame rates and / or higher resolutions.

[0041] In some aspects, systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively referred to herein as "systems and techniques") are described for compressing one or more images, such as during a process of generating a combined image (e.g., during an HDR process (such as an HDR synthesis process) for generating an HDR image). For example, an imaging system may compress a target image based on at least one reference image. In some examples, the reference image may be a long exposure image, and the target image may be a short exposure image (having a shorter or smaller exposure than the long exposure image). The short exposure image may be captured during the capture of the long exposure image. In other examples, the reference image may be a short exposure image, and the target image may be a long exposure image. In some aspects, the systems and techniques may perform lossless compression by determining pixel differences between pixels of the reference image and the target image. In one illustrative example, the short exposure image is normalized relative to the long exposure image, and the difference of each pixel is determined by subtracting the corresponding value of each normalized pixel. If the pixel values ​​between the reference image and the target image are similar, compression efficiency may be achieved. For example, if the normalized values ​​of corresponding pixels differ by a certain value (eg, a value of 32), the pixel difference may be encoded using fewer bits.

[0042] In some aspects of the present disclosure, the systems and techniques may compress one or more images in an image sensor. The systems and techniques may send or transmit the compressed images to an ISP. In one illustrative aspect, the image sensor may read out a reference image from a sensor array and use the reference image to compress subsequent images based on a normalized difference between the reference image and subsequent images. Compressing the subsequent images in the ISP and sending the compressed images may reduce the amount of bandwidth and power consumption.

[0043] Additional details and aspects of the disclosure are described in more detail below with respect to the accompanying drawings.

[0044] Image sensors include one or more arrays of photodiodes or other light-sensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a particular pixel in the image produced by the image sensor. In some cases, different photodiodes may be covered by different color filters of a color filter array and may therefore measure light that matches the color of the color filter covering the photodiode.

[0045] Various color filter arrays may be used, including Bayer color filters, quad color filter arrays (also referred to as quad Bayer filters or QCFAs), and / or other color filter arrays. Figure 1A An example of a Bayer color filter array 100 is shown in FIG. As shown, the Bayer color filter array 100 includes a repeating pattern of red, blue, and green filters. Figure 1BAs shown, QCFA 110 includes a 2x2 (or "quaternary") color filter pattern, including a 2x2 red (R) color filter pattern, a pair of 2x2 green (G) color filter patterns, and a 2x2 blue (B) color filter pattern. The entire photodiode array for a given image sensor is repeated in Figure 1B 1. The pattern of the QCFA 110 shown in . Using the QCFA 110 or Bayer color filter array 100, each pixel of the image is generated based on the following: red light data from at least one photodiode covered in the red filter of the color filter array, blue light data from at least one photodiode covered in the blue filter of the color filter array, and green light data from at least one photodiode covered in the green filter of the color filter array. Other types of color filter arrays may use yellow, magenta and / or cyan (also known as "emerald") filters instead of or in addition to red, blue and / or green filters. Different photodiodes throughout the pixel array may have different spectral sensitivity curves, thereby responding to different wavelengths of light. Monochrome image sensors may also lack color filters and therefore lack color depth.

[0046] In some cases, a subset of adjacent photodiodes (e.g., when using Figure 1B , the 2x2 photodiode patches can measure light of the same color for approximately the same area of ​​the scene. For example, when the photodiodes included in each photodiode subgroup are in close physical proximity, the light incident on each photodiode in the subgroup can originate from approximately the same location in the scene (e.g., a portion of leaves on a tree, a small segment of the sky, etc.).

[0047] In some examples, the brightness range of light from a scene may significantly exceed the brightness levels that an image sensor can capture. For example, a digital single-lens reflex (DSLR) camera may be able to capture light from a scene with a 1:30,000 contrast ratio, while the brightness levels of an HDR scene may exceed a 1:1000,000 contrast ratio.

[0048] In some cases, an HDR sensor can be used to enhance the contrast ratio of an image captured by an image capture device. In some examples, an HDR sensor can be used to obtain multiple exposures within an image or frame, where such multiple exposures can include short (e.g., 5 ms) and long (e.g., 15 ms or more) exposure times. As used herein, a long exposure time generally refers to any exposure time that is longer than a short exposure time.

[0049] In some implementations, the HDR sensor can configure individual photodiodes within a photodiode subgroup (e.g., Figure 1B100) to have different exposure settings. A collection of photodiodes with matching exposure settings is also referred to herein as a photodiode exposure group. Figure 1C FIG. 4 shows a portion of an image sensor array having a QCFA filter configured with four different photodiode exposure groups 1 to 4. Figure 1C As shown in the example photodiode exposure group array 120 in FIG. 1 , each 2×2 tile may include a photodiode from each of the different photodiode exposure groups for a particular image sensor. Figure 1C Four groupings are shown in the specific grouping in , but one of ordinary skill in the art will recognize that different numbers of photodiode exposure groups, different arrangements of photodiode exposure groups within the subgroups, and any combination thereof may be used without departing from the scope of the present disclosure.

[0050] As about Figure 1C As mentioned, in some HDR image sensor implementations, the exposure settings corresponding to different photodiode exposure groups may include different exposure times (also referred to as exposure lengths), such as short exposure, medium exposure, and long exposure. In some cases, different images of a scene associated with different exposure settings may be formed by light captured by the photodiodes in each photodiode exposure group. For example, a first image may be formed by light captured by the photodiodes in photodiode exposure group 1, a second image may be formed by photodiodes in photodiode exposure group 2, a third image may be formed by light captured by photodiodes in photodiode exposure group 3, and a fourth image may be formed by light captured by photodiodes in photodiode exposure group 4. Based on the difference in exposure settings corresponding to each group, the brightness of objects in the scene captured by the image sensor may be different in each image. For example, a well-lit object captured by a photodiode with a long exposure setting may appear saturated (e.g., completely white). In some cases, the image processor may select between pixels of images corresponding to different exposure settings to form a combined image.

[0051] In one illustrative example, the first image corresponds to a short exposure time (also referred to as a short exposure image), the second image corresponds to a medium exposure time (also referred to as a medium exposure image), and the third and fourth images correspond to long exposure times (also referred to as long exposure images). In such an example, pixels of the combined image corresponding to a portion of the scene with low illumination (e.g., a portion of the scene in shadow) may be selected from the long exposure image (e.g., the third image or the fourth image). Similarly, pixels of the combined image corresponding to a portion of the scene with high illumination (e.g., a portion of the scene in direct sunlight) may be selected from the short exposure image (e.g., the first image).

[0052] In some cases, the image sensor can also utilize a photodiode exposure group to capture an object in motion without blur. The length of the exposure time of the photodiode group can correspond to the distance that the object in the scene moves during the exposure time. If the light from the object in motion is captured by the photodiode corresponding to multiple image pixels during the exposure time, the object in motion may appear blurred across the multiple image pixels (also known as motion blur). In some implementations, motion blur can be reduced by configuring one or more photodiode groups with a short exposure time. In some implementations, an image capture device (e.g., a camera) can determine the amount of local motion (e.g., a motion gradient) within a scene by comparing the position of an object between two consecutive captured images. For example, motion can be detected in a preview image captured by an image capture device to provide a preview function to a user on a display. In some cases, a machine learning model can be trained to detect local motion between consecutive images.

[0053] Various aspects of the technology described herein are discussed below with respect to the accompanying figures. Figure 2 2 is a block diagram showing the architecture of the image capture and processing system 200. The image capture and processing system 200 includes various components for capturing and processing an image of a scene (e.g., an image of the scene 210). The image capture and processing system 200 may capture an independent image (or photograph) and / or may capture a video including a plurality of images (or video frames) in a particular order. In some cases, the lens 215 and the image sensor 230 may be associated with an optical axis. In one illustrative example, both the photosensitive area of ​​the image sensor 230 (e.g., a photodiode) and the lens 215 may be centered on the optical axis. The lens 215 of the image capture and processing system 200 faces the scene 210 and receives light from the scene 210. The lens 215 bends the incident light from the scene toward the image sensor 230. The light received by the lens 215 passes through an aperture. In some cases, the aperture (e.g., the aperture size) is controlled by one or more control mechanisms 220 and received by the image sensor 230. In some cases, the aperture may have a fixed size.

[0054] The one or more control mechanisms 220 may control exposure, focus, and / or zoom based on information from the image sensor 230 and / or based on information from the image processor 250. The one or more control mechanisms 220 may include a plurality of mechanisms and components; for example, the control mechanisms 220 may include one or more exposure control mechanisms 225A, one or more focus control mechanisms 225B, and / or one or more zoom control mechanisms 225C. The one or more control mechanisms 220 may also include additional control mechanisms in addition to those shown, such as control mechanisms that control analog gain, flash, HDR, depth of field, and / or other image capture characteristics.

[0055] The focus control mechanism 225B of the control mechanism 220 can obtain the focus setting. In some examples, the focus control mechanism 225B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 225B can adjust the position of the lens 215 relative to the position of the image sensor 230. For example, based on the focus setting, the focus control mechanism 225B can move the lens 215 closer to the image sensor 230 or farther away from the image sensor 230 by driving a motor or servo device (or other lens mechanism), thereby adjusting the focus. In some cases, additional lenses may be included in the image capture and processing system 200, such as one or more micro lenses on each photodiode of the image sensor 230, and the one or more micro lenses each bend the light received from the lens 215 to the corresponding photodiode before the light reaches the photodiode. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid focus (HAF), or some combination thereof. The focus setting may be determined using the control mechanism 220, the image sensor 230, and / or the image processor 250. The focus setting may be referred to as an image capture setting and / or an image processing setting. In some cases, lens 215 may be fixed relative to the image sensor, and focus control mechanism 225B may be omitted without departing from the scope of the present disclosure.

[0056] The exposure control mechanism 225A of the control mechanism 220 can obtain an exposure setting. In some cases, the exposure control mechanism 225A stores the exposure setting in a memory register. Based on the exposure setting, the exposure control mechanism 225A can control the size of the aperture (e.g., aperture size or f / stop), the duration that the aperture is open (e.g., exposure time or shutter speed), the duration that the sensor collects light (e.g., exposure time or electronic shutter speed), the sensitivity of the image sensor 230 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 230, or any combination thereof. The exposure setting can be referred to as an image capture setting and / or an image processing setting.

[0057] The zoom control mechanism 225C of the control mechanism 220 can obtain the zoom setting. In some examples, the zoom control mechanism 225C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 225C can control the focal length of the assembly of lens elements (lens assembly) including the lens 215 and one or more additional lenses. For example, the zoom control mechanism 225C can control the focal length of the lens assembly by driving one or more motors or servos (or other lens mechanisms) to move one or more of these lenses relative to each other. The zoom setting can be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly can include a parfocal zoom lens or a zoom plane zoom lens. In some examples, the lens assembly can include a focusing lens (which can be the lens 215 in some cases) that first receives light from the scene 210, wherein the light then passes through an afocal zoom system (afocal zoom system) between the focusing lens (e.g., the lens 215) and the image sensor 230 before the light reaches the image sensor 230. In some cases, the afocal zoom system may include two positive (e.g., converging, convex) lenses having equal or similar focal lengths (e.g., within a threshold difference of each other) with a negative (e.g., diverging, concave) lens between them. In some cases, zoom control mechanism 225C moves one or more of the lenses in the afocal zoom system, such as one or both of the positive lenses and the negative lens. In some cases, zoom control mechanism 225C may control the zoom by capturing images from an image sensor in a plurality of image sensors (e.g., including image sensor 230) with a zoom corresponding to a zoom setting. For example, image processing system 200 may include a wide-angle image sensor with a relatively low zoom and a telephoto image sensor with a larger zoom. In some cases, based on the selected zoom setting, zoom control mechanism 225C may capture images from the corresponding sensors.

[0058] Image sensor 230 includes one or more arrays of photodiodes or other light sensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a particular pixel in the image produced by image sensor 230. In some cases, different photodiodes can be covered by different color filters. In some cases, different photodiodes are covered in color filters and can therefore measure light that matches the color of the color filter covering the photodiode. Various color filter arrays can be used, including Bayer color filter arrays (such as Figure 1A shown), QCFA (see Figure 1B ) and / or any other color filter.

[0059] Back to Figure 1A and Figure 1B , other types of color filters may use yellow, magenta and / or cyan (also referred to as "emerald") color filters instead of or in addition to red, blue and / or green color filters. In some cases, some photodiodes may be configured to measure infrared light (IR). In some implementations, the photodiode measuring IR light may not be covered by any filter, allowing the IR photodiode to measure both visible light (e.g., color) and IR light. In some examples, the IR photodiode may be covered by an IR filter, allowing IR light to pass through and blocking light from other parts of the spectrum (e.g., visible light, color). Some image sensors (e.g., image sensor 230) may lack color filters (e.g., color, IR, or any other part of the spectrum) entirely, and may instead use different photodiodes (in some cases vertically stacked) throughout the pixel array. Different photodiodes throughout the pixel array may have different spectral sensitivity curves, responding to light of different wavelengths. Monochrome image sensors may also lack color filters, and therefore lack color depth.

[0060] In some cases, the image sensor 230 may alternatively or additionally include an opaque and / or reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles. In some cases, an opaque and / or reflective mask may be used for PDAF. In some cases, an opaque and / or reflective mask may be used to block portions of the electromagnetic spectrum from reaching the photodiodes of the image sensor (e.g., IR cutoff filters, ultraviolet (UV) cutoff filters, bandpass filters, low-pass filters, high-pass filters, etc.). The image sensor 230 may also include an analog gain amplifier for amplifying an analog signal output by a photodiode and / or an analog-to-digital converter (ADC) for converting the analog signal output of the photodiode (and / or amplified by the analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed with respect to one or more of the control mechanisms 220 may be alternatively or additionally included in the image sensor 230. Image sensor 230 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal oxide semiconductor (CMOS), an N-type metal oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0061] The image processor 250 may include one or more processors, such as one or more ISPs (e.g., ISP 254), one or more host processors (e.g., host processor 252), and / or Fig.15The host processor 252 may be a digital signal processor (DSP) and / or other types of processors. In some implementations, the image processor 250 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes the host processor 252 and the 2SP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 256), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), a memory, a connection component (e.g., Bluetooth TM, a global positioning system (GPS), etc.), any combination thereof, and / or other components. I / O ports 256 may include any suitable input / output ports or interfaces according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial general purpose input / output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical layer (PHY) port or interface, an Advanced High-Performance Bus (AHB) bus, any combination thereof), and / or other input / output ports. In one illustrative example, host processor 252 may use an I2C port to communicate with image sensor 230, and ISP 254 may use a MIPI port to communicate with image sensor 230.

[0062] The image processor 250 may perform a number of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The image processor 250 may store the image frames and / or processed images in a random access memory (RAM) 240, a read-only memory (ROM) 245, a cache, a memory unit, another storage device, or some combination thereof.

[0063] Various input / output (I / O) devices 260 may be connected to the image processor 250. The I / O devices 260 may include a display screen, a keyboard, a keypad, a touch screen, a touch pad, a touch sensitive surface, a printer, any other output device 1635, any other input device 1645, or some combination thereof. In some cases, captions may be entered into the image processing device 205B via a physical keyboard or keypad of the I / O device 260, or via a virtual keyboard or keypad of a touch screen of the I / O device 260. The I / O 260 may include one or more ports, jacks, or other connectors that enable a wired connection between the image capture and processing system 200 and one or more peripheral devices, through which the image capture and processing system 200 may receive data from and / or send data to the one or more peripheral devices. The I / O 260 may include one or more wireless transceivers that enable wireless connections between the image capture and processing system 200 and one or more peripheral devices, through which the image capture and processing system 200 can receive data from and / or send data to the one or more peripheral devices. The peripheral devices may include any of the types of I / O devices 260 discussed previously, and once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector, they themselves may be considered I / O devices 260.

[0064] In some cases, the image capture and processing system 200 can be a single device. In some cases, the image capture and processing system 200 can be two or more separate devices, including an image capture device 205A (e.g., a camera) and an image processing device 205B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 205A and the image processing device 205B can be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some implementations, the image capture device 205A and the image processing device 205B can be disconnected from each other.

[0065] like Figure 2 As shown, the vertical dashed line will Figure 2The image capture and processing system 200 is divided into two parts, which are respectively represented by the image capture device 205A and the image processing device 205B. The image capture device 205A includes a lens 215, a control mechanism 220, and an image sensor 230. The image processing device 205B includes an image processor 250 (including an ISP 254 and a host processor 252), a RAM 240, a ROM 245, and an I / O 260. In some cases, some components shown in the image capture device 205A (such as the ISP 254 and / or the host processor 252) can be included in the image capture device 205A.

[0066] The image capture and processing system 200 may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 200 may include one or more wireless transceivers for wireless communication (such as cellular network communication, 802.11 wi-fi communication, wireless local area network (WLAN) communication, or some combination thereof). In some implementations, the image capture device 205A and the image processing device 205B may be different devices. For example, the image capture device 205A may include a camera device, and the image processing device 205B may include a computing device, such as a mobile handset, a desktop computer, or other computing devices.

[0067] Although the image capture and processing system 200 is shown as including certain components, one of ordinary skill will appreciate that the image capture and processing system 200 may include other components. Figure 2 . The components of the image capture and processing system 200 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 200 may include electronic circuits or other electronic hardware (which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other appropriate electronic circuits)) and / or may be implemented using electronic circuits or other electronic hardware, and / or may include computer software, firmware, or any combination thereof and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions, one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device that implements the image capture and processing system 200.

[0068] Figure 3is a block diagram illustrating an example of an image capture system 300. The image capture system 300 includes various components for processing input images or frames to produce output images or frames. As shown, the components of the image capture system 300 include one or more image capture devices 302, an image processing engine 310, and an output device 312. The image processing engine 310 can produce a high dynamic range depiction of a scene, as described in more detail herein.

[0069] The image capture system 300 may include an electronic device or system, or be a portion of an electronic device or system. For example, the image capture system 300 may include an electronic device or system, or be a portion of an electronic device or system, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a vehicle or a computing device / system of a vehicle, a server computer (e.g., in communication with another device or system (such as a mobile device, an XR system / device, a vehicle computing system / device, etc.)), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera device, a display device, a digital media player, a video streaming device, or any other suitable electronic device. In some examples, the image capture system 300 may include one or more wireless transceivers (or separate wireless receivers and transmitters) for wireless communications (such as cellular network communications, 802.11 Wi-Fi communications, WLAN communications, Bluetooth or other short-range communications, any combination thereof, and / or other communications). In some implementations, the components of the image capture system 300 may be part of the same computing device. In some implementations, components of image capture system 300 may be part of two or more separate computing devices.

[0070] Although image capture system 300 is shown as including certain components, one of ordinary skill will appreciate that image capture system 300 may include more than the components shown in FIG. Figure 3 In some cases, the additional components of image capture system 300 may include software, hardware, or one or more combinations of software and hardware. For example, in some cases, image capture system 300 may include Figure 3One or more other sensors not shown in the image capture system 300 (e.g., one or more inertial measurement units (IMUs), radars, light detection and ranging (LIDAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and / or one or more other software and / or hardware components. In some implementations, the additional components of the image capture system 300 may include and / or be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., DSPs, microprocessors, microcontrollers, GPUs, CPUs, any combination thereof, and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the image capture system 300.

[0071] One or more image capture devices 302 may capture image data and generate images (or frames) based on the image data, and / or may provide the image data to an image processing engine 310 for further processing. One or more image capture devices 302 may also provide the image data to an output device 312 for output (e.g., on a display). In some cases, the output device 312 may also include a storage device. An image or frame may include an array of pixels representing a scene. For example, the image may be: a red-green-blue (RGB) image having red, green, and blue components per pixel; a luminance, chrominance red, chrominance blue (YCbCr) image having a luminance component and two chrominance (color) components (chrominance red and chrominance blue) per pixel; or any other suitable type of color or monochrome image. In addition to the image data, the image capture device may also generate supplemental information, such as the amount of time between consecutively captured images, a timestamp of image capture, and the like.

[0072] Figure 4A technique for generating a fused frame (also referred to as a combined frame or combined image) from a short exposure frame and a long exposure frame is shown. As shown, a short exposure frame 402 and a long exposure frame 404 may be employed, which may be fused to provide a fused frame output 406 (e.g., an HDR frame output). Due to the bit depth of the image capture sensor, some pixels of the captured frame may be oversaturated, resulting in an image that does not show some texture of the scene shown in the short exposure frame 402. Therefore, in order to generate an HDR frame, both a short exposure frame and a long exposure frame may be captured, which may be fused (e.g., combined) to generate an HDR output frame. Fusion of the short exposure frame and the long exposure frame may be performed to generate a fused output frame that includes a portion of the short exposure frame and a portion of the long exposure frame. For example, region 408 of the fused frame output 406 may be from the long exposure frame 404, and region 410 of the fused frame output 406 may be from the short exposure frame 402. However, fusing the short exposure frame and the long exposure frame may result in irregularities due to global motion (e.g., motion of the image capture device). For example, from the time the long exposure frame is captured to the time the short exposure frame is captured, the image capture device or objects in the scene may have moved, causing irregularities (if steps are not taken to align the short and long exposure frames before fusing them together). As described in more detail herein, such global motion problems may also occur due to rolling shutter.

[0073] Figure 5 is a schematic diagram showing long exposure flow and short exposure flow (e.g., MIPI flow) from an image sensor (e.g., image sensor 230) to an imaging front end for processing. Line 502 represents the start of long exposure sensing (also referred to herein as normal exposure sensing), while line 504 represents the end of long exposure sensing. Long exposure sensing starts from the first row of the sensor (e.g., Figure 2 230) to the last row of the sensor, as shown. For each row (e.g., a row of photodiodes), once long exposure sensing has been completed, short exposure sensing begins, and long exposure sensing continues to the next row. For example, starting from the first row to the last row of the image sensor, line 506 represents the beginning of short exposure sensing, and line 508 represents the end of short exposure sensing. Long exposure sensing (e.g., in Figure 5 with a duration marked as “Nnormal”) can be used in short exposure sensing (e.g., Figure 5 with a duration marked as “N short”).

[0074] Once long exposure sensing for a particular row is complete, a short delay occurs (e.g., associated with the gap between lines 504, 506) before short exposure sensing begins. Once short exposure sensing for a particular row has ended, information for that row is read out from the image sensor for processing. Due to the gap from long exposure sensing to short exposure sensing (e.g., between Figure 5 There is a chance that the user holding the camera moves and / or that objects in the scene being captured move, resulting in misalignment of features in the short exposure frame and the long exposure frame (e.g., features that are common or identical in the short exposure frame). For example, there may be a motion delay (D) from time 550 (e.g., the time when half of the long exposure data is captured) to time 552 (e.g., the time when half of the short exposure data is captured). The motion delay (D) can be estimated as the average motion delay associated with different long frame capture events and short frame capture events (e.g., different HDR frame captures).

[0075] Because sensing occurs one row at a time (e.g., starting from the first row to the last row), rolling shutter global motion also occurs. From the time data for the first row of sensors is captured to the time data for the last row of sensors is captured, the camera or objects in the scene may move.

[0076] Figure 6 604 and one or more long exposure frames 602. The fusion engine 606 can fuse the one or more short exposure frames 604 and the one or more long exposure frames 602 to generate an HDR frame or image. Figure 5 As described, long exposure data corresponding to one or more long exposure frames 602 may be captured for each row before short exposure data corresponding to one or more short exposure frames 604. Thus, data from each row of one or more long exposure frames 602 may be received and stored in buffer 603 before data for each row of one or more short exposure frames 604 is stored in buffer 605. As shown, accumulation of data for one or more long exposure frames 602 may precede accumulation of data for one or more short exposure frames 604 (e.g., because Figure 5 As shown, the long exposure capture occurs before the short exposure capture).

[0077] In some cases, fusion by the fusion engine 606 may begin once short frame data corresponding to one or more short exposure frames 604 has been accumulated for a particular number of sensor rows or lines (e.g., the first 3 rows / lines, the first 4 rows / lines, the first 8 rows / lines, or another number of rows / lines). For example, a frame alignment operation may begin when short frame data for a particular number of sensor rows is received (e.g., instead of waiting to receive the entire frame). However, various constraints may exist when performing frame alignment. For example, it may not be possible to completely warp a long exposure frame (from one or more long exposure frames 602) to align with a short exposure frame (from one or more short exposure frames 604). In addition, due to hardware timing constraints, programming of alignment may have to be performed two or three frames in advance. In some aspects, a large buffer may be established for capturing frame data. Image data from an image sensor may be written to a central portion of an image buffer, enabling shifts in the x and y dimensions to be applied to the data stored in the buffer for alignment. In addition, certain aspects of the present disclosure provide techniques for alignment prediction to allow alignment operations to be programmed in advance.

[0078] Figure 7 700 is a conceptual diagram illustrating the difference between images captured by image capture systems having different exposure times according to some aspects of the present disclosure. Specifically, the diagram 700 illustrates a short exposure image 702, a medium exposure image 704, and a long exposure image 706 captured by an image processing system. In a conventional HDR image synthesis process, the short exposure image 702, the medium exposure image 704, and the long exposure image 706 are provided from an image sensor to an ISP (e.g., ISP 254) for processing. The ISP may store the short exposure image 702, the medium exposure image 704, and the long exposure image 706 in a memory, and then process the short exposure image 702, the medium exposure image 704, and the long exposure image 706 into a single HDR image.

[0079] As described above, each of the short exposure image 702, the medium exposure image 704, and the long exposure image 706 is read out from the sensor array by the image sensor and has a different exposure time, but captures the same content. Schematic diagram 700 shows that the short exposure image 702 is multiplied by the short exposure to long exposure ratio to normalize the short exposure image relative to the long exposure image 706 in intensity. For example, if the short exposure time is 10 milliseconds (ms) and the long exposure time is 40ms, the brightness of the pixel in the short exposure image 702 is multiplied by the exposure ratio of 4.0. Then, the normalized short exposure image 702 and the long exposure image 706 are compared in the pixel differentiator 710. In an illustrative example, the pixel differentiator 710 calculates the difference between each pixel to generate a differential pixel bitmap, and the graph 720 shows the difference in pixel values ​​between the normalized short exposure image 702 and the long exposure image 706.

[0080] Graph 720 shows the difference of all pixel values ​​between short exposure image 702 and long exposure image 706, where the center point 0 indicates that the pixel values ​​are the same, and the absolute value of the difference corresponds to how different the pixel is between short exposure image 701 and long exposure image 706. In this illustrative example, 95.1% of all pixel values ​​are within a value of 64.

[0081] Schematic diagram 700 shows that a medium exposure image 704 is multiplied by a medium to long exposure ratio to normalize the brightness (e.g., lux) of pixels in a short exposure image 702 relative to a long exposure image 706. For example, if the medium exposure time is 20 milliseconds (ms) and the long exposure time is 40 ms, the brightness of the pixels in the medium exposure image 704 is multiplied by an exposure ratio of 2.0. The normalized medium exposure image 704 and the long exposure image 706 are then compared in a pixel differentiator 711. In one illustrative example, the pixel differentiator 712 calculates the difference between each pixel to generate a differential pixel bitmap, and a graph 730 shows the difference in pixel values ​​between the normalized medium exposure image 704 and the long exposure image 706. For the normalized short exposure image 702, 99.3% of all pixels are within a value of 32.

[0082] According to some aspects of the present disclosure, a reference image (e.g., a long exposure image) can be used to compress the pixels of an image based on pixel differences. By compressing an image based on a reference image, the compressed image can be stored in a memory at a reduced size, which reduces the number of bits that need to be written to the memory device and saves power. In addition, reducing the size of the image written to the memory also enables device manufacturers to support higher frame rates and higher resolutions.

[0083] In one illustrative aspect, a reference image is selected to compare other images with and serves as the basis for the compression described below. Figure 7 It is described that the long exposure image is a reference image and provides the best compression. Alternatively, other exposure images (e.g., short exposure images and medium exposure images) can be used as reference images. For example, the short exposure image can be used as a reference image because it is read out first from the image sensor.

[0084] Fig. 8A 8 is a schematic diagram illustrating an image capture system 800 that compresses at least one image during HDR image synthesis to reduce bandwidth and power according to some aspects of the present disclosure. The image capture system 800 includes a front end 802 of an ISP that receives at least a short exposure image 804 and a long exposure image 806 from an image sensor (not shown) and a fusion engine 820 that synthesizes the short exposure image 804 and the long exposure image 806 into an HDR image 830. In some aspects, the ISP is connected to a memory device 810 for storing and retrieving images received by the front end 802 for processing. For example, the memory device can be a double data rate (DDR) memory.

[0085] In some aspects, the image capture system 800 includes a compressor 808 that receives the short exposure image 804 and the long exposure image 806 and compresses the short exposure image 804 based on the pixel difference between the short exposure image 803 and the long exposure image 806. Fig. 9 A detailed block diagram of the compressor 808 is described. The compressor 808 stores the compressed image in a memory device 810 (eg, a DDR memory), and the front end 802 may also store the long exposure image 806 in the memory device 810.

[0086] In some cases, the image capture system 800 may be processing other content and temporarily storing the compressed short exposure image 804 and long exposure image 806 until the previous image is processed. The fusion engine 820 may request the short exposure image 804 and the long exposure image 806, which causes the memory device 810 to read the compressed short exposure image 802 and the long exposure image 806. The decompressor 812 receives both the compressed short exposure image 804 and the long exposure image 806 and decompresses the short exposure image 804 based on the pixel differences encoded into the short exposure image 804 relative to the pixels in the long exposure image 806, and provides the decompressed short exposure image 804 to the fusion engine 820. The fusion engine also receives the long exposure image 806 and fuses the short exposure image 804 and the long exposure image 808 into an HDR image 830.

[0087] In some aspects, the compressor 808 can be lossless and based on the pixel differences of each pixel within the reference image, or in this illustrative example, based on the long exposure image 806. For example, because the decompressor 812 uses the encoded pixel differences in the compressed short exposure image 804 and combines the pixel differences with the pixels in the long exposure image 806, the original values ​​of the long exposure image 806 can be restored. In some other aspects, the compressor 808 can also use lossy compression based on the average of the pixels in the reference image. As will be described below with reference to Fig.11 As described, the cache may not be of sufficient size and a downsampled reference image may be used to support lossy compression.

[0088] Figure 8B 8 is a schematic diagram illustrating another image capture system 850 that compresses at least one image during HDR image synthesis to reduce bandwidth and power according to some aspects of the present disclosure. The image capture system 850 includes a cache 860 for storing reference images (e.g., long exposure image 806). For example, the cache 860 can be an integral part of a hardware processor integrated into the ISP, and due to the faster readout performance of the cache 860 compared to the memory device 810, it can store the reference image.

[0089] Fig. 9 9 is a block diagram of a compressor 900 configured to losslessly compress at least one image during HDR image synthesis according to some aspects of the present disclosure. The compressor 900 includes a pixel differentiator 910 configured to identify pixel differences as described above and a pixel encoder 950 that encodes the pixel differences into a format that reduces the size of the image.

[0090] In one illustrative example, the pixel differentiator 910 includes a subtractor 912 that receives a reference image and a black level (BL). The black level is the lowest value that an image sensor can read out and is an inherent characteristic associated with an image sensor (e.g., a Bayer filter). For example, assuming that the black level is an 8-bit value, an example black level may be 60 and is the minimum value that the image sensor will read out for a single pixel. The subtractor 912 subtracts the black level from the pixel to produce a normalized reference image that minimizes any variation.

[0091] The pixel differentiator 910 also includes a subtractor 914 that receives a target image to be compressed (e.g., a short exposure image, a medium exposure image, etc.) and, as part of the normalization process, subtracts the black level to minimize any variations in the various images. In one illustrative example, the target image is a short exposure image. After subtracting the black level, the BL corrected target image is provided to a multiplier 916, which also receives an exposure ratio. The multiplier 916 is configured to multiply the BL corrected target image by the exposure ratio to increase the brightness of the image to correspond to a reference image. The multiplier 916 produces a normalized target image, and both the normalized target image and the normalized reference image are provided to a subtractor 918. The subtractor 918 subtracts each pixel from the normalized target image and the normalized reference image to produce a differential value representing any difference between the normalized target image and the normalized reference image. As described above with reference to Figure 7 As described, 95% of all values ​​in the short exposure image and the long exposure image have a difference of 64, which can be represented by 6 bits.

[0092] In one illustrative example, the multiplier 916 outputs the normalized target image to the comparator 920, which compares the pixel in the normalized target image with the maximum value of the reference image. If the value of the pixel in the normalized target image is greater than the maximum value of the reference image, the comparator 920 is configured to control a switch 922 (e.g., a multiplexer) to output the differential value from the subtractor 918 or the maximum value of the reference image.

[0093] Alternatively or additionally, the pixel differentiator 910 can be configured to use a different reference image, and the pixel differentiator 910 can be modified to normalize the reference image and the target image. For example, if the short exposure image is used as the reference image in the pixel differentiator 910, and the long exposure image is the target image, the short exposure image will be normalized relative to the long exposure image (e.g., multiplied based on the exposure ratio). Table 1 shows pseudo code implemented by the pixel differentiator 910 for determining pixel differences for the short exposure image as the reference image.

[0094]

[0095]

[0096] Table 1

[0097] The pixel differentiator 910 outputs the pixel difference to a pixel encoder 950, which includes a sign detector 952 configured to determine a sign (e.g., positive or negative) associated with the pixel difference. The pixel encoder 950 also includes a value detector 954 that calculates an absolute value (e.g., removes the sign), and the absolute value is provided to a comparator 956. The comparator 956 compares the absolute value to a threshold value. The threshold value is the maximum pixel difference used to determine the output value of the pixel encoder 950.

[0098] In some aspects, the output of comparator 956 is provided as a control input to a switch 958 (e.g., a multiplexer) for controlling the primary output and a switch 960 for controlling the secondary output. If the threshold is greater than the pixel difference, switch 958 provides a primary output Y as the pixel difference, and the switch provides a value of zero. Combiner 962 receives the sign from sign detector 952, the primary output from switch 958, and the secondary output from switch 960, and packs these values ​​into an encoded value, but omits the secondary output due to the zero magnitude. Because the threshold is less than the pixel difference, the value of the primary output is reduced in magnitude (e.g., from 16 bits to 6 bits).

[0099] If the threshold is less than the pixel difference (such as in Figure 7 , which is an uncommon case), switch 958 provides the threshold value as an output to combiner 962, and switch 960 provides the output of the pixel difference less than the threshold to combiner 962. In this example, combiner 962 is configured to pack the received value into the encoded value. In this example, since the packing includes the secondary output, the packing does not reduce the size. Table 2 below shows pseudo code implemented by pixel encoder 950 and indicates that the secondary output is provided only when the pixel difference is greater than or equal to the upper boundary of the threshold.

[0100]

[0101] Table 2

[0102] Fig.10 1 is a block diagram of a decompressor 1000 configured to losslessly decompress at least one compressed image during HDR image synthesis according to some aspects of the present disclosure. In some aspects, the decompressor 1000 is configured to receive an encoded pixel stream encoded based on pixel differences between a reference image and a target image to restore the target image. The decompressor 1000 includes a pixel decoder 1010 configured to restore the pixel differences and a pixel restorer 1030 configured to restore values ​​of the target image from the compressed image and the reference image.

[0103] The pixel decoder 1010 includes a divider 1012 configured to separate various discrete values ​​of the compressed image. For example, each pixel in the compressed image includes a sign and a secondary output based on a pixel difference, and may include a primary output. The divider is configured to provide the secondary output to an adder 1014, which adds a threshold value to the secondary output, and the output of the adder 1014 may correspond to a pixel difference between a reference image and a target image. The pixel difference and the primary output from the adder 1014 are provided as inputs to a switch 1016. The primary output is also provided to a comparator 1018, which compares the primary output with a threshold and provides a control input to control the output of the switch 1016. In an illustrative example, if the primary output is greater than or equal to the threshold, the comparator 1018 controls the switch 1016 to output the primary output because the value is not encoded. When the primary output is less than the threshold, the output of the adder 1014 corresponds to the pixel difference between the reference image and the compressed image, and the switch 1016 is controlled to output the pixel difference. The output of the switch 1016 and the sign provided by the divider 1012 are received at the combiner 1020, which generates the output of the pixel decoder 1010. In some aspects, the output of the pixel decoder 1010 is a differential pixel value (e.g., +30, -30) encoded to represent the difference associated with a single pixel, or represents the value of a single pixel. Table 3 below shows pseudo code implemented by the pixel decoder 1010 according to some aspects of the present disclosure.

[0104]

[0105] Table 3

[0106] In some aspects, the pixel restorer 1030 is configured to reconstruct the target image based on the pixel difference. In one illustrative aspect, the pixel restorer 1030 includes a subtractor 1032, which subtracts the black level from the pixel in the reference image. In an illustrative example, the output of the subtractor 1032 is provided to a multiplier and multiplied by an exposure ratio. In some aspects, the exposure ratio is based on the ratio of the reference image to the longest exposure image. In this case, the reference image is the longest exposure image, and the ratio will be 1.

[0107] The multiplier 1034 normalizes the reference image, and the normalized reference image is provided to an adder 1036, which receives the pixel difference and adds it to the normalized reference image. The pixel difference is also provided to a subtractor 1038, which is configured to receive the maximum pixel value. The outputs of the adder 1036 and the subtractor 1038 are provided to a switch 1040, which is configured to select an output based on a comparator 1042. Specifically, the comparator 1042 receives and compares the maximum pixel value and the normalized reference image. If the pixel of the reference image is greater than or equal to the maximum pixel value, the comparator 1042 controls the switch 1040 to output from the subtractor 1038. When the pixel cannot be encoded because its value falls outside the threshold, the output from the subtractor is selected. If the pixel of the reference image is less than the maximum pixel value, the comparator 1042 is configured to control the switch 1040 to output the value from the adder 1036. In some aspects, when the pixel difference is less than a threshold (e.g., a maximum pixel difference of ±32), the output from the adder 1036 is selected and the output is encoded by the compressor. The switch 1040 is configured to select between the encoded value in the compressed image and the unencoded value in the compressed image to completely recreate the target image without any loss. Table 4 below shows the pseudo code implemented by the pixel restorer 1030, and shows that the value of the pixel for the decompressed image is selected based on whether the following is true.

[0108]

[0109] Table 4

[0110] When the difference between a pixel in the target image and a corresponding pixel in the reference image is within a range (eg, ±32, ±64, etc.), the value of the compressed image may represent the encoded value.

[0111] Fig.11 1 is a schematic diagram of another image capture system 1100 illustrating an ISP that compresses at least one image during HDR image synthesis to reduce bandwidth and power in accordance with some aspects of the present disclosure. In some aspects, a front end 1110 of the ISP is configured to receive at least a short exposure image 1112 and a long exposure image 114 from an image sensor (not shown).

[0112] The long exposure image 1114 is configured as a reference image and is provided to a downsizer 1116, which reduces the size of the long exposure image 1114 and stores the downscaled long exposure image 1114 in a cache 1120. In some aspects, the ISP may be configured to have a limited cache to reduce costs associated with the ISP, and the value layer of the ISP may reduce the cache size. In such a case, the size of the long exposure image 1114 may be too large to be stored in the cache 1120, and a downscaled version of the long exposure image 1114 may fit into the cache 1120. In one illustrative aspect, the pixels of the long exposure image 1114 may be grouped and averaged to reduce the size to allow the cache 1120 to store the reference image. For example, a 2x2 pixel grid may be averaged, which reduces the number of pixels by four times and may enable the downscaled long exposure image 1114 to be stored in the cache 1120. The downscaling in this example is described as averaging, but any suitable downscaling technique (such as nearest neighbor resampling, bilinear resampling, bicubic resampling, sinc resampling, spline resampling, Lanczos resampling, etc.) may be used. When compressor 1124 receives short exposure image 1112, compressed long exposure image 1114 is read out of cache 1120 and amplified by amplifier 1122, and compressed short exposure image 1112 is stored in memory together with uncompressed long exposure image 1114.

[0113] The uncompressed short exposure image 1112 is read out of the memory device 1130 and provided to the decompressor 1140 along with the upscaled reference image from the amplifier 1122. In some aspects, the decompressor 1140 restores the values ​​of the short exposure image 1112 based on the downscaled reference image. In this case, the pixels in the decompressed short exposure image 1112 will lose some quality based on the averaging applied to the reference image. The fusion engine 1150 receives the long exposure image 1114 and decompresses the short exposure image 1112 using the downscaled reference image and synthesizes the image into an HDR image 1160.

[0114] In some aspects, the resulting HDR image 1160 will lose some quality because the pixels are restored based on a downscaled version of the reference image. In some cases, the fusion engine 1150 can be configured to synthesize the short exposure image and the long exposure image based on resampling (e.g., downscaling, upscaling), which can reduce any artifacts that appear.

[0115] Fig.121 is a flowchart illustrating an example of a method 1200 for compressing an image during HDR image synthesis in accordance with certain aspects of the present disclosure. The method 1200 may be performed by a computing device (or a component thereof, such as a chipset) having an image sensor, such as a mobile wireless communication device, a camera, an XR device, a wireless-enabled vehicle, or another computing device. In one illustrative example, the computing system 1500 may be configured to perform all or part of the method 1200. In one illustrative example, an ISP such as the ISP 254 may be configured to perform all or part of the method.

[0116] At block 1202, a computing device (or a component thereof) may obtain a first image captured using an image sensor. The first image is associated with a first exposure. In some aspects, the computing device (or a component thereof) may cause the first image to be stored in a cache. For example, the computing device (or a component thereof) may downsample the first image and may cause the downsampled first image to be stored in the cache. The computing device (or a component thereof) may further upsample the downsampled first image to a lossy first image.

[0117] At block 1204, the computing device (or a component thereof) may obtain a second image captured using the image sensor. The second image is associated with a second exposure. For example, the second exposure may be less than the first exposure.

[0118] At box 1206, the computing device (or its component) may compress the second image based on the comparison of the second image with the first image. The compression of the second image may be lossless. In one illustrative example, to compress the second image, the computing device (or its component) may calculate the pixel difference between the pixel of the first image and the pixel of the second image, and may determine the compressed value for at least a portion of the pixel in the second image based on the pixel difference. In some cases, to calculate the pixel difference, the computing device (or its component) may normalize the first image and the second image based on the minimum pixel value associated with the image sensor. In some examples, the computing device (or its component) may multiply the second image by an exposure ratio associated with the second image and the first image to generate a modified image. In some cases, the computing device (or its component) may subtract the value of the pixel in the modified image from the maximum value in the first image. In one illustrative example, the value of the pixel of the modified image is greater than or equal to the maximum value, and based on the value of the pixel of the modified image being greater than or equal to the maximum value, the pixel difference corresponds to the maximum value. In another illustrative example, the value of the pixel of the modified image is less than a maximum value, and based on the value of the pixel in the modified image being less than the maximum value, the pixel difference corresponds to a difference between the value of the pixel in the modified image and the maximum value.

[0119] In some aspects, to determine a compressed value for at least a portion of the pixels in the second image, the computing device (or a component thereof) may compare the pixel difference to a threshold. In one illustrative example, based on a determination that the pixel difference is less than or equal to the threshold, the computing device (or a component thereof) may use the value of the pixel difference as the compressed value for at least a portion of the pixels in the second image. In one illustrative example, based on a determination that the pixel difference is greater than the threshold, the computing device (or a component thereof) may use the value of at least one pixel in the second image as the compressed value for at least a portion of the pixels in the second image. In some cases, the value of at least one pixel used as the compressed value includes the pixel difference.

[0120] At block 1208, the computing device (or a component thereof) may store the compressed second image in a memory. At block 1210, the computing device (or a component thereof) may obtain the compressed second image from the memory.

[0121] At block 1212, the computing device (or its component) may decompress the compressed second image based on the difference between the compressed second image and the first image. In some cases, to decompress the compressed second image, the computing device (or its component) may determine the pixel difference of the pixels within the compressed second image. The computing device (or its component) may sum the pixel difference of the pixels in the compressed second image with the pixels in the first image. In some examples, to determine the pixel difference, the computing device (or its component) may add the threshold to the value of the pixel within the compressed second image based on whether the value of the pixel is greater than the threshold. In some aspects, to compress the second image, the computing device (or its component) may calculate the pixel difference of the pixels between the lossy first image and the second image. The computing device (or its component) may determine the compressed value for at least a portion of the pixels in the second image based on the pixel difference. In some cases, to decompress the compressed second image, the computing device (or its component) may determine the pixel difference of the pixels within the compressed second image, and may sum the pixel difference of the pixels in the compressed second image with the pixels in the lossy first image.

[0122] At block 1214, the computing device (or a component thereof) may generate a combined image at least in part by combining the first image and the second image. In some cases, the combined image is a high dynamic range (HDR) image.

[0123] Fig.131 is a schematic diagram showing another image capture system 1300 that compresses at least one image in an image sensor to reduce bandwidth and power of an HDR image synthesis process according to some aspects of the present disclosure. The image capture system includes an image sensor 1310 that is configured to receive light, convert the light into an electrical signal (the electrical signal is digitized into a raw digital image), and send the raw digital image to a system on chip (SoC) 1330.

[0124] In one illustrative example, the image sensor 1310 includes a pixel array 1312 of photosensors configured to detect light received from a lens and to detect light incident on the photosensors. In one aspect, a Bayer filter may be positioned above the photosensors to enable the photosensors to detect light of a specific wavelength (e.g., color). The detector 1314 is configured to read analog values ​​of the individual photosensors and create a raw digital image in the form of digital pixel values. For example, the detector 1314 may include an ADC that reads individual analog values ​​of light intensity at each photosensor and creates values ​​for individual pixels based on a combination of different photosensors. For example, the individual pixels include at least a photosensor for red light, a photosensor for green light, and a photosensor for blue light. In another illustrative example, the individual pixels may use other color combinations, such as cyan, magenta, and yellow combinations.

[0125] The detector 1314 is configured to read out an image from the pixel array 1312 while light is exposing the pixel array 1312, and to generate a short exposure image, which will be the reference image because it is available first. Since the image sensor 1310 has a limited buffer 1318 capacity, the short exposure image is provided to the downsizer 1316, which reduces the size of the short exposure image. The switch 1324 causes the short exposure image to be provided to the SoC 1330. When the long exposure image is being read out by the detector 1314, the long exposure image is provided to the HDR compressor 1322, and the amplifier 1320 reads the buffer 1308 to retrieve the downscaled reference image and amplify it to the reference image. The HDR compressor 1322 is configured to compress the long exposure image using the difference between the reference image and the target image, which in this illustrative example is the long exposure image. The compressed long exposure image is provided to the switch 1324 and sent to the SoC 1330.

[0126] In some aspects, the switch 1332 of the SoC 1330 is configured to control the flow of images received by the image sensor 1310. In one illustrative example, the switch 1324 and the switch 1332 can be implemented by a virtual channel that does not require any hardware changes and provides a logical control flow of various images. When the switch 1332 receives a short exposure image, the switch 1332 provides the short exposure image to the ISP 1350 and the downsizer 1334. The downsizer 1334 is configured to downscale a reference image (e.g., a short exposure image) and store the downscaled image in a buffer 1336. When the switch 1332 receives a compressed long exposure image, the switch 1332 provides the compressed long exposure image to the HDR decompressor 1340. The HDR decompressor 1340 requests a reference image, and the amplifier 1338 retrieves the downscaled reference image and upscales it to a reference image, and provides the reference image to the HDR decompressor 1340. In some aspects, the HDR decompressor 1340 is configured to restore the long exposure image using pixel differences in the reference image and the compressed long exposure image, as described above with reference to Fig.10 as described.

[0127] In some aspects, ISP 1350 receives the short exposure image and the long exposure image and synthesizes the HDR image as described above. Although this illustrative example uses the short exposure image as the reference image, the long exposure image may also be used as the reference image. In some other aspects, the buffer size of the ISP or image sensor may have sufficient storage capacity to store the image without downscaling, and the HDR image may be omitted. Fig.13 The various devices, functions or modules shown in FIG. For example, a 2 nanometer (nm) process node enables a sufficiently large buffer to be integrated into the image sensor, and scaling down the reference image may not be necessary.

[0128] In some aspects, the process of compressing long exposure images within the image sensor 1310 provides bandwidth savings by reducing the need to send the full image using the MIPI connection. This reduces bandwidth consumption and can achieve higher frame rates using the same connection. In addition, reducing the amount of content sent across the MIPI connection triggers power savings, resulting in improvements to battery and thermal performance.

[0129] Fig.141 is a flowchart illustrating an example of a method 1400 for compressing an image during HDR image synthesis in accordance with certain aspects of the present disclosure. The method 1400 may be performed by a computing device (or a component thereof, such as a chipset) having an image sensor, such as a mobile wireless communication device, a camera, an XR device, a wireless-enabled vehicle, or other computing device. In one illustrative example, the computing system 1500 may be configured to perform all or part of the method 1400. In one illustrative example, an ISP such as the ISP 254 may be configured to perform all or part of the method 1400.

[0130] At box 1402, a computing device (or a component thereof) may read a first image from a pixel array of an image sensor. At box 1404, a computing device (or a component thereof) may read a second image from the pixel array. For example, the first image may have a shorter exposure than the second image. In some examples, the computing device (or a component thereof) may start reading the second image during the reading of the first image. In some cases, the computing device (or a component thereof) may downsample the first image and may store the downsampled first image in a row buffer of the image sensor. Based on a corresponding portion of the second image read out from the pixel array, the computing device (or a component thereof) may enlarge a portion of the downsampled first image and compress the second image based on the enlarged portion.

[0131] At block 1406, the computing device (or a component thereof) may compress the second image based on the first image, e.g., using the techniques described herein. At block 1407, the computing device (or a component thereof) may send the first image and the compressed second image to an image sensor processor (ISP). For example, the ISP may be configured to decompress the second image based on a comparison of the compressed second image and the first image.

[0132] In some examples, the processes described herein (e.g., methods 1200 and 1400, and / or other processes described herein) can be performed by a computing device or apparatus. In one example, methods 1200 and 1400 can be performed by a computer having Fig.15 The computing device (e.g., Figure 2 Executed by the image capture and processing system 200).

[0133] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a television, and / or any other computing device having resource capabilities to perform the methods described herein (including methods 1200 and 1400). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the methods described herein. In some examples, the computing device may include a display, a network interface configured to transmit and / or receive data, any combination thereof, and / or other components. The network interface may be configured to transmit and / or receive IP-based data or other types of data.

[0134] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0135] Method 1200 and method 1400 are illustrated as a logic flow diagram, the operations of which represent a series of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer executable instructions stored on one or more computer-readable storage media, which perform the described operations when executed by one or more processors. Typically, computer executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or may be in parallel to implement the methods.

[0136] Methods 1200 and 1400 and / or other methods or processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, implemented by hardware, or a combination thereof. As mentioned above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes multiple instructions that can be executed by one or more processors. The computer-readable storage medium or machine-readable storage medium may be non-transitory.

[0137] Fig.15 is a schematic diagram showing an example of a system for implementing certain aspects of the technology herein. Specifically, Fig.15 An example of a computing system 1500 is shown, which may be any computing device, for example, making up an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using connection 1505. Connection 1505 may be a physical connection using a bus, or a direct connection into processor 1510 (such as in a chipset architecture). Connection 1505 may also be a virtual connection, a network connection, or a logical connection.

[0138] In some aspects, computing system 1500 is a distributed system, wherein the functionality described in the present disclosure may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a number of such components, each component performing some or all of the functionality described for that component. In some aspects, a component may be a physical or virtual device.

[0139] The example computing system 1500 includes at least one processing unit (CPU or processor) 1510 and connections 1505 coupling various system components including system memory 1515, such as ROM 1520 and RAM 1525, to the processor 1510. The computing system 1500 may include a cache 1512 of high-speed memory directly connected to the processor 1510, close to the processor 1510, or integrated as part of the processor 1510.

[0140] Processor 1510 may include any general purpose processor and hardware or software services configured to control processor 1510 (such as services 1532, 1534, and 1536 stored in storage device 1530), as well as special purpose processors where software instructions are incorporated into the actual processor design. Processor 1510 may essentially be a completely self-contained computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0141] To enable user interaction, the computing system 1500 includes an input device 1545 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1500 may also include an output device 1535, which may be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1500. The computing system 1500 may include a communication interface 1540, which may generally govern and manage user input and system output. The communication interface may use a wired and / or wireless transceiver to perform or facilitate receiving and / or sending wired or wireless communications, including those that utilize the following: an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Proprietary wired ports / plugs, Wireless signal transmission, BLE wireless signal transmission, Wireless signal transmission, RFID wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 WiFi wireless signal transmission, WLAN signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), IR communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 1540 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1500 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US-based GPS, the Russian-based Global Navigation Satellite System (GLONASS), the China-based Beidou Navigation Satellite System (BDS), and the European-based Galileo GNSS system. There is no restriction to operation of any specific hardware arrangement, and therefore the basic functionality herein may be readily substituted for improved hardware or firmware arrangements as they are developed.

[0142] The storage device 1530 may be a non-volatile and / or non-temporary and / or computer-readable memory device, and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a floppy disk, a hard disk, a magnetic tape, a magnetic strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disc-read only memory (CD-ROM) optical disc, a rewritable compact disc (CD) optical disc, a digital video disc (DVD) optical disc, a Blu-ray disc (BDD) optical disc, a holographic optical disc, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a Memory Card, a card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a RAM, a static RAM (SRAM), a dynamic RAM (DRAM), a ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0143] Storage device 1530 may include software services, servers, services, etc., which enable the system to perform functions when processor 1510 executes code defining such software. In some aspects, hardware services that perform specific functions may include software components stored in computer-readable media, which are connected to necessary hardware components (such as processor 1510, connection 1505, output device 1535, etc.) for performing the function. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored and does not include the following: carrier waves and / or transient electronic signals propagated wirelessly or on a wired connection. Examples of non-transitory media may include, but are not limited to: disks or tapes, optical storage media such as CDs or DVDs, flash memory, memory, or memory devices. Computer readable medium can have code and / or machine executable instructions stored thereon, and code and / or machine executable instructions can represent any combination of process, function, subprogram, program, routine, subroutine, module, software package, class or instruction, data structure or program statement. Code segment can be coupled to another code segment or hardware circuit by transmitting and / or receiving information, data, independent variable, parameter or memory content. Can transmit, forward or send information, independent variable, parameter, data etc. via any suitable means including memory sharing, message passing, token passing, network transmission etc.

[0144] In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, a computing device may include a display, one or more network interfaces configured to communicate and / or receive data, any combination thereof, and / or other components. The one or more network interfaces may be configured to communicate and / or receive wired and / or wireless data, including data in accordance with 3G, 4G, 5G, and / or other cellular standards, data in accordance with Wi-Fi (802.11x) standards, data in accordance with Bluetooth standards, and / or other wireless communication protocols. TM Standard data, data according to IP standards, and / or other types of data.

[0145] The components of the computing device may be implemented in circuits. For example, these components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0146] In some aspects, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0147] Specific details are provided in the description above to provide a comprehensive understanding of the aspects and examples provided herein. However, it will be appreciated by those of ordinary skill in the art that these aspects can be put into practice without these specific details. For clarity of explanation, in some cases, the technology herein can be presented as a separate functional block including the following functional blocks, which include equipment, device components, steps or routines in the method embodied in software, or a combination of hardware and software. In addition to the components shown in the figures and / or described herein, additional components can also be used. For example, circuits, systems, networks, processes, and other components can be shown as components in block diagram form, so as not to obscure these aspects in unnecessary details. In other cases, known circuits, processes, algorithms, structures, and techniques can be shown as not having unnecessary details, so as to avoid obscuring these aspects.

[0148] The above text can describe each aspect as a process or method, and the process or method is depicted as a flow chart, a schematic flow chart, a data flow chart, a structure diagram or a block diagram. Although the flow chart can describe the operation as a sequential process, many operations in these operations can be performed in parallel or concurrently. In addition, the order of the operation can be rearranged. The process is terminated after its operation is completed, but there may be additional steps that are not included in the figure. A process (process) can correspond to a method, a function, a procedure (procedure), a subroutine, a subprogram, etc. When a process corresponds to a function, its termination can correspond to the function returning to a calling function or a main function.

[0149] The processes and methods according to the above examples can be implemented using computer executable instructions, which are stored in computer readable media or otherwise available from computer readable media. Such instructions may include, for example, instructions or data that cause a general-purpose computer, a special-purpose computer, or a processing device to execute or otherwise be configured to execute a specific function or a specific set of functions. Parts of the computer resources used can be accessed over a network. Computer executable instructions can be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memory, USB devices provided with non-volatile memory, network storage devices, etc.

[0150] The equipment implementing the process and method according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description language or any combination thereof, and may adopt any of a variety of form factors. When implemented in software, firmware, middleware or microcode, the program code or code segment (e.g., computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc., of laptop computers, smart phones, mobile phones, tablet devices or other small form factors. The functions described herein may also be embodied in peripheral devices or plug-in cards. By further example, such functions may also be implemented on circuit boards between different chips or different processes executed in a single device.

[0151] Instructions, media for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0152] In the foregoing description, various aspects of the present application are described with reference to specific aspects of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative aspects of the present application have been described in detail herein, it should be understood that the inventive concept can be embodied and adopted differently in other ways, and the appended claims are intended to be interpreted as including such variations, except for variations limited by the prior art. The various features and aspects of the above-mentioned applications can be used individually or in combination. Further, without departing from the broader spirit and scope of this specification, the various aspects can be used in any number of environments and applications other than the environments and applications described herein. Therefore, the description and the accompanying drawings are considered to be illustrative rather than restrictive. For the purpose of illustration, the method is described in a specific order. It should be understood that in alternative aspects, the method can be performed in an order different from the described order.

[0153] A person of ordinary skill in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0154] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing an electronic circuit or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other appropriate electronic circuit) to perform the operation, or any combination thereof.

[0155] The phrase "coupled to" refers to any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other appropriate communication interface).

[0156] Claim language or other language that recites "at least one of" in a set and / or "one or more of" in a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language that recites "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language that recites "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" in a set and / or "one or more of" in a set does not limit the set to the items listed in the set. For example, claim language that recites "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0157] Various illustrative logic blocks, modules, circuits and algorithmic steps described in conjunction with aspects disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above around their functions. Whether such functions are implemented as hardware or software depends on specific applications and the design constraints imposed on the entire system. Technicians can implement the described functions in different ways for each specific application, but such implementation decisions should not be interpreted as causing departure from the scope of the application.

[0158] The technology described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such technology may be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device handheld device, or an integrated circuit device with multiple uses (including applications in wireless communication devices handheld devices and other devices). Any features described as modules or components may be implemented together in an integrated logic device, or implemented as separate but interoperable logic devices, respectively. If implemented in software, the technology may be implemented at least in part by a computer-readable data storage medium, which includes a program code, and the program code includes instructions for executing one or more of the above methods when executed. The computer-readable data storage medium may form a part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a RAM (such as a synchronous dynamic random access memory (SDRAM)), a ROM, a non-volatile random access memory (NVRAM), an EEPROM, a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the techniques may be implemented at least in part by a computer-readable communication medium (such as a propagated signal or wave) that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0159] The program code can be executed by a processor, which may include one or more processors, such as one or more DSPs, general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor can be configured to perform any technology described in this disclosure. A general-purpose processor can be a microprocessor, but in an alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the technology described herein.

[0160] Illustrative examples of the present disclosure include:

[0161] Aspect 1. A device for generating one or more images, the device comprising: at least one memory; and at least one processor, which is coupled to the at least one memory and is configured to: obtain a first image captured using an image sensor, the first image being associated with a first exposure; obtain a second image captured using the image sensor, the second image being associated with a second exposure; compress the second image based on a comparison of the second image with the first image; store the compressed second image in a memory; obtain the compressed second image from the memory; decompress the compressed second image based on a difference between the compressed second image and the first image; and generate a combined image at least in part by combining the first image and the second image.

[0162] Aspect 2. The apparatus according to aspect 1, wherein the compression of the second image is lossless.

[0163] Aspect 3. An apparatus according to any one of aspects 1 to 2, wherein the second exposure is less than the first exposure, and wherein the combined image is a high dynamic range (HDR) image.

[0164] Aspect 4. An apparatus according to any one of Aspects 1 to 3, wherein, in order to compress the second image, the at least one processor is configured to: calculate pixel differences between pixels of the first image and pixels of the second image; and determine compressed values ​​for at least a portion of the pixels in the second image based on the pixel differences.

[0165] Aspect 5. An apparatus according to aspect 4, wherein, in order to calculate the pixel difference, the at least one processor is configured to: normalize the first image and the second image based on a minimum pixel value associated with the image sensor.

[0166] Aspect 6. An apparatus according to any one of aspects 4 or 5, wherein the at least one processor is configured to: multiply the second image by an exposure ratio associated with the second image and the first image to generate a modified image.

[0167] Aspect 7. An apparatus according to aspect 6, wherein the at least one processor is configured to: subtract the value of the pixel in the modified image from the maximum value in the first image.

[0168] Aspect 8. An apparatus according to Aspect 7, wherein the value of the pixel of the modified image is greater than or equal to the maximum value, and wherein, based on the value of the pixel of the modified image being greater than or equal to the maximum value, the pixel difference corresponds to the maximum value.

[0169] Aspect 9. An apparatus according to Aspect 7, wherein the value of the pixel of the modified image is less than the maximum value, and wherein, based on the value of the pixel of the modified image being less than the maximum value, the pixel difference corresponds to the difference between the value of the pixel in the modified image and the maximum value.

[0170] Aspect 10. An apparatus according to Aspect 4, wherein, in order to determine the compressed value for at least the portion of the pixels in the second image, the at least one processor is configured to: compare the pixel difference with a threshold; and based on a determination that the pixel difference is less than or equal to the threshold, use the value of the pixel difference as the compressed value for at least the portion of the pixels in the second image.

[0171] Aspect 11. An apparatus according to Aspect 4, wherein, in order to determine the compressed value for at least the portion of the pixels in the second image, the at least one processor is configured to: compare the pixel difference with a threshold; and based on a determination that the pixel difference is greater than the threshold, use the value of at least one pixel in the second image as the compressed value for at least the portion of the pixels in the second image.

[0172] Clause 12. The apparatus according to clause 11, wherein the value of the at least one pixel used as the compressed value comprises the pixel difference.

[0173] Aspect 13. An apparatus according to any one of Aspects 1 to 12, wherein, in order to decompress the compressed second image, the at least one processor is configured to: determine pixel differences of pixels within the compressed second image; and sum the pixel differences of the pixels in the compressed second image with the pixels in the first image.

[0174] Aspect 14. An apparatus according to Aspect 13, wherein, in order to determine the pixel difference, the at least one processor is configured to: add the threshold to the value of the pixel based on whether the value of the pixel within the compressed second image is greater than the threshold.

[0175] Aspect 15. The apparatus according to any one of aspects 1 to 14, wherein the at least one processor is configured to: cause the first image to be stored in a cache.

[0176] Aspect 16. An apparatus according to any one of Aspects 1 to 15, wherein the at least one processor is configured to: downsample the first image; store the downsampled first image in a cache; and upsample the downsampled first image into a lossy first image.

[0177] Aspect 17. An apparatus according to Aspect 16, wherein, in order to compress the second image, the at least one processor is configured to: calculate pixel differences for pixels between the lossy first image and the second image; and determine compressed values ​​for at least a portion of the pixels in the second image based on the pixel differences.

[0178] Aspect 18. An apparatus according to any one of Aspects 16 or 17, wherein, in order to decompress the compressed second image, the at least one processor is configured to: determine pixel differences of pixels within the compressed second image; and sum the pixel differences of the pixels in the compressed second image with the pixels in the lossy first image.

[0179] Aspect 19. A method for generating an image, the method comprising: obtaining a first image captured using an image sensor, the first image being associated with a first exposure; obtaining a second image captured using the image sensor, the second image being associated with a second exposure; compressing the second image based on a comparison of the second image with the first image; storing the compressed second image in a memory; obtaining the compressed second image from the memory; decompressing the compressed second image based on a difference between the compressed second image and the first image; and generating a combined image at least in part by combining the first image and the second image.

[0180] Clause 20. The method according to clause 19, wherein the compression of the second image is lossless.

[0181] Clause 21. A method according to any one of clauses 19 to 20, wherein the second exposure is less than the first exposure, and wherein the combined image is a high dynamic range (HDR) image.

[0182] Aspect 22. A method according to any one of Aspects 19 to 21, wherein compressing the second image includes: calculating pixel differences between pixels of the first image and pixels of the second image; and determining compressed values ​​for at least a portion of the pixels in the second image based on the pixel differences.

[0183] Clause 23. The method according to clause 22, wherein calculating the pixel difference comprises normalizing the first image and the second image based on a minimum pixel value associated with the image sensor.

[0184] Clause 24. The method according to any one of clauses 22 or 23, further comprising: multiplying the second image by an exposure ratio associated with the second image and the first image to generate a modified image.

[0185] Clause 25. The method according to clause 24, further comprising: subtracting the value of the pixel in the modified image from the maximum value in the first image.

[0186] Aspect 26. A method according to Aspect 25, wherein the value of the pixel of the modified image is greater than or equal to the maximum value, and wherein, based on the value of the pixel of the modified image being greater than or equal to the maximum value, the pixel difference corresponds to the maximum value.

[0187] Aspect 27. A method according to Aspect 25, wherein the value of the pixel of the modified image is less than the maximum value, and wherein, based on the value of the pixel of the modified image being less than the maximum value, the pixel difference corresponds to the difference between the value of the pixel in the modified image and the maximum value.

[0188] Aspect 28. A method according to Aspect 22, wherein determining the compressed value for at least the portion of the pixels in the second image includes: comparing the pixel difference with a threshold; and based on a determination that the pixel difference is less than or equal to the threshold, using the value of the pixel difference as the compressed value for at least the portion of the pixels in the second image.

[0189] Aspect 29. A method according to Aspect 22, wherein determining the compressed value for at least the portion of the pixels in the second image comprises: comparing the pixel difference with a threshold; and based on a determination that the pixel difference is greater than the threshold, using the value of at least one pixel in the second image as the compressed value for at least the portion of the pixels in the second image.

[0190] Clause 30. The method according to clause 29, wherein the value of the at least one pixel used as the compressed value comprises the pixel difference.

[0191] Aspect 31. A method according to any one of Aspects 19 to 30, wherein decompressing the compressed second image comprises: determining pixel differences of pixels within the compressed second image; and summing the pixel differences of the pixels in the compressed second image with the pixels in the first image.

[0192] Aspect 32. The method according to aspect 31, wherein determining the pixel difference comprises: adding a threshold to the value of the pixel based on whether the value of the pixel within the compressed second image is greater than a threshold.

[0193] Clause 33. The method according to any one of clauses 19 to 32, further comprising: storing the first image in a cache.

[0194] Aspect 34. The method according to any one of Aspects 19 to 33 further includes: downsampling the first image; storing the downsampled first image in a cache; and upsampling the downsampled first image into a lossy first image.

[0195] Aspect 35. A method according to Aspect 34, wherein compressing the second image includes: calculating pixel differences for pixels between the lossy first image and the second image; and determining compressed values ​​for at least a portion of the pixels in the second image based on the pixel differences.

[0196] Aspect 36. A method according to aspect 34 or 35, wherein decompressing the compressed second image includes: determining pixel differences of pixels within the compressed second image; and summing the pixel differences of the pixels in the compressed second image with the pixels in the lossy first image.

[0197] Aspect 37. A device for processing one or more images, the device comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: read a first image from a pixel array of an image sensor; read a second image from the pixel array; compress the second image based on the first image; and send the first image and the compressed second image to an image sensor processor (ISP).

[0198] Aspect 38. An apparatus according to Aspect 37, wherein the at least one processor is configured to: downsample the first image and store the downsampled first image in a row buffer of the image sensor; and enlarge a portion of the downsampled first image based on a corresponding portion of the second image read out from the pixel array, and compress the second image based on the enlarged portion.

[0199] Aspect 39. An apparatus according to any one of Aspects 37 to 38, wherein the at least one processor is configured to: start reading the second image during the reading of the first image, and wherein the first image has a shorter exposure than the second image.

[0200] Aspect 40. An apparatus according to any one of aspects 37 to 39, wherein the ISP is configured to decompress the second image based on a comparison of the compressed second image and the first image.

[0201] Aspect 41. A method for processing one or more images, the method comprising: reading a first image from a pixel array of an image sensor; reading a second image from the pixel array; compressing the second image based on the first image; and sending the first image and the compressed second image to an image sensor processor (ISP).

[0202] Aspect 42. The method according to Aspect 41 further includes: downsampling the first image and storing the downsampled first image in a row buffer of the image sensor; and based on a corresponding portion of the second image read out from the pixel array, enlarging a portion of the downsampled first image, and compressing the second image based on the enlarged portion.

[0203] Clause 43. A method according to any one of clauses 41 to 42, wherein the reading of the second image starts during the reading of the first image, and wherein the first image has a shorter exposure than the second image.

[0204] Clause 44. A method according to any one of clauses 41 to 43, wherein the ISP is configured to decompress the second image based on a comparison of the compressed second image and the first image.

[0205] Aspect 45. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform the operations described in any one of aspects 19 to 36.

[0206] Aspect 46. An apparatus for generating one or more images, the apparatus comprising one or more means for performing the operations according to any one of aspects 19 to 36.

[0207] Aspect 47. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform the operations described in any one of aspects 41 to 44.

[0208] Aspect 48: An apparatus for processing one or more images, the apparatus comprising one or more units for performing the operations according to any one of aspects 41 to 44.

Claims

1. A device for generating one or more images, the device comprising: at least one memory; as well as at least one processor coupled to the at least one memory and configured to: obtaining a first image captured using an image sensor, the first image being associated with a first exposure; obtaining a second image captured using the image sensor, the second image being associated with a second exposure; compressing the second image based on a comparison of the second image with the first image; storing the compressed second image in a memory; obtaining the compressed second image from the memory; decompressing the compressed second image based on a difference between the compressed second image and the first image; as well as A combined image is generated at least in part by combining the first image and the second image.

2. The device according to claim 1, wherein: The compression of the second image is lossless.

3. The device according to claim 1, wherein: The second exposure is less than the first exposure, and wherein the combined image is a high dynamic range (HDR) image.

4. The device according to claim 1, wherein: To compress the second image, the at least one processor is configured to: calculating pixel differences between pixels of the first image and pixels of the second image; as well as Compressed values ​​for at least a portion of pixels in the second image are determined based on the pixel differences.

5. The device according to claim 4, wherein: To calculate the pixel difference, the at least one processor is configured to: The first image and the second image are normalized based on a minimum pixel value associated with the image sensor.

6. The device according to claim 4, wherein: The at least one processor is configured to: The second image is multiplied by an exposure ratio associated with the second image and the first image to generate a modified image.

7. The device according to claim 6, wherein: The at least one processor is configured to: The value of a pixel in the modified image is subtracted from the maximum value in the first image.

8. The device according to claim 7, wherein: The value of the pixel of the modified image is greater than or equal to the maximum value, and wherein the pixel difference corresponds to the maximum value based on the value of the pixel of the modified image being greater than or equal to the maximum value.

9. The device according to claim 7, wherein: The value of the pixel of the modified image is less than the maximum value, and wherein, based on the value of the pixel of the modified image being less than the maximum value, the pixel difference corresponds to the difference between the value of the pixel in the modified image and the maximum value.

10. The device according to claim 4, wherein: To determine the compressed value for at least the portion of pixels in the second image, the at least one processor is configured to: comparing the pixel difference with a threshold; as well as Based on a determination that the pixel difference is less than or equal to the threshold, a value of the pixel difference is used as a compressed value for at least the portion of pixels in the second image.

11. The device according to claim 4, wherein: To determine the compressed value for at least the portion of pixels in the second image, the at least one processor is configured to: comparing the pixel difference with a threshold; as well as Based on a determination that the pixel difference is greater than the threshold, a value of at least one pixel in the second image is used as a compressed value for at least the portion of pixels in the second image.

12. The device according to claim 11, wherein The value of the at least one pixel used as the compressed value comprises the pixel difference.

13. The device according to claim 1, wherein: To decompress the compressed second image, the at least one processor is configured to: determining pixel differences of pixels within the compressed second image; as well as The pixel differences of pixels in the compressed second image are summed with pixels in the first image.

14. The device according to claim 13, wherein: To determine the pixel difference, the at least one processor is configured to: A threshold is added to the value of a pixel within the compressed second image based on whether the value of the pixel is greater than a threshold.

15. The device according to claim 1, wherein: The at least one processor is configured to: The first image is caused to be stored in a cache.

16. The device according to claim 1, wherein The at least one processor is configured to: downsampling the first image; causing the downsampled first image to be stored in a cache; and The downsampled first image is upsampled into a lossy first image.

17. The device according to claim 16, wherein: To compress the second image, the at least one processor is configured to: calculating pixel differences for pixels between the impaired first image and the second image; as well as Compressed values ​​for at least a portion of pixels in the second image are determined based on the pixel differences.

18. The device according to claim 16, wherein: To decompress the compressed second image, the at least one processor is configured to: determining pixel differences of pixels within the compressed second image; as well as The pixel differences of pixels in the compressed second image are summed with pixels in the lossy first image.

19. A method for generating an image, the method comprising: obtaining a first image captured using an image sensor, the first image being associated with a first exposure; obtaining a second image captured using the image sensor, the second image being associated with a second exposure; compressing the second image based on a comparison of the second image with the first image; storing the compressed second image in a memory; obtaining the compressed second image from the memory; decompressing the compressed second image based on a difference between the compressed second image and the first image; as well as A combined image is generated at least in part by combining the first image and the second image.

20. The method according to claim 19, wherein: The second exposure is less than the first exposure, and wherein the combined image is a high dynamic range (HDR) image.

21. The method according to claim 19, wherein: Compressing the second image comprises: calculating pixel differences between pixels of the first image and pixels of the second image; and Compressed values ​​for at least a portion of pixels in the second image are determined based on the pixel differences.

22. The method according to claim 21, further comprising: multiplying the second image by an exposure ratio associated with the second image and the first image to generate a modified image; as well as The value of a pixel in the modified image is subtracted from the maximum value in the first image.

23. The method according to claim 21, wherein: Determining the compressed value for at least the portion of pixels in the second image comprises: comparing the pixel difference to a threshold; and Based on a determination that the pixel difference is less than or equal to the threshold, a value of the pixel difference is used as a compressed value for at least the portion of pixels in the second image.

24. The method according to claim 21, wherein: Determining the compressed value for at least the portion of pixels in the second image comprises: comparing the pixel difference to a threshold; and Based on a determination that the pixel difference is greater than the threshold, a value of at least one pixel in the second image is used as a compressed value for at least the portion of pixels in the second image.

25. The method according to claim 24, wherein: The value of the at least one pixel used as the compressed value comprises the pixel difference.

26. The method of claim 19, wherein: Decompressing the compressed second image comprises: determining pixel differences for pixels within the compressed second image; and The pixel differences of pixels in the compressed second image are summed with pixels in the first image.

27. The method according to claim 26, wherein: Determining the pixel difference comprises: A threshold is added to the value of a pixel within the compressed second image based on whether the value of the pixel is greater than a threshold.

28. The method of claim 19, further comprising: downsampling the first image; storing the downsampled first image in a cache; as well as The downsampled first image is upsampled into a lossy first image.

29. The method according to claim 28, wherein: Compressing the second image comprises: calculating pixel differences for pixels between the impaired first image and the second image; and Compressed values ​​for at least a portion of pixels in the second image are determined based on the pixel differences.

30. The method of claim 28, wherein: Decompressing the compressed second image comprises: determining pixel differences for pixels within the compressed second image; and The pixel differences of pixels in the compressed second image are added to pixels in the lossy first image.

Images