High dynamic range (HDR) image generation with multi-domain motion correction

Abstract

Systems, devices, processes, and computer-readable media for capturing images having topics at different depths of field are disclosed. A method of processing image data includes obtaining a first image captured using an image sensor, the first image associated with a first exposure; obtaining a second image captured using the image sensor, the second image associated with a second exposure longer than the first exposure; modifying a first region of the first image based on the first transform and modifying a second region of the first image based on the second transform to generate a modified first image; and generating a merged image at least in part by merging the modified first image and the second image.

Description

Technical Field

[0001] In some examples, systems and techniques are described for generating high dynamic range (HDR) images using motion correction in different domains. Background Art

[0002] A camera is a device that receives light and captures image frames (such as still images or video frames) using an image sensor. A camera may include a processor, such as an image signal processor (ISP), that can receive one or more image frames and process the one or more image frames. For example, a raw image frame captured by a camera sensor may be processed by an ISP to generate a final image. A camera may be configured with various image capture and image processing settings to change the appearance of an image. Some camera settings are determined and applied before or during the taking of a photo, such as ISO, exposure time, aperture size, f / stop, shutter speed, focus, and gain. Other camera settings may configure post-processing of a photo, such as changes to contrast, brightness, saturation, sharpness, tones, curves, or color.

[0003] The camera can be configured with various image capture and image processing settings. Applying different settings may produce frames or images with different appearances. Some camera settings are determined and applied before or during the capture of a photo, such as ISO, exposure time (also called exposure duration), aperture size, f / stop, shutter speed, focus, and gain. Other camera settings can configure post-processing of the photo, such as changes to contrast, brightness, saturation, sharpness, levels, curves, or color. Summary of the invention

[0004] In some examples, systems and techniques for generating high dynamic range (HDR) images using multi-domain motion correction are described. These systems and techniques can improve the image quality of HDR images, such as by reducing noise or other defects caused by motion in the HDR images (e.g., reducing ghosting).

[0005] In some examples, systems and techniques for generating HDR images using multi-domain motion correction are described. Systems, devices, methods, and computer-readable media for processing one or more images are disclosed. According to at least one example, a method for processing one or more images is provided. The method includes: obtaining a first image captured using an image sensor, the first image being associated with a first exposure; obtaining a second image captured using the image sensor, the second image being associated with a second exposure longer than the first exposure; modifying a first region of the first image based on a first transform, and modifying a second region of the first image based on a second transform to generate a modified first image; and generating a merged image at least in part by merging the modified first image and the second image.

[0006] In another example, an apparatus for processing one or more images is provided, the apparatus comprising at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to: obtain a first image captured using an image sensor, the first image being associated with a first exposure; obtain a second image captured using the image sensor, the second image being associated with a second exposure longer than the first exposure; modify a first region of the first image based on a first transformation, and modify a second region of the first image based on a second transformation to generate a modified first image; and generate a merged image at least in part by merging the modified first image and the second image.

[0007] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: obtain a first image captured using an image sensor, the first image being associated with a first exposure; obtain a second image captured using the image sensor, the second image being associated with a second exposure that is longer than the first exposure; modify a first region of the first image based on a first transformation, and modify a second region of the first image based on a second transformation to generate a modified first image; and generate a merged image at least in part by merging the modified first image and the second image.

[0008] In another example, an apparatus for processing one or more images is provided. The apparatus includes: means for obtaining a first image captured using an image sensor, the first image being associated with a first exposure; means for obtaining a second image captured using the image sensor, the second image being associated with a second exposure longer than the first exposure; means for modifying a first region of the first image based on a first transformation and a second region of the first image based on a second transformation to generate a modified first image; and means for generating a merged image at least in part by merging the modified first image and the second image.

[0009] In some aspects, the image sensor is oriented in the same direction as a display used to display a preview image captured by the image sensor.

[0010] In some aspects, the first region is associated with an object at a first depth in a scene relative to the image sensor, and the second region includes a background region at a second depth in the scene relative to the image sensor.

[0011] In some aspects, one or more of the methods, apparatus, and computer-readable media described above further include: generating a first matrix for performing the first transformation; and generating a second matrix for performing the second transformation.

[0012] In some aspects, the second matrix is ​​generated based on movement detected by a motion sensor between a first time when the first image is captured and a second time when the second image is captured.

[0013] In some aspects, the motion sensor comprises a gyroscope sensor, and wherein the second transform comprises a rotational transform.

[0014] In some aspects, one or more of the above methods, apparatuses, and computer-readable media further include: extracting first feature points from the first image; and extracting second feature points from the second image.

[0015] In some aspects, one or more of the above methods, apparatus, and computer-readable media further include increasing the brightness of the first image based on an exposure ratio difference between the first image and the second image.

[0016] In some aspects, one or more of the above methods, apparatus, and computer-readable media further include: detecting an object in the second image; and determining a bounding area associated with the location of the object in the second image.

[0017] In some aspects, one or more of the above methods, devices and computer-readable media also include: identifying a subset of the first feature points within the boundary area; identifying a subset of the second feature points within the boundary area; and generating the first matrix based on the subset of the first feature points and the subset of the second feature points.

[0018] In some aspects, a blending transformation matrix for modifying the first region of the first image and the second region of the first image is based on the first matrix and the second matrix.

[0019] In some aspects, one or more of the above methods, devices, and computer-readable media also include: adding values ​​from the first matrix corresponding to at least the first region to the hybrid transformation matrix; and adding values ​​from the second matrix corresponding to at least the second region to the hybrid transformation matrix.

[0020] In some aspects, one or more of the above-mentioned methods, devices, and computer-readable media also include: determining a transition area between the first area and the second area based on the size of a boundary area associated with the position of an object in at least one of the first image or the second image; determining a value associated with the transition area based on the representation of the first matrix and the second matrix; and adding the value associated with the transition area to the mixing transformation matrix.

[0021] In some aspects, the representation of the first matrix and the second matrix comprises a weighted average of the first matrix and the first matrix.

[0022] In some aspects, the representation of the first matrix and the second matrix is ​​based on a proportional distance from an inner edge of the transition region to an outer edge of the transition region.

[0023] In some aspects, the first transformation comprises a translation matrix associated with movement of the image sensor during the obtaining the first image and the obtaining the second image.

[0024] In some aspects, the merged image is a HDR image.

[0025] In some aspects, the apparatus is, is part of, and / or includes a wearable device, an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head mounted device (HMD) device, a wireless communication device, a mobile device (e.g., a mobile phone and / or a mobile handset and / or a so-called "smartphone" or other mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or component of a vehicle, other devices, or a combination thereof. In some aspects, the apparatus includes one or more cameras for capturing one or more images. In some aspects, the apparatus also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyroscopes, one or more accelerometers, any combination thereof, and / or other sensors).

[0026] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The claimed subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0027] The foregoing and other features and aspects will become more apparent by reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Illustrative aspects of the present application are described in detail below with reference to the accompanying drawings:

[0029] Figure 1A , Figure 1B and Figure 1C is a diagram illustrating an example configuration of an image sensor of an image capture device according to aspects of the present disclosure.

[0030] Figure 2 is a block diagram illustrating the architecture of an image capture and processing device in accordance with aspects of the present disclosure.

[0031] Figure 3 is a block diagram illustrating an example of an image capture system in accordance with aspects of the present disclosure.

[0032] Figure 4 is a diagram illustrating generation of a fused frame from a short exposure frame and a long exposure frame according to aspects of the present disclosure.

[0033] Figure 5 is a diagram illustrating long exposure flow and short exposure flow from an image sensor in accordance with certain aspects of the present disclosure.

[0034] Figure 6 is a diagram illustrating an example of online fusion of one or more short exposure frames and one or more long exposure frames according to aspects of the present disclosure.

[0035] Fig. 7A is a diagram illustrating long exposure images and short exposure images captured by an image capture system undergoing displacement in multiple domains according to some aspects.

[0036] Figure 7B FIG. 1 shows a graph of a 3D image of a 3D image after rotation correction according to some aspects of the present disclosure. Fig. 7A Long exposure images and short exposure images in .

[0037] Figure 8 is an example high dynamic range (HDR) image generated by an HDR fusion system after correcting for rotational movement according to some aspects of the present disclosure.

[0038] Fig. 9 A diagram is shown illustrating an example image processing system for synthesizing HDR images using multi-domain motion correction in accordance with some aspects of the present disclosure.

[0039] Figures 10A-10D is a conceptual diagram illustrating various matrices generated by an HDR fusion system to correct for spatial and rotational alignment in accordance with some aspects of the present disclosure.

[0040] Fig.11 is a flow chart illustrating an example of a method for aligning an image using rotation information using a gyro sensor in accordance with certain aspects of the present disclosure.

[0041] Fig.12 is a conceptual diagram of keypoint detection for identifying displacement of an image capture system in accordance with certain aspects of the present disclosure.

[0042] Fig.13 is a flow chart illustrating an example method for synthesizing HDR images using multi-domain motion correction according to aspects of the present disclosure.

[0043] Fig.14 is an illustrative example of a deep learning neural network that can be used to implement machine learning based alignment prediction in accordance with aspects of the present disclosure.

[0044] Fig.15 is an illustrative example of a convolutional neural network (CNN) in accordance with aspects of the present disclosure.

[0045] Fig.16 is a diagram illustrating an example of a system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0046] Some aspects of the present disclosure are provided below. Some of these aspects can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, for the purpose of explanation, specific details are given to provide a thorough understanding of various aspects of the application. However, it is obvious that various aspects can be practiced without these specific details. These drawings and descriptions are not restrictive.

[0047] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. Rather, the following description of the example aspects will provide an enabling description for implementing the example aspects to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0048] The following description provides only exemplary aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. Rather, the following description of exemplary aspects will provide an enabling description for implementing an aspect of the present disclosure to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0049] The terms "exemplary" and / or "example" are used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" and / or "example" is not necessarily to be construed as advantageous over other aspects. Likewise, the term "aspects of the disclosure" does not require that all aspects of the disclosure include the discussed feature, advantage, or mode of operation.

[0050] A camera is a device that receives light and uses an image sensor to capture an image frame (such as a still image or a video frame). The terms "image", "image frame" and "frame" are used interchangeably herein. A camera can be configured with various image capture and image processing settings. Different settings produce images with different appearances. Some camera settings are determined and applied before or during the capture of one or more image frames, such as ISO, exposure time, aperture size, f / stop, shutter speed, focus and gain. For example, settings or parameters can be applied to an image sensor for capturing one or more image frames. Other camera settings can configure post-processing of one or more image frames, such as changes to contrast, brightness, saturation, sharpness, color scale, curves or colors. For example, settings or parameters can be applied to a processor (e.g., an image signal processor (ISP)) for processing one or more image frames captured by an image sensor.

[0051] The dynamic range of a digital imaging device (such as a digital camera) is the ratio between the maximum amount of light that the device can capture without light saturation and the lowest amount of light that the device can accurately measure and distinguish from inherent image noise (electrical noise, thermal noise, etc.). Traditionally, digital cameras can only capture a small portion of the natural illumination range of real-world scenes. For example, the dynamic range of a scene may be 100,000:1, while the dynamic range of a digital camera's image sensor may be 100:1. When the dynamic range of a scene exceeds the dynamic range of the sensor, detail is lost in the areas of highest light levels and / or in the areas of lowest light levels.

[0052] An imaging device may generate a high dynamic range (HDR) image by merging multiple images captured at different exposure settings. For example, an imaging device may generate an HDR image by merging a short exposure image captured at a short exposure time, a medium exposure image captured at a medium exposure time longer than the short exposure time, and a long exposure image captured at a long exposure time longer than the medium exposure time. Because short exposure images are generally dark, they generally retain most of the details in the highlights (bright areas) of the scene being photographed. Medium exposure images and long exposure images are generally brighter than short exposure images, and may be overexposed (e.g., too bright to discern details) in the highlights (bright areas) of the scene. Because long exposure images generally include bright parts, they may retain details in the shadows (dark areas) of the scene being photographed. Medium exposure images and short exposure images are generally darker than long exposure images, and may be underexposed (e.g., too dark to discern details) in the shadows (dark areas) of the scene, making their depiction of the shadows too dark to observe details. To generate an HDR image, the imaging device may, for example, use portions of the short exposure image to depict highlights (bright areas) of the captured scene, use portions of the long exposure image to depict shadows (dark areas) of the scene, and use portions of the medium exposure image to depict other areas of the scene (other than highlights and shadows).

[0053] In some cases, an image capture system (e.g., a mobile device) may include a camera for capturing a forward-facing image relative to a display to allow a user to capture a selfie and participate in a video call. For example, in a selfie, the user is capturing an image of an object (e.g., a landmark) in the foreground (the user's face) and the background. To capture an image, the user moves the image capture system away from their body while previewing the image, and enters a command such as pressing a physical button or a virtual button on the display of the image capture system. In order to capture a high-quality HDR image with low noise, the user cannot move the image capture system because the image capture system is capturing multiple images (e.g., long exposure images and short exposure images) at different times. The user extends the image capture system away from their body (e.g., the user's face) and frames an object of interest in the background. When the user has extended the image capture system away from their body, the user will experience a conventional tremor that displaces the image capture system, and noise and other artifacts may be generated within the HDR image. In addition, the user enters a command by pressing a button or providing a screen input that applies force to the image capture system, and the image capture system can move based on the input. For example, the image capture system may include a physical button on the side, and when the user's hand is holding the image capture system and also clicking the physical button, the input may inadvertently cause rotational movement of the image capture system when capturing the HDR image. Some displacement of the image capture system (e.g., movement in Cartesian coordinates) may also be caused by intrinsic effects (e.g., tremor) or extrinsic effects (e.g., wind). Rotation and displacement may also be further affected by longer exposure times because the camera's forward-facing lens is smaller and inherently limits the amount of light, which increases the exposure time required for short and long exposure images, and increases the likelihood of some aberrational movement due to intrinsic or extrinsic motion.

[0054] The long exposure image and the short exposure image may not be aligned based on the rotational movement of the image capture system and the displacement of the image capture system. The rotation can be recorded by a gyroscope sensor, and the long exposure image and the short exposure image can be compensated based on the detected motion over time. The HDR image created by the rotation correction of the long exposure image and the short exposure image will be well aligned with the object at a farther depth (e.g., at a long distance from the image capture system), but the close object (e.g., the user's face) may not be aligned. The HDR image can also be corrected based on detecting key points within the image. An example method for detecting key points is to detect edges on the image or other strong features of the image, such as the corners of the object. HDR image correction by detecting key points is beneficial to stronger features appearing at a closer depth (e.g., closer to the camera), but there is a potential misalignment between close objects (e.g., the user's face) and objects at a farther depth (e.g., landmark objects in the background).

[0055] In some aspects, systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively referred to herein as "systems and techniques") for generating HDR images using a combination of rotation and translation corrections are described. For example, an imaging system can identify a foreground object, such as a face of a user in a selfie. The imaging system can use the foreground object to determine the displacement of the imaging system, such as identifying a translation matrix of the movement. In one illustrative example, the imaging system can identify key points associated with the foreground object and identify the displacement between the short exposure image and the long exposure image. The imaging system can use a sensor, such as a gyroscope sensor, to identify a rotation associated with the imaging system.

[0056] Based on the displacement and rotation of the imaging system, the imaging system can perform rotation correction on the background object and perform spatial correction on the foreground object. In one illustrative aspect, the imaging system is configured to determine a boundary area between the foreground object and other objects (e.g., background objects), and correct the boundary area based on an interpolation of the rotation correction and the spatial correction.

[0057] Additional details and aspects of the disclosure will be described in more detail below with reference to the accompanying drawings.

[0058] The image sensor includes one or more photodiode arrays or other light-sensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in the image produced by the image sensor. In some cases, different photodiodes can be covered by different color filters of the color filter array and can therefore measure light that matches the color of the color filter covering the photodiode.

[0059] Various color filter arrays may be used, including a Bayer color filter array, a quad color filter array (also referred to as a quad Bayer filter or QCFA), and / or other color filter arrays. An example of a Bayer color filter array 100 is shown in FIG. Figure 1A As shown in the figure, the Bayer color filter array 100 includes a repeating pattern of red color filters, blue color filters, and green color filters. Figure 1B As shown, QCFA 110 includes a 2x2 (or "quad") pattern of color filters, including a 2x2 pattern of red (R) color filters, a pair of 2x2 patterns of green (G) color filters, and a 2x2 pattern of blue (B) color filters. For the entire photodiode array of a given image sensor, the Figure 1B100. Using the QCFA 110 or Bayer color filter array 100, each pixel of the image is generated based on red light data from at least one photodiode covered in the red color filter of the color filter array, blue light data from at least one photodiode covered in the blue color filter of the color filter array, and green light data from at least one photodiode covered in the green color filter of the color filter array. Other types of color filter arrays may use yellow color filters, magenta color filters, and / or cyan (also known as "emerald") color filters instead of, or in addition to, red color filters, blue color filters, and / or green color filters. Different photodiodes throughout the pixel array may have different spectral sensitivity curves, thereby responding to different wavelengths of light. Monochrome image sensors may also lack color filters and, therefore, lack color depth.

[0060] In some cases, a subset of adjacent photodiodes (e.g., when using Figure 1B When the QCFA 110 is shown, the 2x2 photodiodes (2x2 photodiodes) can measure light of the same color for substantially the same area of ​​the scene. For example, when the photodiodes included in each photodiode subgroup are physically close, the light incident on each photodiode of the subgroup can originate from substantially the same location in the scene (e.g., a portion of leaves on a tree, a small portion of the sky, etc.).

[0061] In some examples, the brightness range of light from a scene may significantly exceed the brightness levels that an image sensor can capture. For example, a digital single-lens reflex (DSLR) camera may be able to capture light from a scene with a contrast ratio of 1:30,000, while the brightness levels of an HDR scene may exceed a contrast ratio of 1:1,000,000.

[0062] In some cases, an HDR sensor can be used to enhance the contrast of an image captured by an image capture device. In some examples, an HDR sensor can be used to obtain multiple exposures within an image or frame, where such multiple exposures can include short (e.g., 5 ms) exposure times and long (e.g., 15 ms or longer) exposure times. As used herein, a long exposure time generally refers to any exposure time that is longer than a short exposure time.

[0063] In some implementations, the HDR sensor may be able to separate individual photodiodes within a photodiode subgroup (e.g., four individual R photodiodes, four individual B photodiodes, and Figure 1BThe four individual G photodiodes of each of the two 2x2 G tiles in the illustrated QCFA 110 are configured with different exposure settings. A collection of photodiodes with matching exposure settings is also referred to herein as a photodiode exposure group. Figure 1C FIG. 4 shows a portion of an image sensor array having a QCFA filter configured with four different photodiode exposure groups 1 to 4. Figure 1C As shown in the example photodiode exposure group array 120 in FIG. 1 , each 2×2 tile may include photodiodes from each of the different photodiode exposure groups for a particular image sensor. Figure 1C Four groupings are shown in the specific grouping in , but those skilled in the art will recognize that different numbers of photodiode exposure groups, different arrangements of photodiode exposure groups within a subgroup, and any combination thereof may be used without departing from the scope of the present disclosure.

[0064] As about Figure 1C As described, in some HDR image sensor implementations, the exposure settings corresponding to different photodiode exposure groups may include different exposure times (also referred to as exposure lengths), such as short exposure, medium exposure, and long exposure. In some cases, different images of a scene associated with different exposure settings may be formed from light captured by the photodiodes of each photodiode exposure group. For example, a first image may be formed from light captured by the photodiodes of photodiode exposure group 1, a second image may be formed from light captured by the photodiodes of photodiode exposure group 2, a third image may be formed from light captured by the photodiodes of photodiode exposure group 3, and a fourth image may be formed from light captured by the photodiodes of photodiode exposure group 4. Based on the differences in the exposure settings corresponding to each group, the brightness of the objects in the scene captured by the image sensor may be different in each image. For example, a well-lit object captured by a photodiode with a long exposure setting may appear saturated (e.g., completely white). In some cases, the image processor may select between pixels of images corresponding to different exposure settings to form a merged image.

[0065] In one illustrative example, the first image corresponds to a short exposure time (also referred to as a short exposure image), the second image corresponds to a medium exposure time (also referred to as a medium exposure image), and the third image and the fourth image correspond to long exposure times (also referred to as long exposure images). In such an example, pixels of the merged image corresponding to a portion of the scene with low illumination (e.g., a portion of the scene in shadow) may be selected from the long exposure image (e.g., the third image or the fourth image). Similarly, pixels of the merged image corresponding to a portion of the scene with high illumination (e.g., a portion of the scene in direct sunlight) may be selected from the short exposure image (e.g., the first image).

[0066] In some cases, the image sensor may also utilize a photodiode exposure group to capture an object in motion without blurring. The length of the exposure time of the photodiode group may correspond to the distance that an object in the scene moves during the exposure time. If light from an object in motion is captured by a photodiode corresponding to a plurality of image pixels during the exposure time, the object in motion may appear blurred across the plurality of image pixels (also referred to as motion blur). In some implementations, motion blur may be reduced by configuring one or more photodiode groups with a short exposure time. In some implementations, an image capture device (e.g., a camera) may determine the amount of local motion (e.g., a motion gradient) within a scene by comparing the position of an object between two consecutive captured images. For example, motion may be detected in a preview image captured by an image capture device to provide a preview function to a user on a display. In some cases, a machine learning model may be trained to detect local motion between consecutive images.

[0067] Various aspects of the technology described herein will be discussed below with reference to the accompanying drawings. Figure 2 2 is a block diagram illustrating the architecture of image capture and processing system 200. Image capture and processing system 200 includes various components used to capture and process images of a scene (e.g., images of scene 210). Image capture and processing system 200 can capture independent images (or photos) and / or can capture a video including multiple images (or video frames) in a specific sequence. In some cases, lens 215 and image sensor 230 can be associated with an optical axis. In an illustrative example, the photosensitive area of ​​lens 215 and image sensor 230 (e.g., a photodiode) can be centered on the optical axis. Lens 215 of image capture and processing system 200 faces scene 210 and receives light from scene 210. Lens 215 bends incident light from the scene toward image sensor 230. Light received by lens 215 passes through an aperture. In some cases, the aperture (e.g., the aperture size) is controlled by one or more control mechanisms 220 and received by image sensor 230. In some cases, the aperture can have a fixed size.

[0068] The one or more control mechanisms 220 may control exposure, focus, and / or zoom based on information from the image sensor 230 and / or based on information from the image processor 250. The one or more control mechanisms 220 may include a plurality of mechanisms and components; for example, the control mechanisms 220 may include one or more exposure control mechanisms 225A, one or more focus control mechanisms 225B, and / or one or more zoom control mechanisms 225C. The one or more control mechanisms 220 may also include additional control mechanisms other than those shown, such as control mechanisms for controlling analog gain, flash, HDR, depth of field, and / or other image capture characteristics.

[0069] A focus control mechanism 225B in the control mechanism 220 may obtain a focus setting. In some examples, the focus control mechanism 225B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 225B may adjust the position of the lens 215 relative to the position of the image sensor 230. For example, based on the focus setting, the focus control mechanism 225B may move the lens 215 closer to the image sensor 230 or further away from the image sensor 230 by driving a motor or servo mechanism (or other lens mechanism), thereby adjusting the focus. In some cases, additional lenses may be included in the image capture and processing system 200, such as one or more microlenses on each photodiode of the image sensor 230, each of which bends light received from the lens 215 toward the corresponding photodiode before the light reaches the photodiode. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The focus setting may be determined using the control mechanism 220, the image sensor 230, and / or the image processor 250. The focus setting may be referred to as an image capture setting and / or an image processing setting. In some cases, lens 215 may be fixed relative to the image sensor and focus control mechanism 225B may be omitted without departing from the scope of the present disclosure.

[0070] The exposure control mechanism 225A of the control mechanism 220 can obtain an exposure setting. In some cases, the exposure control mechanism 225A stores the exposure setting in a memory register. Based on the exposure setting, the exposure control mechanism 225A can control the size of the aperture (e.g., aperture size or f / stop), the duration that the aperture is open (e.g., exposure time or shutter speed), the duration that the sensor collects light (e.g., exposure time or electronic shutter speed), the sensitivity of the image sensor 230 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 230, or any combination thereof. The exposure setting can be referred to as an image capture setting and / or an image processing setting.

[0071] The zoom control mechanism 225C of the control mechanism 220 can obtain the zoom setting. In some examples, the zoom control mechanism 225C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 225C can control the focal length of the lens element assembly (lens assembly) including the lens 215 and one or more additional lenses. For example, the zoom control mechanism 225C can control the focal length of the lens assembly by driving one or more motors or servo mechanisms (or other lens mechanisms) to move one or more lenses relative to each other. The zoom setting can be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly can include a parfocal zoom lens or a zoom zoom lens. In some examples, the lens assembly can include a focusing lens (which in some cases can be a lens 215) that first receives light from the scene 210, and then the light passes through an afocal zoom system between the focusing lens (e.g., lens 215) and the image sensor 230 before the light reaches the image sensor 230. In some cases, the afocal zoom system may include two positive (e.g., converging, convex) lenses of equal or similar focal lengths (e.g., within a threshold difference of each other), with a negative (e.g., diverging, concave) lens between the two lenses. In some cases, zoom control mechanism 225C moves one or more lenses in the afocal zoom system, such as a negative lens and one or two positive lenses. In some cases, zoom control mechanism 225C may control the zoom by capturing images from an image sensor in a plurality of image sensors (e.g., including image sensor 230) with a zoom corresponding to a zoom setting. For example, image processing system 200 may include a wide-angle image sensor with a relatively low zoom and a telephoto image sensor with a larger zoom. In some cases, based on the selected zoom setting, zoom control mechanism 225C may capture images from the corresponding sensors.

[0072] Image sensor 230 includes one or more photodiode arrays or other light sensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a particular pixel in the image produced by image sensor 230. In some cases, different photodiodes may be covered by different filters. In some cases, different photodiodes may be covered in color filters and thus may measure light that matches the color of the filter covering the photodiode. Various color filter arrays may be used, including Bayer color filter arrays (such as Figure 1A shown), QCFA (see Figure 1B ) and / or any other color filter array.

[0073] return Figure 1A and Figure 1B , instead of or in addition to a red color filter, a blue color filter, and / or a green color filter, other types of color filters may use a yellow color filter, a magenta color filter, and / or a cyan (also known as an "emerald") color filter. In some cases, some photodiodes may be configured to measure infrared (IR) light. In some implementations, the photodiodes that measure IR light may not be covered by any filter, thereby allowing the IR photodiodes to measure visible light (e.g., color) and IR light. In some examples, the IR photodiodes may be covered by IR filters to allow IR light to pass through and block light from other parts of the spectrum (e.g., visible light, color). Some image sensors (e.g., image sensor 230) may lack filters entirely (e.g., color, IR, or any other part of the spectrum), but may use different photodiodes (in some cases vertically stacked) throughout the pixel array. Different photodiodes throughout the pixel array may have different spectral sensitivity curves, thereby responding to light of different wavelengths. Monochrome image sensors may also lack filters and, therefore, lack color depth.

[0074] In some cases, the image sensor 230 may alternatively or additionally include an opaque and / or reflective mask that blocks light from reaching a specific photodiode or a portion of a specific photodiode at a specific time and / or from a specific angle. In some cases, an opaque and / or reflective mask may be used for PDAF. In some cases, an opaque and / or reflective mask may be used to block portions of the electromagnetic spectrum from reaching the photodiode of the image sensor (e.g., an IR cutoff filter, an ultraviolet (UV) cutoff filter, a bandpass filter, a low-pass filter, a high-pass filter, etc.). The image sensor 230 may also include an analog gain amplifier for amplifying an analog signal output by a photodiode and / or an analog-to-digital converter (ADC) for converting an analog signal output by a photodiode (and / or amplified by an analog gain amplifier) ​​into a digital signal. In some cases, specific components or functions discussed with respect to one or more control mechanisms 220 may be alternatively or additionally included in the image sensor 230. Image sensor 230 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal oxide semiconductor (CMOS), an N-type metal oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0075] The image processor 250 may include one or more processors, such as one or more ISPs (e.g., ISP 254), one or more host processors (e.g., host processor 252), and / or Fig.15 The host processor 252 may be a digital signal processor (DSP) and / or other type of processor. In some implementations, the image processor 250 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes the host processor 252 and the ISP 254. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 256), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth TM, global positioning system (GPS), etc.), any combination thereof, and / or other components. The I / O port 256 may include any suitable input / output port or interface according to one or more protocols or specifications, such as an integrated circuit interconnect 2 (I2C) interface, an integrated circuit interconnect 3 (I3C) interface, a serial peripheral interface (SPI) interface, a serial general purpose input / output (GPIO) interface, a mobile industry processor interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, an advanced high-performance bus (AHB) bus, any combination thereof, and / or other input / output ports. In an illustrative example, the host processor 252 may communicate with the image sensor 230 using an I2C port, and the ISP 254 may communicate with the image sensor 230 using a MIPI port.

[0076] The image processor 250 may perform many tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The image processor 250 may store the image frames and / or processed images in a random access memory (RAM) 240, a read-only memory (ROM) 245, a cache, a storage unit, another storage device, or some combination thereof.

[0077] Various input / output (I / O) devices 260 may be connected to the image processor 250. The I / O devices 260 may include a display screen, a keyboard, a keypad, a touch screen, a touchpad, a touch-sensitive surface, a printer, any other output device 1635, any other input device 1645, or some combination thereof. In some cases, subtitles may be input into the image processing device 205B via a physical keyboard or keypad of the I / O device 260, or via a virtual keyboard or keypad of a touch screen of the I / O device 260. The I / O devices 260 may include one or more ports, jacks, or other connectors that enable a wired connection between the image capture and processing system 200 and one or more peripheral devices, through which the image capture and processing system 200 may receive data from and / or send data to the one or more peripheral devices. The I / O devices 260 may include one or more wireless transceivers that enable wireless connections between the image capture and processing system 200 and one or more peripheral devices, through which the image capture and processing system 200 may receive data from and / or send data to the one or more peripheral devices. The peripheral devices may include any of the types of I / O devices 260 discussed previously, and once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector, they themselves may be considered I / O devices 260.

[0078] In some cases, the image capture and processing system 200 can be a single device. In some cases, the image capture and processing system 200 can be two or more separate devices, including an image capture device 205A (e.g., a camera) and an image processing device 205B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 205A and the image processing device 205B can be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly via one or more wireless transceivers. In some implementations, the image capture device 205A and the image processing device 205B can be disconnected from each other.

[0079] like Figure 2 As shown, the vertical dashed line will Figure 2The image capture and processing system 200 is divided into two parts, representing an image capture device 205A and an image processing device 205B. The image capture device 205A includes a lens 215, a control mechanism 220, and an image sensor 230. The image processing device 205B includes an image processor 250 (including an ISP 254 and a host processor 252), a RAM 240, a ROM 245, and an I / O 260. In some cases, specific components shown in the image capture device 205A (such as the ISP 254 and / or the host processor 252) may be included in the image capture device 205.

[0080] The image capture and processing system 200 may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 200 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 wi-fi communication, wireless local area network (WLAN) communication, or some combination thereof. In some implementations, the image capture device 205A and the image processing device 205B may be different devices. For example, the image capture device 205A may include a camera device, and the image processing device 205B may include a computing device, such as a mobile handset, a desktop computer, or other computing devices.

[0081] Although the image capture and processing system 200 is shown as including specific components, one of ordinary skill will appreciate that the image capture and processing system 200 may include more than one component. Figure 2 . The components of the image capture and processing system 200 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 200 may include electronic circuits or other electronic hardware and / or may be implemented using electronic circuits or other electronic hardware, wherein the electronic circuits or other electronic hardware may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device that implements the image capture and processing system 200.

[0082] Figure 3is a block diagram illustrating an example of an image capture system 300. The image capture system 300 includes various components that are used to process input images or frames to produce output images or frames. As shown, the components of the image capture system 300 include one or more image capture devices 302, an image processing engine 310, and an output device 312. The image processing engine 310 can produce a high dynamic range depiction of a scene, as described in more detail herein.

[0083] The image capture system 300 may include an electronic device or system, or be a portion of an electronic device or electronic system. For example, the image capture system 300 may include an electronic device or system, or be a portion of an electronic device or electronic system, such as a mobile or fixed telephone handset (e.g., a smartphone, a cellular phone, etc.), an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a vehicle or a computing device / system of a vehicle, a server computer (e.g., communicating with another device or system, such as a mobile device, an XR system / device, a vehicle computing system / device, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera device, a display device, a digital media player, a video streaming device, or any other suitable electronic device. In some examples, the image capture system 300 may include one or more wireless transceivers (or separate wireless receivers and transmitters) for wireless communications, such as cellular network communications, 802.11 Wi-Fi communications, WLAN communications, Bluetooth or other short-range communications, other combinations, and / or other communications. In some implementations, the components of image capture system 300 can be part of the same computing device. In some implementations, the components of image capture system 300 can be part of two or more separate computing devices.

[0084] Although image capture system 300 is shown as including specific components, one of ordinary skill will appreciate that image capture system 300 may include more than Figure 3 In some cases, additional components of image capture system 300 may include software, hardware, or one or more combinations of software and hardware. For example, in some cases, image capture system 300 may include one or more other sensors (e.g., one or more inertial measurement units (IMUs), radars, light detection and ranging (LIDAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and / or Figure 3One or more other software and / or hardware components not shown in the figure. In some implementations, additional components of the image capture system 300 may include and / or be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., DSPs, microprocessors, microcontrollers, GPUs, CPUs, any combination thereof, and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the image capture system 300.

[0085] One or more image capture devices 302 may capture image data and generate images (or frames) based on the image data, and / or may provide the image data to an image processing engine 310 for further processing. One or more image capture devices 302 may also provide the image data to an output device 312 for output (e.g., on a display). In some cases, the output device 312 may also include storage. An image or frame may include an array of pixels representing a scene. For example, an image may be a red-green-blue (RGB) image having a red component, a green component, and a blue component per pixel; a luminance, chrominance red, chrominance blue (YCbCr) image having a luminance component and two chrominance (color) components (chrominance red and chrominance blue) per pixel; or any other suitable type of color or monochrome image. In addition to the image data, the image capture device may also generate supplemental information, such as the amount of time between consecutively captured images, a timestamp of image capture, and the like.

[0086] Figure 4A technique for generating a fused frame from a short exposure frame and a long exposure frame is shown. As shown, a short exposure frame 402 and a long exposure frame 404 may be employed, which may be fused to provide a fused frame output 406 (e.g., an HDR frame output). Due to the bit depth of the image capture sensor, some pixels of the captured frame may be oversaturated, which results in the image not showing some textures of the scene as shown in the short exposure frame 402. Therefore, in order to generate an HDR frame, both a short exposure frame and a long exposure frame may be captured, which may be fused (e.g., merged) to generate an HDR output frame. Fusion of the short exposure frame and the long exposure frame may be performed to generate a fused output frame, which includes a portion of the short exposure frame and a portion of the long exposure frame. For example, a region 408 of the fused frame output 406 may be from the long exposure frame 404, and a region 410 of the fused frame output 406 may be from the short exposure frame 402. However, fusing the short exposure frame and the long exposure frame may result in irregularities due to global motion (e.g., motion of the image capture device). For example, from the time the long exposure frame is captured to the time the short exposure frame is captured, objects in the scene or the image capture device may have moved, causing irregularities if steps are not taken to align the short exposure frame with the long exposure frame before fusing them together. As described in more detail herein, such global motion problems may also occur due to rolling shutter.

[0087] Figure 5 is a diagram showing long exposure flow and short exposure flow (e.g., MIPI flow) for processing from an image sensor (e.g., image sensor 230) to an imaging front end. Line 502 represents the beginning of long exposure sensing (also referred to herein as normal exposure sensing), and line 504 represents the end of long exposure sensing. As shown, long exposure sensing starts from the first row of the sensor (e.g., Figure 2 230) to the last row of the sensor. For each row (e.g., a row of photodiodes), once long exposure sensing has been completed, short exposure sensing begins while long exposure sensing continues to the next row. For example, line 506 represents the beginning of short exposure sensing and line 508 represents the end of short exposure sensing, starting from the first row to the last row of the image sensor. Long exposure sensing (e.g., Figure 5 with a duration labeled “N Normal” in the example) may begin before short exposure sensing (e.g., Figure 5 with a duration marked as “N short”).

[0088] Once long exposure sensing is completed for a particular row, a short delay occurs (e.g., associated with the gap between lines 504, 506) before short exposure sensing begins. Once short exposure sensing is completed for a particular row, the information for that row is read out from the image sensor for processing. Due to the gap from long exposure sensing to short exposure sensing (e.g., between Figure 5 There is a chance of movement for the user holding the camera and / or for objects in the scene being captured, which results in misalignment of features in the short exposure frame and the long exposure frame (e.g., features that are common or identical in the short exposure frame and the long exposure frame). For example, motion delay (D) may exist from time 550 (e.g., the time when half of the long exposure data is captured) to time 552 (e.g., the time when half of the short exposure data is captured). Motion delay (D) may be estimated as the average motion delay associated with different long frame capture events and short frame capture events (e.g., different HDR frame captures).

[0089] Because sensing occurs one row at a time (e.g., starting from the first row to the last row), rolling shutter global motion also occurs. An object or camera in the scene may move from when data for the first row of sensors is captured to when data for the last row of sensors is captured.

[0090] Figure 6 604 and one or more long exposure frames 602. The fusion engine 606 can fuse the one or more short exposure frames 604 and the one or more long exposure frames 602 to generate an HDR frame. Figure 5 As described, long exposure data corresponding to one or more long exposure frames 602 may be captured for each row before short exposure data corresponding to one or more short exposure frames 604. Thus, data from each row of one or more long exposure frames 602 may be received and stored in buffer 603 before data for each row of one or more short exposure frames 604 is stored in buffer 605. As shown, accumulation of data for one or more long exposure frames 602 may be accumulated before accumulation of data for one or more short exposure frames 604 (e.g., because Figure 5 As shown, the long exposure capture occurs before the short exposure capture).

[0091] In some cases, fusion by the fusion engine 606 may begin once a particular number of sensor rows or rows (e.g., first 3 rows / rows, first 4 rows / rows, first 8 rows / rows, or other number of rows / rows) of short frame data corresponding to one or more short exposure frames 604 have been accumulated. For example, operations for frame alignment may begin upon receipt of short frame data for a particular number of sensor rows (e.g., rather than waiting for the entire frame to be received). However, various constraints may exist when performing frame alignment. For example, it may not be possible to fully warp a long exposure frame (from one or more long exposure frames 602) to align with a short exposure frame (from one or more short exposure frames 604). In addition, programming for alignment may have to be performed two or three frames in advance due to hardware timing limitations. In some aspects, a large buffer may be established for capturing frame data. Image data from an image sensor may be written to a central portion of an image buffer, which enables the application of offsets in the x and y dimensions to the data stored in the buffer for alignment. In addition, certain aspects of the present disclosure provide techniques for alignment prediction to allow pre-programmed alignment operations.

[0092] Fig. 7A 700 is a diagram illustrating long exposure images and short exposure images captured by an image capture system 702 undergoing displacement in multiple domains according to some aspects. In one aspect, a person may hold the image capture system 702 to capture a selfie that includes background objects (such as a tree). In this illustrative example, the selfie has two objects of interest: the user's face and the background. When the user presses a button on the image capture system, the image capture system begins capturing a first image (e.g., a short exposure image) and the image capture system may move (e.g., due to a tremor that is exacerbated based on the user extending the image capture system as far as possible) and completes capturing the short exposure image when the mobile device is at position 704, where, for clarity, position 704 is illustrated as a center point of the image capture system. After the short exposure image is captured, the image capture system may begin capturing a long exposure image while the image capture system may continue to move due to intrinsic or extrinsic factors. At position 706, the image capture system may complete capturing the long exposure image and the image capture system has moved in multiple directions. In particular, the image capture system has been offset in at least one direction (eg, to the right relative to the user) and rotated on at least one axis (eg, yaw, pitch, or roll).

[0093] In one illustrative example, region 710 depicts a foreground object (e.g., a person) in a short exposure image, and region 715 depicts a foreground object in a long exposure image. Region 720 shows background objects (e.g., trees, landmarks, etc.) in a short exposure image, and region 725 shows background objects in a long exposure image. Fig. 7AAs shown, region 710 is different from region 715 because the movement of image capture system 702 affects the alignment error based on the distance from the camera. For example, region 715 is offset due to the displacement of image capture system 702 and rotated along region 715 due to the rotation of image capture system 702. In this example

[0094] Diagram 700 also includes region 720 depicting background objects (e.g., trees) in the short exposure image and region 725 in the long exposure image. Because background objects have different depths (e.g., farther from the camera), background objects have less error than foreground objects. In some aspects, the approximation error is based on focal length multiplied by movement (e.g., rotation, displacement) divided by the object distance. For example, the alignment error is inversely proportional to the distance from the camera to the object, with objects closer to the camera exhibiting larger alignment errors and objects farther from the camera exhibiting smaller alignment errors.

[0095] Figure 7B is a diagram showing the rotation correction according to some aspects of the present disclosure. Fig. 7A 750 of a long exposure image and a short exposure image in FIG. The image capture system 702 can be configured to record movement data from the gyroscope sensor and determine the amount of rotation that occurred between capturing the short exposure image and capturing the long exposure image. Based on the captured information, the image capture system 702 is configured to modify the short exposure image and / or the long exposure image (e.g., the last captured short exposure image, and / or the last captured long exposure image). Figure 7B As shown, the area 755 corresponding to the person in the long exposure image is corrected based on the rotation information from the gyro sensor. While solving the rotation deformation, the area 755 corresponding to the person in the long exposure image (and the area 760 corresponding to the background object) is corrected for one direction of displacement of the image capture system 702, but the alignment error in the other direction of displacement of the image capture system 702 still exists and cannot be corrected. As shown below Figure 8 As shown, alignment errors cause noise and ghosting problems that degrade the quality of HDR images.

[0096] Figure 8 is an example of an HDR image (also referred to as an HDR frame) generated by an HDR fusion system after correcting for rotational movement according to some aspects. Figure 8As shown, in an image capture system showing a foreground region and a background region, an image is captured by a forward-facing camera. In one illustrative aspect, region 810 shows a buckle that is reproduced twice with a slight offset based on the displacement of the imaging system. The image also includes a noise region 820 having discontinuities (e.g., noise) based on a failure to align the long exposure image and the short exposure image. A ghost region 830 on the opposite side of the foreground object also exists, which is generated based on the edge of the misaligned foreground object. For example, an edge between a facial region of the image and the background is depicted in ghost region 830. In one illustrative aspect, ghost regions 810 and 830 exist because content from either the long exposure image or the short exposure image is lost, and the noise region exists because the long exposure image is lost and the image content must be selected from the short exposure image, where the short exposure image has more noise due to the shorter exposure.

[0097] In some aspects, alignment adversely affects the ability of an image capture system to capture high-quality HDR images using a front-facing camera. Image capture system manufacturers either prohibit the use of a front-facing camera to capture HDR images, or limit exposure ratios to reduce the noise associated with the HDR images produced. As a result, conventional selfies captured by image capture systems have limited dynamic range and constrain the quality of the resulting HDR images.

[0098] Fig. 9 906 has a longer exposure time than the exposure time used for the short exposure image 904.

[0099] In one illustrative aspect, short exposure image 904 is provided to exposure compensator 910, which increases the brightness of short exposure image 903 based on the exposure ratio associated with short exposure image 902 and long exposure image 906. For example, if short exposure image 904 has an exposure time of 5 milliseconds (ms) and long exposure image 906 has an exposure time of 20 ms, the exposure ratio is 4, and the brightness of long exposure image 906 should be 4 times the brightness of short exposure image 903. Exposure compensator 910 normalizes the brightness of short exposure image 904 to correspond to the brightness of long exposure image 906. The modified short exposure image 904 and long exposure image 906 are provided to feature detector 915 to identify key points within each image. Feature detector 915 identifies different key points from each image and provides the key points to motion estimator 920. For example, feature detector 915 is configured to provide a first set of keypoints associated with short exposure image 904 to motion estimator 920 , and to provide a second set of keypoints associated with long exposure image 906 to motion estimator 920 .

[0100] In one aspect, long exposure image 906 is also provided to object detector 925 to identify objects of interest in the foreground. In one illustrative example, object detector 925 is configured to recognize a face or multiple faces (e.g., a selfie) and is configured to output a bounding region (e.g., a bounding region, or other similar geometric region for identifying a foreground region in long exposure image 906) to motion estimator 920.

[0101] In some aspects, the motion estimator 920 uses a bounding area and discards key points outside the bounding area. For example, the bounding area corresponds to a foreground area, and discarding key points outside the foreground area is used to remove key points associated with the background area from the short exposure image 904 and the long exposure image 906. The motion estimator 920 analyzes the key points associated with the foreground of the short exposure image 904 and the key points associated with the foreground of the long exposure image 906, and determines a translation matrix based on these key points, and the homography matrix can be used to correct the displacement (e.g., movement) of the image capture system during image capture. In one illustrative aspect, the translation matrix can be a homography matrix determined based on the short exposure image 904 and the long exposure image 906. According to some examples, multiple images of the same plane (e.g., multiple images captured as part of an HDR image) are related by a homography matrix, which can be used for various corrections in any image, such as scaling correction, translation correction, and rotation correction. In some aspects, the motion transformation can be calculated based on the difference of the key points associated with the foreground area of ​​the short exposure image 904 and the long exposure image 906.

[0102] The motion transform corresponds to an amount of movement of the image capture system between capturing short exposure image 904 and capturing long exposure image 906. In one illustrative example, motion estimator 920 generates a translation matrix that represents a function associated with at least one pixel in an image or a group of pixels in an image, and the translation matrix can be used to correct a portion of short exposure image 904 or long exposure image 906, such as a foreground area detected by object detector 925. Alternatively or additionally, the translation matrix can be configured to represent the entire image.

[0103] The image processing system 900 includes a gyroscope sensor 935 that provides relative or absolute rotation information (e.g., yaw, pitch, and roll) related to the orientation of the image capture system. For example, the gyroscope sensor 935 can be configured to detect the rotation information when capturing (e.g., reading out) the short exposure image 904 from the image sensor and when capturing the long exposure image 906. The rotation information is provided to the rotation estimator 940, which determines the amount by which the image capture system rotation estimator 940 was rotated between capturing the short exposure image 904 and capturing the long exposure image 906.

[0104] According to an illustrative example, the rotation estimator 940 is configured to generate a rotation matrix (e.g., a homography matrix) based on the rotation information from the gyroscope sensor 935, and the rotation matrix is ​​configured to correct the rotation associated with the image capture system. For example, the rotation matrix can be configured to detect rotation along any axis (e.g., yaw, pitch, roll). The rotation matrix can be applied to one of the short exposure image 904 or the long exposure image 906 to align the content within the short exposure image 902 and the long exposure image 908.

[0105] In some aspects, the translation matrix from the motion estimator 920, the rotation matrix from the rotation estimator 940, and the bounding region from the object detector 925 are provided by the motion estimator 930. In some aspects, the motion estimator 930 is configured to identify a transformation to be applied to the short exposure image 904 based on the translation matrix, the rotation matrix, and the bounding region. Alternatively or in addition, the transformation can be applied to the long exposure image 906. In one illustrative aspect, the motion estimator 930 is configured to synthesize a hybrid transformation matrix that includes correction information associated with the translation matrix (e.g., translation correction) and correction information associated with a second matrix (e.g., rotation correction). For example, the motion estimator 930 can use the bounding region from the object detector 925 to determine an area of ​​the short exposure image 904 that will be corrected based on the translation correction (e.g., based on the movement of key points in the bounding region between the short exposure image 904 and the long exposure image 906), and the remaining area can be associated with the rotation correction, and the following reference is made to Fig. 10CAn example of this matrix is ​​further described.

[0106] In another aspect, the motion estimator 930 may construct a hybrid transform having a first region corresponding to a foreground object based on a translation matrix, a second region corresponding to a background region based on a rotation matrix, and a third boundary region around the boundary region interpolating values ​​from the translation matrix and the rotation matrix, and referring to Fig. 10D An example of the matrix is ​​further described. An example of a boundary region may include an outer edge region extending beyond the boundary region and an inner edge region within the boundary region. Other examples include a boundary region corresponding to an inner edge of the boundary region, or a boundary region corresponding to an outer edge of the boundary region. The size of the boundary region can be determined based on various factors such as the size of the boundary region. In another example, the size of the boundary can be based on a depth map that identifies various regions at different depths, and the size of the boundary region can be dynamically determined based on specific details associated with objects in the short exposure image 904 and the long exposure image 906.

[0107] The motion estimator 930 may be configured to determine whether a rotation estimation is necessary. For example, if the bounding region corresponds to a threshold (eg, 90%) of the size of the entire image, the motion estimator 930 may determine that correction of background objects is not required.

[0108] According to one example, the motion estimator 930 can generate a translation matrix that is provided to the image corrector 945. In one illustrative aspect, the image corrector 945 receives the short exposure image 904 and the transformation matrix from the motion estimator 930, and transforms the short exposure image 904 into an aligned short exposure image 950. As described above, the transformation matrix can be configured to transform a portion of the image based on rotation information from the gyroscope sensor 935, based on movement information determined by key point detection within the motion estimator 920. Alternatively or in addition, the image corrector 945 can be configured to correct the long exposure image or a supplemental image such as a medium exposure image. The fusion engine 960 is configured to receive the aligned short exposure image 950 and the long exposure image 906, and generate an HDR image 970. As described above, the fusion engine 960 can also receive additional images for various purposes, such as a medium exposure image for reducing noise within the HDR image 970. The additional medium exposure image can also be used to supplement the various motion estimates described above. In this document, reference is made to Fig.13 A flow chart for identifying a selective contour fill according to some aspects is shown. Figures 10A-10D An exemplary diagram for explaining various matrices is shown in Fig.11 An exemplary method of correcting rotation is shown in Fig.12 An example of key point detection is shown in .

[0109] Figures 10A-10D is a conceptual diagram illustrating various matrices generated by an HDR fusion system to correct spatial and rotational alignment according to some aspects of the present disclosure. Although each matrix is ​​described with fill, the fill represents a different type of transformation. For example, Fig. 10A A rotation matrix configured to apply at least one rotational transformation to an image is shown. The amount of rotation of a portion of an image varies based on a plurality of parameters. Non-limiting examples of parameters of the rotation include a center point of the rotational movement, a depth of the object in the image, the amount of rotation, and a radius of the object relative to the center point of the rotational movement. For example, a background object 400 meters (m) away from the image capture system will be rotated a greater amount than an object 1 meter away from the image capture system based on different radii.

[0110] Fig. 10B A translation matrix is ​​shown that is configured to apply at least one spatial transformation to an image. As described above with respect to the motion estimator 920, the translation matrix represents movement in a coordinate system. For example, the translation matrix can represent movement in Cartesian coordinates. In another illustrative example, the movement transformation can be represented by an equation that identifies a linear transformation or a nonlinear translation in at least one direction.

[0111] Fig. 10C shows a merged matrix configured to apply Fig. 10A The rotation matrix shown is the same as Fig. 10B The translation matrix is ​​configured to be applied to a region corresponding to the bounding region 1002 (eg, the bounding region created by the object detector 925 ). Table 1 below shows pseudo code for selecting values ​​from the translation matrix and the rotation matrix based on the bounding region 1002 .

[0112]

[0113] Table 1

[0114] In the illustrative example in Table 1, a matrix is ​​generated based on the dimensions of the long exposure image 906 so that each pixel in the long exposure image 906 corresponds to a cell within the merged matrix. The matrix in Table 1 can also be generated based on a rotation matrix or a translation matrix. The merged matrix can also have different dimensions (e.g., larger or smaller). The value of each cell is determined based on whether the cell is determined to be within the boundary area 1002 (e.g., from the object detector 925). If the cell is within the boundary area 1002, the cell value is set equal to the corresponding cell in the translation matrix. If the cell is not within the boundary area 1002, the value is set equal to the corresponding cell in the rotation matrix.

[0115] Fig. 10D The merged matrix is ​​shown, which is configured to: utilize interpolation within the boundary area 1010 associated with the boundary area 1002, apply Fig. 10A The rotation matrix shown is the same as Fig. 10B For example, the values ​​of cells within the inner edge 1004 of the bounding area 1002 may be set to Fig. 10B The values ​​of the corresponding cells in the translation matrix shown, and the values ​​of the cells outside the outer edge 1006 of the bounding area 1002 can be set to Fig. 10A The values ​​of the corresponding cells in the rotation matrix are shown. The cells between the inner edge 1004 and the outer edge 1006 are interpolated based on the position of the cells and the corresponding cells in the displacement matrix and the rotation matrix.

[0116] For example, the combined matrix H can be described as:

[0117]

[0118] Where p corresponds to a point in the image, margin corresponds to the size of the boundary, boundary corresponds to the boundary area 1002, and H R Corresponding to the rotation matrix, H M corresponds to the translation matrix, and H I is the interpolation of the rotation matrix and the translation matrix, as shown in Equation 2 below:

[0119]

[0120] According to various examples, the size of the boundary region 1010 may be configured based on a scaled value associated with the area of ​​the boundary region. Fig. 10D As shown, the area of ​​the boundary region 1010 is configured to be 10% of the boundary region 1002, and a portion of the boundary region 1010 can overlap with the boundary region 1002. The boundary region 1010 creates a seamless transition between the foreground region and the background region, and reduces visual artifacts between different types of corrections. Based on the merged correction, the imaging system of the present disclosure reduces the aberration movement associated with different types of motion of the image capture system and generates a high-quality HDR image. For example, a forward-facing camera of an image capture system including the image processing system 900 can generate an HDR image that reduces visual artifacts such as ghosting and noise and enables the capture of an HDR image.

[0121] Fig.111 is a flow chart illustrating an example of a method 1100 for aligning images to reduce ghosting and noise in an HDR image in accordance with certain aspects of the present disclosure. The method 1100 may be performed by a computing device having an image sensor, such as a mobile wireless communication device, a camera, an XR device, a wireless-enabled vehicle, or other computing device. In one illustrative example, the computing system 1600 may be configured to perform all or part of the method 1100. In one illustrative example, an ISP such as the ISP 254 or the computing system 1600 may be configured to perform all or part of the method.

[0122] The method 1100 includes a computing system that can receive gyroscope samples 1110 with rotation information using a gyroscope sensor. In one illustrative example, the gyroscope samples 1110 can include angular velocity (e.g., in radians per second) relative to Cartesian coordinates (e.g., X, Y, and Z axes). At block 1120, the computing system can integrate the rotation information to determine an angular displacement (e.g., Euler angles) relative to the Cartesian coordinates. In one aspect, Equation 3 below shows integrating the angular rotation to determine the Euler angles.

[0123]

[0124] At block 1130, the computing system may calculate the rotation matrix. In one illustrative aspect, the computing system may calculate the rotation using the Euler transform equation, as shown in Equation 4 below.

[0125] R=R X R Y R z (Equation 4)

[0126] Other types of rotational transformations may be used. At block 1140, the computing system calculates a warp matrix W based on receiving calibration information 1145. In some aspects, the calibration information may include intrinsic information associated with various components of the camera system. The computing system may use the warp matrix at block 1150 to correct for rotational motion associated with camera image 1160 to produce an aligned image 1170. In one aspect, the warp matrix is ​​determined between a current frame relative to a previous frame.

[0127] In some aspects, method 1100 may be implemented at least in part in a machine learning (ML) model. In one illustrative example, the ML model for identifying rotations may construct a depth map for rotating a portion in an image based on distances perceived by the ML model. In another example, the ML model may generate a rotation matrix based on at least two sequential images.

[0128] Fig.12is a conceptual diagram of key point detection for identifying movement of an image capture system according to certain aspects of the present disclosure. In one aspect, key detection can be used to detect various features in long exposure images and short exposure images. Examples of feature detectors include Harris corner detection, fast corner detection, histogram of oriented gradients, scale-invariant feature transforms, etc. Since the short exposure images are brightened based on the exposure ratio as described above, the feature detection algorithm should detect the same features between images. Based on the detected features, a feature matching algorithm (e.g., normalized cross correlation, Kullback-Leibler divergence (KLD) distance, block matching, etc.) is used to match corresponding feature points in the short exposure and long exposure images. The homography matrix calculation between frames removes abnormal key points between frames and provides accurate transformation estimates.

[0129] In one aspect, Fig.12 Depicted are key points detected in the original image 1205 mapped to the transformed image 1210. In particular, the transformed image is rotated and scaled, and points between various features in the image, such as the lion's left hind foot and the lion's tail are mapped between the images. Using this mapping, a homography matrix can be generated that corrects for displacement of the image capture system between frames.

[0130] In some aspects, the homography matrix can be at least partially determined using an ML model. In one illustrative aspect, at least one of a keypoint detection algorithm and a feature matching algorithm can be implemented in the ML model. In one illustrative example, the ML model can identify bounding regions corresponding to a person's shape in different images (e.g., a person's outline mask).

[0131] Fig.13 1 is a flowchart illustrating an example of a method 1300 for synthesizing an HDR image using background fill techniques to reduce ghosting according to certain aspects of the present disclosure. The method 1300 may be performed by a computing system or device (or a component thereof, such as a chipset) having an image sensor, such as a mobile wireless communication device, a camera, an XR device, a wireless-enabled vehicle, or other computing device. In one illustrative example, the computing system 1600 may be configured to perform all or part of the method 1300. In one illustrative example, an ISP such as the ISP 254 may be configured to perform all or part of the method.

[0132] At block 1302, a computing system (or a component thereof) may be configured to obtain a first image captured using an image sensor, wherein the first image is associated with a first exposure. At block 1304, a computing system (or a component thereof) may be configured to obtain a second image captured using an image sensor. The second image is associated with a second exposure that is longer than the first exposure. For example, the first image may be a short exposure image and the second image may be a long exposure image. According to an illustrative example, the image sensor is oriented in the same direction as a display for displaying a preview image captured by the image sensor. For example, the image sensor may be configured to capture a selfie image of a user of the system.

[0133] The computing system (or a component thereof) may be configured to generate a first matrix for performing a first transformation and to generate a second matrix for performing a second transformation. In one illustrative example, the first transformation includes a translation matrix associated with movement of the image sensor during acquisition of the first image and acquisition of the second image.

[0134] For example, the translation matrix can be a homography matrix used to correct for movement of the image capture device that occurs during image capture. In one example, the computing system can normalize the image based on the exposure ratio difference between the first image and the second image to increase the brightness of the first image. The computing system can detect an object such as a person's face in the second image and determine a boundary area associated with the position of the object in the second image. The computing system can be configured to extract first feature points from the first image and can extract second feature points from the second image. Based on the detected feature points and the boundary area, the computing system can identify a subset of the first feature points within the boundary area, identify a subset of the second feature points in the boundary area, and generate a first matrix based on the subset of the first feature points and the subset of the second feature points.

[0135] In some aspects, the second matrix can be generated based on movement detected by a motion sensor between a first time when the first image is captured and a second time when the second image is captured. For example, the motion sensor is a gyroscope sensor and detects rotation between the first image and the second image. In this case, the second transform can be a homography matrix that can be corrected for the rotational transform.

[0136] According to some aspects, the computing system may generate a blending transformation matrix based on the first matrix and the second matrix for modifying the first region of the first image and the second region of the first image. For example, the computing system may initiate a new matrix called a blending transformation matrix, add values ​​from the first matrix corresponding to at least the first region to the blending transformation matrix, and add values ​​from the second matrix corresponding to at least the second region to the blending transformation matrix.

[0137] In one illustrative example, the hybrid transformation matrix may have two different regions, a first region corresponding to a background region and a second region corresponding to a foreground. In another illustrative example, the hybrid transformation matrix may include a third region that exists between the first region and the second region and mixes the first transformation matrix and the second transformation matrix. For example, the computing system may determine a transition region between the first region and the second region based on the size of a boundary region associated with the position of an object in at least one of the first image or the second image, determine a value associated with the transition region based on a representation of the first matrix and the second matrix, and add the value associated with the transition region to the hybrid transformation matrix. For example, the representation of the first matrix and the second matrix includes a weighted average of the first matrix and the second matrix. In another example, the representation of the first matrix and the second matrix is ​​based on a proportional distance from an inner edge of the transition region to an outer edge of the transition region.

[0138] At block 1306, the computing system (or a component thereof) may be configured to modify a first region of the first image based on the first transformation, and to modify a second region of the first image based on the second transformation to generate a modified first image. In one example, the first region is associated with an object at a first depth in the scene relative to the image sensor, and the second region includes a background region at a second depth in the scene relative to the image sensor. For example, the first region may be associated with a user's face, or a subject of an image, and the second region may be a background.

[0139] At block 1308, the computing system (or a component thereof) may generate a merged image at least in part by merging the modified first image and the second image. In one example, the merged image is an HDR image. According to some aspects, the final HDR image is corrected for motion in multiple domains (e.g., rotation, translation). As a result, the final HDR image is corrected for multi-domain motion and the HDR image may be captured with high visual fidelity.

[0140] As described above, the processes or methods described herein (e.g., methods 1100 and 1300, and / or other processes described herein) can be performed by a computing system (or device or apparatus). In one example, methods 1100 and 1300 can be performed by a computer having Fig.16 The computing architecture of the computing system 1600 shown is a computing device (e.g., Figure 2The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, an AR glasses, a network-connected watch or smart watch, or other wearable device), a server computer, a computing device of an autonomous vehicle or an autonomous vehicle, a robotic device, a television, and / or any other computing device with resource capabilities to perform the methods described herein (including methods 1100 and 1300). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the methods described herein. In some examples, the computing device may include a display, a network interface configured to transmit and / or receive data, any combination thereof, and / or other components. The network interface may be configured to transmit and / or receive IP-based data or other types of data.

[0141] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0142] Methods 1100 and 1300 are illustrated as logical flow diagrams, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the method.

[0143] Methods 1100 and 1300 and / or other methods or processes described herein may be executed under the control of one or more computer systems configured with executable instructions, or may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is collectively executed on one or more processors, implemented by hardware, or a combination thereof. As described above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes multiple instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0144] As described above, various aspects of the present disclosure may utilize a machine learning model or system. Fig.14 is an illustrative example of a deep learning neural network 1400 that can be used to implement the above-mentioned machine learning-based alignment prediction. Input layer 1420 includes input data. In an illustrative example, input layer 1420 may include data representing pixels of an input video frame. Neural network 1400 includes multiple hidden layers 1422a, 1422b to 1422n. Hidden layers 1422a, 1422b to 1422n include a certain number "n" hidden layers, where "n" is an integer greater than or equal to 1. The number of hidden layers can include any number of layers required for a given application. Neural network 1400 also includes an output layer 1421, which provides an output generated by the processing performed by hidden layers 1422a, 1422b to 1422n. In an illustrative example, output layer 1421 can provide classification for objects in the input video frame. Classification can include categories that identify activity types (e.g., looking up, looking down, closing eyes, yawning, etc.).

[0145] Neural network 1400 is a multi-layer neural network of interconnected nodes. Each node can represent a piece of information. Information associated with a node is shared between different layers, and each layer retains the information as it is processed. In some cases, neural network 1400 may include a feedforward network, in which case there is no feedback connection where the output of the network is fed back to itself. In some cases, neural network 1400 may include a recursive neural network, which may have loops that allow information to be carried across nodes when reading inputs.

[0146] Information can be exchanged between nodes by node-to-node interconnection between different layers. The nodes of input layer 1420 can activate a group of nodes in the first hidden layer 1422a. For example, as shown in the figure, each input node of input layer 1420 is connected to each node of the first hidden layer 1422a. The nodes of the first hidden layer 1422a can transform the information of each input node by applying an activation function to the input node information. The information derived according to the transformation can then be passed to and can activate the nodes of the next hidden layer 1422b, which can perform the functions specified by themselves. Example functions include convolution, upsampling, data transformation and / or any other suitable function. Then, the output of hidden layer 1422b can activate the nodes of the next hidden layer, and so on. The output of the last hidden layer 1422n can activate one or more nodes of output layer 1421, wherein output is provided at the one or more nodes. In some cases, although the nodes (e.g., nodes 1426) in neural network 1400 are shown as having multiple output lines, the nodes have a single output, and all lines shown as being output by the nodes represent the same output value.

[0147] In some cases, each node or interconnection between nodes can have a weight, which is a set of parameters derived from training the neural network 1400. Once the neural network 1400 is trained, it can be referred to as a trained neural network, which can be used to classify one or more activities. For example, the interconnection between nodes can represent a piece of information learned about the interconnected nodes. The interconnections can have tunable numerical weights that can be tuned (e.g., based on a training data set) to allow the neural network 1400 to adapt to the input and be able to learn as more and more data is processed.

[0148] The neural network 1400 is pre-trained to process features from data in the input layer 1420 using different hidden layers 1422a, 1422b to 1422n to provide output through the output layer 1421. In examples where the neural network 1400 is used to identify features and / or objects in an image, as described above, the neural network 1400 can be trained using training data that includes both images and labels. For example, training images can be input into the network, where each training frame has a label indicating a feature in the image (for a feature extraction machine learning system) or a label indicating a class of activity in each frame. In one example using object classification for illustration purposes, the training frame may include an image of the number 2, in which case the label for the image may be [0 0 1 0 0 0 0 0 0 0].

[0149] In some cases, the neural network 1400 can use a training process called back propagation to adjust the weights of the nodes. As described above, the back propagation process can include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter update are performed for one training iteration. This process can be repeated a certain number of iterations for each set of training images until the neural network 1400 is trained well enough so that the weights of the layers are accurately tuned.

[0150] For the example of identifying features and / or objects in an image, a forward pass can include passing a training image through the neural network 1400. Prior to training the neural network 1400, the weights are initially randomized. As an illustrative example, a frame can include an array of numbers representing pixels of an image. Each number in the array can include a value from 0 to 255 describing the intensity of the pixel at that location in the array. In one example, the array can include a 28 x 28 x 3 array of numbers having 28 rows and 28 columns of pixels and 3 color components (such as red, green, and blue, or luminance and two chrominance components, etc.).

[0151] As described above, for the first training iteration for the neural network 1400, the output will likely include values ​​that do not prioritize any particular class due to the random selection of weights at initialization. For example, if the output is a vector with probabilities that the object includes different classes, the probability values ​​for each of the different classes may be equal or at least very similar (e.g., for ten possible classes, each class may have a probability value of 0.1). Using the initial weights, the neural network 1400 is unable to determine low-level features and, therefore, is unable to accurately determine what the classification of the object may be. A loss function may be used to analyze the error in the output. Any suitable loss function definition may be used, such as a cross entropy loss. Another example of a loss function includes the mean squared error (MSE), defined as The loss can be set equal to E total The value of .

[0152] For the first training image, the loss (or error) will be high because the actual value will be very different from the predicted output. The goal of training is to minimize the amount of loss so that the predicted output is the same as the training label. The neural network 1400 can perform a backward pass by determining which inputs (weights) contribute the most to the loss of the network, and the weights can be adjusted so that the loss decreases and is ultimately minimized. The derivative of the loss with respect to the weight (expressed as dL / dW, where W is the weight at a particular layer) can be calculated to determine the weight that contributes the most to the loss of the network. After calculating the derivative, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated so that they change in the opposite direction of the gradient. The weight update can be expressed as Among them, w represents the weight, w i Represents the initial weight, and η represents the learning rate. The learning rate can be set to any suitable value, where a higher learning rate includes a larger weight update, while a lower value indicates a smaller weight update.

[0153] The neural network 1400 may include any suitable deep network. An example includes a convolutional neural network (CNN), which includes an input layer and an output layer with multiple hidden layers between the input layer and the output layer. The hidden layers of the CNN include a series of convolutional layers, nonlinear layers, pooling (for downsampling) layers, and fully connected layers. The neural network 1400 may include any other deep network other than a CNN, such as an autoencoder, a deep belief network (DBN), a recursive neural network (RNN), etc.

[0154] Fig.15 is an illustrative example of a CNN 1500. The input layer 1520 of the CNN 1500 includes data representing an image or frame. For example, the data may include an array of numbers representing pixels of the image, where each number in the array includes a value from 0 to 255 that describes the intensity of the pixel at that position in the array. Using the previous example above, the array may include a 28 x 28 x 3 array of numbers with 28 rows and 28 columns of pixels and 3 color components (e.g., red, green, and blue, or luminance and two chrominance components, etc.). The image may be passed through a convolutional hidden layer 1522a, an optional non-linear activation layer, a pooling hidden layer 1522b, and a fully connected hidden layer 1522c to obtain an output at an output layer 1524. Although in Fig.15 Only one of each hidden layer is shown in , but one of ordinary skill will appreciate that multiple convolutional hidden layers, nonlinear layers, pooling hidden layers, and / or fully connected layers may be included in CNN 1500. As previously described, the output may indicate a single category of an object, or may include a probability of a category that best describes an object in the image.

[0155] The first layer of CNN 1500 is a convolutional hidden layer 1522a. Convolutional hidden layer 1522a analyzes the image data of input layer 1520. Each node of convolutional hidden layer 1522a is connected to a node (pixel) region of the input image called a receptive field. Convolutional hidden layer 1522a can be considered as one or more filters (each filter corresponds to a different activation or feature map), where each convolution iteration of the filter is a node or neuron of convolutional hidden layer 1522a. For example, the area of ​​the input image covered by the filter at each convolution iteration will be the receptive field for the filter. In an illustrative example, if the input image includes a 28×28 array and each filter (and corresponding receptive field) is a 5×5 array, there will be 24×24 nodes in convolutional hidden layer 1522a. Each connection between a node and a receptive field for that node learns a weight, and in some cases learns an overall bias, so that each node learns to analyze its specific local receptive field in the input image. Each node of hidden layer 1522a will have the same weights and biases (referred to as shared weights and shared biases). For example, the filter has an array of weights (numbers) and has the same depth as the input. For the video frame example, the depth of the filter will be 3 (according to the three color components of the input image). An illustrative size of the filter array is 5x 5x 3, corresponding to the size of the node's receptive field.

[0156] The convolutional nature of the convolutional hidden layer 1522a is due to the fact that each node of the convolutional layer is applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 1522a can start from the upper left corner of the input image array and can be convolved around the input image. As described above, each convolution iteration of the filter can be considered as a node or neuron of the convolutional hidden layer 1522a. At each convolution iteration, the value of the filter is multiplied by the corresponding number of the original pixel value of the image (for example, a 5x5 filter array is multiplied by a 5x5 array of input pixel values ​​at the upper left corner of the input image array). The multiplication results from each convolution iteration can be added together to obtain the sum for the iteration or node. According to the receptive field of the next node in the convolutional hidden layer 1522a, the process is continued at the next position of the input image. For example, the filter can be moved by a step length (called stride) to the next receptive field. The stride can be set to 1 or other suitable amount. For example, if the stride is set to 1, the filter will be moved to the right by 1 pixel at each convolution iteration. Processing the filter at each unique location in the input volume produces a number representing the result of the filter for that location, resulting in a summed value being determined for each node of the convolutional hidden layer 1522a.

[0157] The mapping from the input layer to the convolutional hidden layer 1522a is called an activation map (or feature map). The activation map includes a value for each node that represents the result of the filter at each location in the input volume. The activation map may include an array that includes the various summed values ​​produced by each iteration of the filter over the input volume. For example, if a 5 x 5 filter is applied to each pixel of a 28 x 28 input image (with a stride of 1), the activation map would include a 24 x 24 array. The convolutional hidden layer 1522a may include multiple activation maps in order to identify multiple features in the image. Fig.15 The example shown in includes three activation maps. Using the three activation maps, the convolutional hidden layer 1522a can detect three different types of features, each of which is detectable throughout the image.

[0158] In some examples, a nonlinear hidden layer can be applied after the convolutional hidden layer 1522a. Nonlinear layers can be used to introduce nonlinearity into a system that is already computing linear operations. An illustrative example of a nonlinear layer is a rectified linear unit (ReLU) layer. The ReLU layer can apply the function f(x) = max(0, x) to all values ​​in the input volume, which changes all negative activations to 0. ReLU can therefore increase the nonlinear nature of the CNN 1500 without affecting the receptive field of the convolutional hidden layer 1522a.

[0159] The pooling hidden layer 1522b may be applied after the convolutional hidden layer 1522a (and after the nonlinear hidden layer when used). The pooling hidden layer 1522b is used to simplify the information in the output from the convolutional hidden layer 1522a. For example, the pooling hidden layer 1522b may take each activation map output from the convolutional hidden layer 1522a and generate a condensed activation map (or feature map) using a pooling function. Max pooling is an example of a function performed by a pooling hidden layer. The pooling hidden layer 1522b may use other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions. A pooling function (e.g., a max pooling filter, an L2 norm filter, or other suitable pooling filter) is applied to each activation map included in the convolutional hidden layer 1522a. In Fig.15 In the example shown, three pooling filters are used for the three activation maps in the convolutional hidden layer 1522a.

[0160] In some examples, max pooling can be used by applying a max pooling filter (e.g., having a size of 2x2) with a stride (e.g., equal to the dimension of the filter, such as stride 2) to the activation map output from the convolutional hidden layer 1522a. The output from the max pooling filter includes the maximum number in each sub-region on which the filter convolves. Taking a 2x2 filter as an example, each unit in the pooling layer can summarize a region of 2×2 nodes in the previous layer (where each node is a value in the activation map). For example, four values ​​(nodes) in the activation map will be analyzed by a 2x2 max pooling filter at each iteration of the filter, where the maximum of the four values ​​will be output as the "maximum" value. If such a max pooling filter is applied to an activation filter from a convolutional hidden layer 1522a of dimension 24x24 nodes, the output from the pooling hidden layer 1522b will be an array of 18x12 nodes.

[0161] In some examples, an L2 norm pooling filter may also be used. The L2 norm pooling filter involves calculating the square root of the sum of the squares of the values ​​in a 2×2 region (or other suitable region) of the activation map (rather than calculating the maximum value as in max pooling), and using the calculated value as output.

[0162] Intuitively, a pooling function (e.g., max pooling, L2 norm pooling, or other pooling functions) determines whether a given feature is found anywhere in the image region and discards the exact location information. This can be done without affecting the results of feature detection because once a feature is found, the exact location of the feature is not as important as its approximate location relative to other features. The benefit provided by max pooling (as well as other pooling methods) is that fewer features are pooled, thereby reducing the number of parameters required in later layers of CNN 1500.

[0163] The last layer of connections in the network is a fully connected layer that connects each node from the pooling hidden layer 1522b to each output node in the output layer 1524. Using the example above, the input layer includes 28 x 28 nodes that encode the pixel intensities of the input image, the convolutional hidden layer 1522a includes 3 x 24 x 24 hidden feature nodes based on applying a 5 x 5 local receptive field (for the filter) to the three activation maps, and the pooling hidden layer 1522b includes a layer with 3 x 12 x 12 hidden feature nodes based on applying a max pooling filter to a 2 x 2 region across each of the three feature maps. Extending this example, the output layer 1524 may include ten output nodes. In such an example, each node of the 3 x 12 x 12 pooling hidden layer 1522b is connected to each node of the output layer 1524.

[0164] The fully connected layer 1522c can obtain the output of the previous pooled hidden layer 1522b (which should represent an activation map of high-level features) and determine the features most relevant to a particular category. For example, the fully connected layer 1522c can determine the high-level features that are most strongly associated with a particular category, and can include weights (nodes) for the high-level features. The product between the weights of the fully connected layer 1522c and the weights of the pooled hidden layer 1522b can be calculated to obtain probabilities for different categories. For example, if CNN1500 is used to predict that an object in a video frame is a person, a higher value will appear in the activation map representing the high-level features of a person (e.g., two legs appear, a face appears at the top of the object, two eyes appear at the upper left and upper right of the face, a nose appears in the middle of the face, a mouth appears at the bottom of the face, and / or other features common to people).

[0165] In some examples, the output from the output layer 1524 may include an M-dimensional vector (in the previous example, M=10). M indicates the number of categories that the CNN 1500 must choose from when classifying an object in an image. Other example outputs may also be provided. Each number in the M-dimensional vector may represent the probability that an object belongs to a certain category. In an illustrative example, if the 10-dimensional output vector representing ten different categories of objects is [0 0 0.05 0.8 0 0.15 0 0 00], the vector indicates that the image is a 5% probability of an object of the third category (e.g., a dog), an 80% probability of an object of the fourth category (e.g., a person), and a 15% probability of an object of the sixth category (e.g., a kangaroo). The probability for a category can be considered as the confidence level that the object is part of that category.

[0166] Fig.16 is a diagram showing an example of a system for implementing certain aspects of the technology herein. Specifically, Fig.16 An example of a computing system 1600 is shown, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using connection 1605. Connection 1605 may be a physical connection using a bus, or a direct connection into processor 1610, such as in a chipset architecture. Connection 1605 may also be a virtual connection, a networked connection, or a logical connection.

[0167] In some aspects, computing system 1600 is a distributed system, where the functionality described in the present disclosure may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a number of such components, each performing some or all of the functionality of the described component. In some aspects, a component may be a physical device or a virtual device.

[0168] The example computing system 1600 includes at least one processing unit (CPU or processor) 1610 and connections 1605 coupling various system components including system memory 1615, such as ROM 1620 and RAM 1625, to the processor 1610. The computing system 1600 may include a cache 1612 of high-speed memory directly connected to the processor 1610, in close proximity to the processor 1610, or integrated as part of the processor 1610.

[0169] Processor 1610 may include any general purpose processor and hardware services or software services (such as services 1632, 1634, and 1636 stored in storage device 1630) configured to control processor 1610 as well as special purpose processors, where the software instructions are incorporated into the actual processor design. Processor 1610 may essentially be a completely independent computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0170] To enable user interaction, the computing system 1600 includes input devices 1645, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1600 can also include output devices 1635, which can be one or more of a plurality of output mechanisms. In some cases, a multimodal system can enable a user to provide multiple types of input / output to communicate with the computing system 1600. The computing system 1600 can include a communication interface 1640, which can generally govern and manage user input and system output. The communication interface can perform or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including those that utilize the following: audio jacks / plugs, microphone jacks / plugs, universal serial bus (USB) ports / plugs, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Wireless signal transmission, BLE wireless signal transmission, Wireless signal transmission, RFID wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 WiFi wireless signal transmission, WLAN signal transmission, visible light communication (VLC), World Interoperability for Microwave Access (WiMAX), IR communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or a combination thereof. The communication interface 1640 may also include one or more global navigation satellite system (GNSS) receivers or transceivers, which are used to determine the location of the computing system 1600 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States-based Global Positioning System, the Russian-based Global Navigation Satellite (GLONASS), the China-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the basic features herein may readily be substituted by improved hardware or firmware arrangements as they are developed.

[0171] The storage device 1630 may be a non-volatile and / or non-temporary and / or computer-readable storage device, and may be a hard disk or other type of computer-readable medium (which may store computer-accessible data), such as a magnetic cassette, a flash memory card, a solid-state storage device, a digital versatile disk, a cassette, a floppy disk, a floppy disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disc-read only memory (CD-ROM) disc, a rewritable compact disc (CD) disc, a digital video disc (DVD) disc, a Blu-ray disc (BDD) disc, a holographic disc, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a Memory Card ... card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a RAM, a static RAM (SRAM), a dynamic RAM (DRAM), a ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0172] Storage device 1630 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1610, enables the system to perform functions. In some aspects, hardware services that perform specific functions may include software components stored in computer-readable media that are combined with necessary hardware components (such as processor 1610, connection 1605, output device 1635, etc.) to perform functions. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored and does not include carrier waves and / or transient electronic signals propagated wirelessly or through wired connections. Examples of non-transitory media may include, but are not limited to, disks or tapes, optical storage media such as CDs or DVDs, flash memory, memory, or storage devices. Codes and / or machine-executable instructions may be stored on computer-readable media, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or hardware circuit by passing and / or receiving information, data, parameters, arguments, or memory contents. Information, parameters, arguments, data, etc. can be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0173] In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the process steps described herein. In some examples, a computing device may include a display, one or more network interfaces configured to transmit and / or receive data, any combination thereof, and / or other components. The one or more network interfaces may be configured to transmit and / or receive wired and / or wireless data, including data in accordance with 3G, 4G, 5G, and / or other cellular standards, data in accordance with Wi-Fi (802.11x), data in accordance with Bluetooth TM Standard data, data according to IP standards and / or other types of data.

[0174] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0175] In some aspects, computer-readable storage devices, media, and memories may include wired or wireless signals including bit streams, etc. However, when referred to, non-transitory computer-readable storage media expressly excludes media such as energy, carrier signals, electromagnetic waves, and signals per se.

[0176] Specific details are provided in the above description to provide a comprehensive understanding of the aspects and examples provided herein. However, one of ordinary skill in the art will appreciate that these aspects can be implemented without these specific details. For clarity of explanation, in some cases, the present invention may be presented as including various functional blocks, which include functional blocks including devices, device components, steps or routines in the method implemented with a combination of software or hardware and software. In addition to the components shown in the figure and / or described herein, other components may also be used. For example, circuits, systems, networks, processes and other components may be shown as components in the form of block diagrams, so as not to obscure various aspects with unnecessary details. In other cases, in order to avoid confusing various aspects, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details.

[0177] The above various aspects may be described as a process or method depicted as a flow chart, a flowchart, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe an operation as a sequential process, many operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process terminates when its operations are completed, but there may be additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0178] The processes and methods according to the above examples can be implemented using computer executable instructions that are stored or otherwise available from computer readable media. For example, such instructions may include instructions and data that enable or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a specific function or group of functions. The used portion of the computer resources may be accessed via a network. Computer executable instructions may be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer readable media that may be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memory, USB devices provided with non-volatile memory, network storage devices, etc.

[0179] Devices implementing the processes and methods disclosed herein may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable medium or a machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptops, smart phones, mobile phones, tablet computers or other small personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functions described herein may also be implemented in peripheral devices or add-on cards. By way of further example, such functions may also be implemented on a circuit board between different chips or different processes executed in a single device.

[0180] Instructions, media for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0181] In the foregoing description, various aspects of the present application are described with reference to specific aspects of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative aspects of the present application have been described in detail herein, it should be understood that the inventive concept can be implemented and adopted differently in other ways, and the appended claims are intended to be interpreted as including such changes, unless limited by the prior art. The various features and aspects of the above-mentioned applications can be used alone or in combination. In addition, without departing from the broader spirit and scope of this specification, various aspects can be used in any number of environments and applications outside the environment and application described herein. Therefore, the description and the accompanying drawings should be regarded as illustrative, not restrictive. For ease of explanation, the method is described in a specific order. It should be understood that in alternative aspects, the method can be performed in an order different from the described order.

[0182] A person of ordinary skill in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0183] Where a component is described as being “configured to” perform a particular operation, such configuration may be achieved by designing electronic circuits or other hardware to perform the operation, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0184] The phrase "coupled to" refers to any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).

[0185] Claim language or other language reciting "at least one of" a set and / or "one or more" in a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" refers to A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" refers to A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one" in a set and / or "one or more" in a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" may refer to A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0186] The various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the various aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above with respect to their functionality. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Those skilled in the art may implement the described functions in different ways for each specific application, but such implementation decisions should not be interpreted as causing deviations from the scope of the present application.

[0187] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. These techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handsets, or integrated circuit devices with multiple uses, including applications in wireless communication device handsets and other devices. Any feature described as a module or component may be implemented together in an integrated logic device, or individually implemented as a discrete but interoperable logic device. If implemented in software, the technique may be implemented at least in part by a computer-readable data storage medium including a program code, which includes instructions for executing one or more of the above methods when executed. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a RAM (such as a synchronous dynamic random access memory (SDRAM)), a ROM, a non-volatile random access memory (NVRAM), an EEPROM, a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part through a computer-readable communication medium (such as a propagated signal or wave) that carries or communicates program code in the form of instructions or data structures and can be accessed, read and / or executed by a computer.

[0188] The program code can be executed by a processor, which may include one or more processors, such as one or more DSPs, general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor can be configured to perform any of the techniques described in this disclosure. A general-purpose processor can be a microprocessor; but in an alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein.

[0189] Illustrative examples of the present disclosure include:

[0190] Aspect 1. A method for processing one or more images, comprising: obtaining a first image captured using an image sensor, the first image being associated with a first exposure; obtaining a second image captured using the image sensor, the second image being associated with a second exposure longer than the first exposure; modifying a first region of the first image based on a first transformation, and modifying a second region of the first image based on a second transformation to generate a modified first image; and generating a merged image at least in part by merging the modified first image and the second image.

[0191] Aspect 2. The method according to aspect 1, wherein the image sensor is oriented in the same direction as a display used to display the preview image captured by the image sensor.

[0192] Aspect 3. A method according to any one of Aspects 1 to 2, wherein the first area is associated with an object at a first depth in the scene relative to the image sensor, and wherein the second area includes a background area at a second depth in the scene relative to the image sensor.

[0193] Aspect 4: The method according to any one of aspects 1 to 3 further includes: generating a first matrix for performing the first transformation; and generating a second matrix for performing the second transformation.

[0194] Aspect 5. A method according to any one of aspects 1 to 4, wherein the second matrix is ​​generated based on movement detected by a motion sensor between a first time when the first image is captured and a second time when the second image is captured.

[0195] Aspect 6. The method according to any one of aspects 1 to 5, wherein the motion sensor comprises a gyroscope sensor, and wherein the second transformation comprises a rotational transformation.

[0196] Aspect 7. The method according to any one of aspects 1 to 6, wherein generating the first matrix comprises: extracting first feature points from the first image; and extracting second feature points from the second image.

[0197] Clause 8: The method according to any one of clauses 1 to 7, further comprising: increasing the brightness of the first image based on an exposure ratio difference between the first image and the second image.

[0198] Aspect 9. The method according to any one of aspects 1 to 8 further includes: detecting an object in the second image; and determining a boundary area associated with the position of the object in the second image.

[0199] Aspect 10. The method according to any one of Aspects 1 to 9 further includes: identifying a subset of the first feature points within the boundary area; identifying a subset of the second feature points within the boundary area; and generating the first matrix based on the subset of the first feature points and the subset of the second feature points.

[0200] Aspect 11. According to the method of any one of aspects 1 to 10, based on the first matrix and the second matrix, a mixing transformation matrix for modifying the first region of the first image and the second region of the first image is generated.

[0201] Aspect 12. A method according to any one of Aspects 1 to 11, wherein generating the hybrid transformation matrix comprises: adding values ​​from the first matrix corresponding to at least the first region to the hybrid transformation matrix; and adding values ​​from the second matrix corresponding to at least the second region to the hybrid transformation matrix.

[0202] Aspect 13. The method according to any one of Aspects 1 to 12 further includes: determining a transition region between the first region and the second region based on the size of a boundary region associated with the position of an object in at least one of the first image or the second image; determining a value associated with the transition region based on the representation of the first matrix and the second matrix; and adding the value associated with the transition region to the mixing transformation matrix.

[0203] Aspect 14. The method according to any one of aspects 1 to 13, wherein the representation of the first matrix and the second matrix comprises a weighted average of the first matrix and the second matrix.

[0204] Aspect 15. A method according to any one of aspects 1 to 14, wherein the representation of the first matrix and the second matrix is ​​based on a proportional distance from an inner edge of the transition region to an outer edge of the transition region.

[0205] Clause 16. A method according to any one of clauses 1 to 15, wherein the first transformation comprises a translation matrix associated with movement of the image sensor during the obtaining of the first image and the obtaining of the second image.

[0206] Aspect 17. A method according to any one of aspects 1 to 16, wherein the merged image is an HDR image.

[0207] Aspect 18. An apparatus for processing one or more images, comprising at least one modem (e.g., implemented in a circuit), one (or more) processors coupled to a transceiver, and at least one memory coupled in communication with the at least one processor and storing processor-readable code. The processor-readable code, when executed by the at least one processor in conjunction with the at least one modem, is configured to: obtain a first image captured using an image sensor, the first image being associated with a first exposure; obtain a second image captured using the image sensor, the second image being associated with a second exposure longer than the first exposure; modify a first region of the first image based on a first transform, and modify a second region of the first image based on a second transform to generate a modified first image; and generate a merged image at least in part by merging the modified first image and the second image.

[0208] Clause 19. The apparatus of clause 18, wherein the image sensor is oriented in the same direction as a display used to display a preview image captured by the image sensor.

[0209] Aspect 20. An apparatus according to any one of Aspects 18 to 19, wherein the first area is associated with an object at a first depth in the scene relative to the image sensor, and wherein the second area includes a background area at a second depth in the scene relative to the image sensor.

[0210] Aspect 21. An apparatus according to any one of Aspects 18 to 20, wherein the processor-readable code, when executed by the at least one processor in combination with the at least one modem, is configured to: generate a first matrix for performing the first transformation; and generate a second matrix for performing the second transformation.

[0211] Aspect 22. An apparatus according to any one of aspects 18 to 21, wherein the second matrix is ​​generated based on movement detected by a motion sensor between a first time when the first image is captured and a second time when the second image is captured.

[0212] Aspect 23. The apparatus according to any one of aspects 18 to 22, wherein the motion sensor comprises a gyroscope sensor, and wherein the second transformation comprises a rotational transformation.

[0213] Aspect 24. An apparatus according to any one of Aspects 18 to 23, wherein the processor readable code, when executed by the at least one processor in combination with the at least one modem, is configured to: extract first feature points from the first image; and extract second feature points from the second image.

[0214] Aspect 25. An apparatus according to any one of Aspects 18 to 24, wherein the processor-readable code, when executed by the at least one processor in combination with the at least one modem, is configured to: increase the brightness of the first image based on an exposure ratio difference between the first image and the second image.

[0215] Aspect 26. An apparatus according to any one of Aspects 18 to 25, wherein the processor-readable code, when executed by the at least one processor in combination with the at least one modem, is configured to: detect an object in the second image; and determine a boundary area associated with the position of the object in the second image.

[0216] Aspect 27. An apparatus according to any one of Aspects 18 to 26, wherein the processor-readable code, when executed by the at least one processor in combination with the at least one modem, is configured to: identify a subset of the first feature points within the boundary area; identify a subset of the second feature points within the boundary area; and generate the first matrix based on the subset of the first feature points and the subset of the second feature points.

[0217] Aspect 28. An apparatus according to any one of Aspects 18 to 27, wherein a mixing transformation matrix for modifying the first region of the first image and the second region of the first image is based on the first matrix and the second matrix.

[0218] Aspect 29. An apparatus according to any one of Aspects 18 to 28, wherein the processor-readable code, when executed by the at least one processor in combination with the at least one modem, is configured to: add values ​​from the first matrix corresponding to at least the first region to the hybrid transformation matrix; and add values ​​from the second matrix corresponding to at least the second region to the hybrid transformation matrix.

[0219] Aspect 30. An apparatus according to any one of Aspects 18 to 29, wherein the processor-readable code, when executed by the at least one processor in combination with the at least one modem, is configured to: determine a transition region between the first region and the second region based on a size of a boundary region associated with a position of an object in at least one of the first image or the second image; determine a value associated with the transition region based on representations of the first matrix and the second matrix; and add the value associated with the transition region to the mixing transformation matrix.

[0220] Aspect 31. An apparatus according to any one of aspects 18 to 30, wherein the representation of the first matrix and the second matrix comprises a weighted average of the first matrix and the first matrix.

[0221] Aspect 32. An apparatus according to any one of aspects 18 to 31, wherein the representation of the first matrix and the second matrix is ​​based on a proportional distance from an inner edge of the transition region to an outer edge of the transition region.

[0222] Clause 33. An apparatus according to any one of Clauses 18 to 32, wherein the first transformation comprises a translation matrix associated with movement of the image sensor during the obtaining of the first image and the obtaining of the second image.

[0223] Clause 34. An apparatus according to any one of clauses 18 to 33, wherein the merged image is an HDR image.

[0224] Aspect 35. A non-transitory computer-readable medium comprising instructions, which when executed by one or more processors cause the one or more processors to perform the operations according to any one of aspects 1 to 17.

[0225] Aspect 36. An apparatus comprising means for performing the operations according to any one of aspects 1 to 17.

Claims

1. A method for processing one or more images, comprising: obtaining a first image captured using an image sensor, the first image being associated with a first exposure; obtaining a second image captured using the image sensor, the second image being associated with a second exposure that is longer than the first exposure; modifying a first region of the first image based on a first transformation and modifying a second region of the first image based on a second transformation to generate a modified first image; as well as A merged image is generated at least in part by merging the modified first image and the second image.

2. The method according to claim 1, wherein: The image sensor is oriented in the same direction as a display used to display a preview image captured by the image sensor.

3. The method according to claim 1, wherein: The first region is associated with an object at a first depth in a scene relative to the image sensor, and wherein the second region includes a background region at a second depth in the scene relative to the image sensor.

4. The method according to claim 1, further comprising: generating a first matrix for performing the first transformation; as well as A second matrix for performing the second transformation is generated.

5. The method according to claim 4, wherein: The second matrix is ​​generated based on movement detected by a motion sensor between a first time when the first image is captured and a second time when the second image is captured.

6. The method according to claim 5, wherein: The motion sensor comprises a gyroscopic sensor, and wherein the second transform comprises a rotational transform.

7. The method according to claim 4, wherein: Generating the first matrix includes: extracting first feature points from the first image; and Second feature points are extracted from the second image.

8. The method according to claim 7, further comprising: The brightness of the first image is increased based on an exposure ratio difference between the first image and the second image.

9. The method according to claim 7, further comprising: detecting an object in the second image; as well as A bounding region associated with the location of the object in the second image is determined.

10. The method according to claim 9, further comprising: identifying a subset of the first feature points within the bounding area; identifying a subset of the second feature points within the bounding area; as well as The first matrix is ​​generated based on a subset of the first feature points and a subset of the second feature points.

11. The method according to claim 4, further comprising: Based on the first matrix and the second matrix, a mixing transformation matrix for modifying the first region of the first image and the second region of the first image is generated.

12. The method according to claim 11, wherein: Generating the hybrid transformation matrix comprises: adding values ​​from the first matrix corresponding to at least the first region to the hybrid transform matrix; and Values ​​from the second matrix corresponding to at least the second region are added to the blending transform matrix.

13. The method according to claim 12, further comprising: determining a transition region between the first region and the second region based on a size of a bounding region associated with a location of an object in at least one of the first image or the second image; determining a value associated with the transition region based on representations of the first matrix and the second matrix; as well as The values ​​associated with the transition region are added to the blending transformation matrix.

14. The method according to claim 13, wherein: The representation of the first matrix and the second matrix comprises a weighted average of the first matrix and the first matrix.

15. The method according to claim 13, wherein: The representation of the first matrix and the second matrix is ​​based on a proportional distance from an inner edge of the transition region to an outer edge of the transition region.

16. The method according to claim 1, wherein: The first transformation comprises a translation matrix associated with movement of the image sensor during the obtaining of the first image and the obtaining of the second image.

17. The method according to claim 1, wherein: The merged image is a high dynamic range (HDR) image.

18. An apparatus for processing one or more images, the apparatus comprising: at least one memory; as well as at least one processor coupled to the at least one memory, wherein the at least one processor is configured to: obtaining a first image captured using an image sensor, the first image being associated with a first exposure; obtaining a second image captured using the image sensor, the second image being associated with a second exposure that is longer than the first exposure; modifying a first region of the first image based on a first transformation and modifying a second region of the first image based on a second transformation to generate a modified first image; and A merged image is generated at least in part by merging the modified first image and the second image.

19. The device according to claim 18, wherein: The image sensor is oriented in the same direction as a display used to display a preview image captured by the image sensor.

20. The device according to claim 18, wherein The first region is associated with an object at a first depth in a scene relative to the image sensor, and wherein the second region includes a background region at a second depth in the scene relative to the image sensor.

21. The device according to claim 18, wherein The at least one processor is configured to: generating a first matrix for performing the first transformation; and A second matrix for performing the second transformation is generated.

22. The device according to claim 21, wherein The at least one processor is configured to generate the second matrix based on movement detected by a motion sensor between a first time when the first image is captured and a second time when the second image is captured.

23. The device according to claim 22, wherein: The motion sensor comprises a gyroscopic sensor, and wherein the second transform comprises a rotational transform.

24. The device according to claim 21, wherein The at least one processor is configured to: extracting first feature points from the first image; and Second feature points are extracted from the second image.

25. The device according to claim 24, wherein: The at least one processor is configured to: The brightness of the first image is increased based on an exposure ratio difference between the first image and the second image.

26. The device according to claim 24, wherein: The at least one processor is configured to: detecting an object in the second image; and A bounding region associated with the location of the object in the second image is determined.

27. The device according to claim 26, wherein: The at least one processor is configured to: identifying a subset of the first feature points within the bounding area; identifying a subset of the second feature points within the bounding area; as well as The first matrix is ​​generated based on a subset of the first feature points and a subset of the second feature points.

28. The device according to claim 21, wherein The at least one processor is configured to: A mixing transformation matrix for modifying the first region of the first image and the second region of the first image is generated based on the first matrix and the second matrix.

29. The device according to claim 28, wherein The at least one processor is configured to: adding values ​​from the first matrix corresponding to at least the first region to the blending transformation matrix; as well as Values ​​from the second matrix corresponding to at least the second region are added to the blending transform matrix.

30. The device according to claim 29, wherein: The at least one processor is configured to: determining a transition region between the first region and the second region based on a size of a bounding region associated with a location of an object in at least one of the first image or the second image; determining a value associated with the transition region based on representations of the first matrix and the second matrix; as well as The values ​​associated with the transition region are added to the blending transformation matrix.

Images