Method and apparatus for image capture

Abstract

Systems and methods for image fusion are described. In an example, a first image of a scene associated with a first parameter may be obtained, and a second image of the scene associated with a second parameter may be obtained. The first image and the second image may be fused to obtain at least one first fused image. A third image of the scene associated with a third parameter may be obtained, and a fourth image of the scene associated with the fourth image may be obtained. At least one parameter associated with the images for obtaining the third and fourth images may be adjusted, wherein the adjustment may be based at least in part on a field of view of the at least one first fused image. The third image and the fourth image may be fused to obtain a second fused image.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 305,304, filed April 21, 2023, entitled “METHOD AND APPARATUS FOR IMAGECAPTURING,” the disclosure of which is incorporated herein by reference. Background Technology

[0003] This disclosure relates to various aspects of image processing, and more specifically to methods and apparatus for high dynamic range (HDR) image capture. Over the past few decades, the use of electronic devices has become ubiquitous. Advances in electronic technology have reduced the cost of increasingly complex and useful electronic devices. Cost reduction and consumer demand have led to a surge in the use of electronic devices, making them virtually ubiquitous in modern society. As the use of electronic devices has expanded, so too has the demand for new functions and improved features. Specifically, electronic devices that perform functions faster, more efficiently, or with higher quality are often sought after.

[0004] Devices such as digital cameras, mobile phones with embedded cameras, or other camera or sensor devices can be used to create and store images of a scene. Recently, multi-exposure HDR image capture has become highly relevant in photography and videography. This involves taking images of the same scene at different exposure levels and then combining them. Combining multiple images in this way results in an image with a greater dynamic range than is possible by capturing a single image. This technique can also be used to capture video by taking multiple exposures and combining them for each frame. Many mobile phones have an automated HDR feature that relies on computational imaging techniques to capture and combine multiple exposures. Recently, the use of pixel binning techniques to obtain HDR images from a single shot has also become common. As known in the art, pixel binning involves grouping adjacent pixels together (merging) to form superpixels. This merging occurs before the output of the superpixels is converted into digital information and can provide better low-light performance at the cost of a certain image resolution. As previously described, an HDR image is made by fusing at least two images. These images can be captured individually; however, when using pixel binning, images can also be composed of sub-pixels of a superpixel, where one image is composed of one set of sub-pixels of the superpixel, and another image is composed of another set of sub-pixels of the superpixel, where the first and second sets are different. This is possible because each of the individual sub-pixels of a superpixel can be independently tuned to better capture highlights or shadows before their outputs are combined. The result is a superpixel capable of capturing both dark shadows and bright highlights simultaneously. This real-time HDR feature makes it easier to capture HDR images of moving objects, while also enabling HDR video capture.

[0005] However, combining or fusing at least two images, captured individually or in subpixel form, is a computationally intensive process, and capturing images in this way can consume significantly more power or device resources than is preferred by the device user, especially if the device is a mobile device with limited battery power. Therefore, methods and apparatuses that improve the efficiency of capturing HDR images may be needed. Summary of the Invention

[0006] A method, apparatus, and non-transitory computer-readable medium for improving image capture and fusion are described. The method, apparatus, and non-transitory computer-readable medium may relate to HDR image capture. However, the techniques described herein are equally applicable to other image capture techniques that rely on the combination of more than one image. Furthermore, although images are mentioned, those skilled in the art will recognize that the techniques described herein are equally applicable to video frames as image sequences.

[0007] The following is a simplified summary of the invention relating to one or more aspects disclosed herein. Therefore, this summary should not be considered an exhaustive overview relating to all contemplated aspects, nor should it be considered to identify key or decisive elements relating to all contemplated aspects or to depict the scope associated with any particular aspect. Thus, the sole purpose of this summary is to present, in a concise form, certain concepts relating to one or more aspects of the technology disclosed herein before the detailed description presented below. The invention is defined by the claims. Embodiments and aspects not falling within the scope of the claims are merely examples used to explain the invention.

[0008] In one aspect, a method for image fusion is described. The method includes obtaining a first image of a scene associated with a first parameter, and obtaining a second image of the scene associated with a second parameter. Although only two images are described, obtaining more than two images is also contemplated. These two images can be obtained separately by at least one or more image capture devices, or they can be two shots of the same scene, or the images can represent different sets of subpixels of a single shot taken by an image capture device. The method may further include fusing the first and second images to obtain at least one first fused image. Subsequently, a third image of the scene associated with a third parameter and a fourth image of the scene associated with a fourth image can be obtained. Again, only two images are named here, but more than two images are contemplated. Furthermore, the third and fourth images can be obtained separately, or they can again be different sets of subpixels. At least one parameter associated with these images for obtaining the third and fourth images can be adjusted, wherein the adjustment is at least partially based on the field of view of at least one first fused image. The method can then include fusing the third and fourth images to obtain a second fused image.

[0009] In one aspect, the method may further include previewing at least one first fused image before obtaining the third and fourth images. To preview the fused image, tone mapping may be performed, which adapts the fused image to the corresponding display of the device on which the first fused image is displayed.

[0010] In one aspect of the method, the parameter associated with the image is the output pixel count value. The pixel count value is directly related to the image resolution. A higher count value means that the image capture device uses more pixels individually to acquire the image, resulting in a high-resolution image. A lower count value indicates that fewer pixels are typically used or more pixels are merged to form superpixels. The count value itself can also be related to the merging mode used. The parameter can also be referred to as the setting used to acquire the image. This setting can thus be used by at least one image capture device to acquire the image.

[0011] In one aspect, the method may further include adjusting at least one of a third or fourth parameter based on the power mode. The power mode may be the power mode of the entire device used to capture the image, or the power mode of the image capture device itself, which forms part of the entire device used to capture the image. Thus, in a low power mode, at least one of the parameters can be adjusted to have a lower value than in a normal power mode. For example, when the parameter relates to a pixel count value, a lower count value indicates that more pixels are merged, in which case the image is obtained at a lower resolution, while a higher value can indicate that fewer pixels are merged into no pixels at all, in which case the image is obtained at full resolution.

[0012] In one aspect of the method, the adjustment can be based on determining whether a threshold is exceeded. The threshold can be associated with a power threshold of the device. For example, once the power threshold is exceeded, adjustment can be triggered. Additionally or alternatively, the threshold can also be associated with the resolution of the first fused image in the field of view. Thus, the threshold can be associated with a scaling factor used in the field of view. For example, if the first fused image is obtained by fusing a first image obtained at full resolution and a second image obtained at half resolution, a scaling factor of two will still not result in noticeable artifacts, while a scaling factor of three may result in noticeable artifacts, necessitating an increase in the resolution of at least one of the third or fourth images. In this example, the threshold would be associated with the scaling factor. However, it should also be understood by those skilled in the art that, in the foregoing example, which should not be considered limiting, the threshold of scaling factor two also depends on the display capability of displaying the first fused image. Depending on the display capability, noticeable artifacts may only become visible at or before a higher scaling factor.

[0013] In one aspect of the method, different exposure values ​​can be used to capture the first and second images, and / or the third and fourth images.

[0014] In one aspect of the method, the fusion of the first image and the second image includes fusing the first image and the second image on a pixel-by-pixel basis, and / or the fusion of the third image and the fourth image includes fusing the third image and the fourth image on a pixel-by-pixel basis.

[0015] In one aspect, the method may further include receiving input from a user associated with a region of interest (ROI) in the first fused image, and determining a field of view based on the ROI. The user input may be a zoom-in or zoom-out action. In the case of a zoom-in action, the zoom factor can be changed according to the ROI, and thus the field of view of the first fused image can be changed. For example, a user may zoom in on a specific ROI in the first fused image, which will change the zoom factor of the first fused image and the field of view presented to the user. The user input may be received via a touchscreen of the device. The user's action can thus have an effect on the adjustment of at least a third or fourth parameter used to obtain the third and fourth images. For example, if it is zoomed in and the parameters used to obtain the first and second images result in significant artifacts in the new field of view of the first fused image, at least one of the third or fourth parameters will be adjusted to obtain the third and fourth images such that the resulting second fused image will not exhibit significant artifacts. In this case, the adjustment may be to use a higher pixel count value (e.g., higher resolution) as used to obtain the first or second image to obtain at least one of the third or fourth images. Conversely, it is also possible that, if the user zooms out, the resolution of at least one of the third or fourth images can be reduced compared to the resolution used for the first and / or second images.

[0016] In one aspect, the method may further include storing at least one second fused image. Storage may include storing the image in the device's memory and / or uploading the second fused image to a cloud storage system.

[0017] In one aspect of the method, obtaining may include acquiring an image from at least one image capturing device. The at least one image capturing device may be configured to capture images with different pixel count values. Thus, the image capturing device may include an image sensor capable of acquiring images with different pixel count values, applying different merging modes respectively, so that images with different resolutions can be obtained.

[0018] Although the aspects of the method are described in separate paragraphs, it should be assumed that all aspects of the method are composable, and no aspect excludes another. Therefore, each aspect described herein can be combined, individually or together, with the general aspects of the method described.

[0019] In one aspect, an apparatus for image capture and fusion is described. The apparatus includes at least one image capture device configured to acquire at least a first image of a scene associated with a first parameter, a second image of a scene associated with a second parameter, a third image of a scene associated with a third parameter, and a fourth image of a scene associated with a fourth parameter. It is contemplated that the described images are acquired individually by the at least one image capture device, or that the first and second images, as well as the third and fourth images, are composed of sub-pixels of the two images. The apparatus also includes at least one or more processors configured to: fuse the first and second images to obtain a first fused image; adjust at least one of a third or fourth parameter to be used to obtain the third or fourth image, at least in part based on the field of view of the first fused image; and fuse the third and fourth images to obtain a second fused image. The apparatus may also include at least one memory for storing the fused image.

[0020] In one aspect of the apparatus, at least one image capture device can be configured to acquire images with different pixel count values, and parameters associated with the images can be related to the pixel count values ​​used to acquire the respective images. Thus, the pixel count values ​​are related to the resolution of the acquired images. This can also be associated with a merging mode used by the at least one image capture device.

[0021] In one aspect of the device, the adjustment can also depend on the device's power mode. Thus, in low-power mode, at least one of the third or fourth parameters can be adjusted to have a lower value than in normal power mode and / or compared to at least one of the first or second parameters. It is conceivable that more than one parameter can be adjusted, and that different parameters can be adjusted under different conditions.

[0022] In one aspect of the device, one or more processors may be configured to adjust at least one of a third or fourth parameter when a threshold is exceeded. Thus, the threshold may be associated with a resolution loss of the fused image in the field of view (e.g., noticeable artifacts in a magnified region of interest). Additionally or alternatively, the threshold may be associated with the power consumption of the device.

[0023] In one aspect of the device, the first and second images, and / or the third and fourth images, are captured at different exposure values.

[0024] In one aspect of the apparatus, one or more processors may be configured to fuse a first image and a second image on a pixel-by-pixel basis and / or fuse a third image and a fourth image on a pixel-by-pixel basis. Thus, the one or more processors may also be configured to: select a pixel from the first image or the second image for each pixel of the first fused image, and select a pixel from the third image or the fourth image for each pixel of the second fused image. It is also conceivable that the first image may be composed of sub-pixels of a superpixel image, and the second image may be composed of other sub-pixels of a superpixel image, and the one or more processors are configured to fuse the sub-pixels; and / or wherein the third image may be composed of sub-pixels of a superpixel image, and the fourth image may be composed of other sub-pixels of a superpixel image, and the one or more processors may be configured to fuse the sub-pixels.

[0025] In one aspect of the device, one or more processors may be configured to receive input from a user associated with a region of interest in a first fused image, and to determine a field of view based on the region of interest. The user input may be a zoom-in or zoom-out action. The user input may be received via the device's touchscreen.

[0026] In one aspect, the device can be a handheld camera or part of a mobile phone.

[0027] Although aspects of the apparatus are described in separate paragraphs, it should be assumed that all aspects of the apparatus are combinable, and that no aspect excludes another. Therefore, each aspect described herein can be combined, individually or together, with the general aspects of the apparatus described.

[0028] In one aspect, an apparatus for image capture and fusion is described. The apparatus includes: components for acquiring a first image of a scene associated with a first parameter; components for acquiring a second image of the scene associated with a second parameter; components for fusing the first and second images to obtain a first fused image; components for adjusting at least one of a third parameter or a fourth parameter to be used to acquire a third and a fourth image, wherein the adjustment is at least partially based on the field of view in at least one of the first fused images; components for acquiring a third image of the scene associated with the third parameter; components for acquiring a fourth image of the scene associated with a fourth parameter; and components for fusing the third and fourth images to obtain a second fused image.

[0029] In one aspect, a non-transitory computer-readable medium having instructions stored thereon is described. When executed by one or more processors, the instructions cause the one or more processors to: obtain a first image of a scene associated with a first parameter, and obtain a second image of the scene associated with a second parameter, fuse the first and second images to obtain a first fused image, obtain a third image of the scene associated with a third parameter, and obtain a fourth image of the scene associated with a fourth parameter, wherein at least one of the third or fourth parameters is adjusted, wherein the adjustment is at least partially based on the field of view in at least one of the first fused images, and fuse the third and fourth images to obtain a second fused image.

[0030] Based on the accompanying drawings and detailed description, other objects and advantages associated with the aspects disclosed herein will be apparent to those skilled in the art. Attached Figure Description

[0031] The accompanying drawings are provided to aid in describing various aspects of this disclosure, and are provided for illustrative purposes only and not to limit the various aspects. In the drawings, similar components or features may have the same reference numerals. Furthermore, various components of the same type may be distinguished by the letters attached to the reference numerals. If reference numerals only are used in the specification, the description may apply to any of the similar components having the same first reference numeral, regardless of the attached letters.

[0032] Figure 1 Aspects of an executable image capture and fusion apparatus according to the embodiments described herein are illustrated;

[0033] Figure 2 Examples Figure 1 The device is shown in various aspects, with the magnified view of the scene at a low resolution;

[0034] Figure 3 Examples Figure 1 The device is shown in various aspects, with the scene magnified in high resolution;

[0035] Figure 4 A decision graph is depicted for changing the parameters used to perform image capture according to the embodiment described herein from a normal power mode to a low power mode;

[0036] Figure 5 A method for performing image capture according to the embodiment described herein is depicted; and

[0037] Figure 6 Examples are shown that can be used as follows Figure 1 The device shown is one aspect of an image capture and processing system that performs image capture according to the embodiment described herein. Detailed Implementation

[0038] Various aspects of this disclosure are provided in the following description and accompanying drawings of various examples provided for illustrative purposes. Alternative aspects may be devised without departing from the scope of this disclosure. Additionally, well-known elements of this disclosure will not be described in detail or will be omitted so as not to obscure the relevant details of this disclosure. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the scope of this application as set forth in the appended claims.

[0039] The terms “exemplary” and / or “example” are used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” and / or “example” is not necessarily to be construed as superior to or better than other aspects. Similarly, the term “aspects of this disclosure” does not require that all aspects of this disclosure include the features, advantages, or modes of operation discussed.

[0040] Those skilled in the art will understand that any of a variety of different techniques and methods can be used to represent the information and signals described below. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the following description can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof, depending in part on the specific application, in part on the desired design, and in part on the corresponding technology, etc.

[0041] Furthermore, many aspects are described according to a sequence of actions to be performed by elements of, for example, a computing device. It will be appreciated that the various actions described herein can be performed by a particular circuit (e.g., an application-specific integrated circuit (ASIC)), by program instructions executed by one or more processors, or by a combination of both. Additionally, the sequence of actions described herein can be considered to be entirely embodied in any form of non-transitory computer-readable storage medium storing a corresponding set of computer instructions that, when executed, will cause or command the associated processor of the device to perform the functionality described herein. Therefore, various aspects of this disclosure can be embodied in a variety of different forms, all of which are contemplated within the scope of the claimed subject matter. Furthermore, for each aspect described herein, any corresponding form of any such aspect may be described herein as, for example, "a logical component configured to perform the described actions."

[0042] As used herein, a “scene” refers to a location or view of an area captured by the device’s camera in one or more related images. Therefore, related images of a film captured by the device can be images of the same scene, even if each image does not cover exactly the same space. In some embodiments, a scene or a portion of a scene may also refer to a location or view that includes only a portion of the area or field of view captured by the device.

[0043] A camera is a device that uses an image sensor to receive light and capture images, such as still images or video frames. Examples of sensors in a camera may include semiconductor charge-coupled devices (CCDs) in complementary metal-oxide-semiconductor (CMOS) or N-type metal-oxide-semiconductor (NMOS) and active pixel sensors. The terms “image,” “image frame,” and “frame” are used interchangeably herein. A camera may include a processor, such as an image signal processor (ISP), which receives and processes one or more images. For example, a raw image captured by a camera sensor may be processed by an ISP to generate a final image. The processing performed by the ISP may be performed by multiple filters or processing blocks applied to the captured image, such as denoising or noise filtering, edge enhancement, color balancing, contrast adjustment, intensity adjustment (such as darkening or brightening), tone adjustment, etc. Image processing blocks or modules may include lens / sensor noise correction, Bayer filters, de-mosaicing, color conversion, correction or enhancement / suppression of image attributes, noise reduction filters, sharpening filters, etc.

[0044] Cameras can be configured with various image capture and image processing settings. Applying different settings produces images with different appearances. These settings can also be referred to as parameters. Some camera settings are determined and applied before or during image capture, such as ISO, exposure time, resolution, aperture size, aperture stops, shutter speed, focus, and gain. Other camera settings can configure post-processing of the image, such as changes to contrast, brightness, saturation, sharpness, levels, curves, or color.

[0045] In many camera systems, the host processor (HP) (also known in some cases as the application processor (AP)) is used to dynamically configure the image sensor with new parameters. The HP is also used to dynamically configure the parameter settings of the ISP pipeline to match the exact settings of the input image sensor frames, thereby correctly processing the image data.

[0046] When a camera system is used to capture HDR images, at least two or more images of the scene are obtained by the image sensor using bracketing exposure or conversion gain with a single exposure time, and then combined by the ISP or HP to produce images that preserve local contrast for natural rendering or exaggerate local contrast for artistic effects. HDR is very useful for capturing scenes containing very bright, direct sunlight, or extreme shadows. The combination of images can also be referred to as image fusion or image merging. Thus, images can be captured individually by the image sensor, in which case it can be said that two shots of the scene were taken, or if merging is used, at least two images can also be represented by different sets of subpixels of superpixels. However, it should also be conceivable that different merging modes can be used to capture individual images for each of the individually captured images. Therefore, different numbers of combined (i.e., merged) pixels can be used to capture each image.

[0047] Acquiring more than one image from an image sensor and providing a preview of the fused image to the user consumes significant device resources. This is especially true when the device is a mobile device, which may be battery-powered, making it necessary to avoid any unnecessary power consumption. The disclosure of the present invention addresses this need by providing a method, apparatus, and non-transitory computer-readable medium for acquiring and fusing images in a power-efficient manner.

[0048] Figure 1 An example is illustrated of a device 110 including a display 130. Device 110 may be a dedicated camera device or a multi-functional device, such as a mobile phone with an integrated camera including at least one image sensor. In other embodiments, device 110 may be any device, such as a laptop computer, telephone, tablet computer, phablet, desktop computer with a built-in or plug-in camera, or any other such computing device.

[0049] like Figure 1 As shown, the display 130 of device 110 is filled with an image 125 of scene 100, which can be captured by the image capture device of device 110. The image capture device is not in... Figure 1 As shown, it is on the back of device 110, and therefore on the side opposite to display 130. The image capture device will be explained in more detail below. Display 130 may be based on liquid crystal display (LCD) technology, light-emitting polymer display (LPD) technology, organic light-emitting diode (OLED) technology, or some other display technology.

[0050] Scene 100 shown here is a rural area with roads. Image 125, as shown on display 130 of device 110, is a combination of several images 115A, 115B, and 115C of scene 100. Thus, images 115A, 115B, and 115C can be acquired by different image sensors or by the same image sensor of the image capture device of device 110. It is also conceivable, for example, that images 115A and 115B are acquired from the same image sensor, while image 115C is acquired from another image sensor. However, this should not be construed as limiting; images 115A, 115B, and 115C can be acquired by any number of image sensors in any possible combination. It is also conceivable that each of images 115A, 115B, and 115C consists of different sets of subpixels of a superpixel image. Each of the three images 115A, 115B, and 115C can be associated with at least one parameter, which can be related to the settings of the image sensors. For example, at least one parameter could be the exposure time of the corresponding image and / or the pixel count value that the image sensor is using to acquire the corresponding image. It will be apparent to those skilled in the art that although three combined images 115A, 115B, and 115C are shown here, any number of images can be combined. A combination is already given if at least two images are combined (e.g., images 115A and 115B). Therefore, in the following text, only the combination of images 115A and 115B is mentioned, but it should be contemplated that any number of images can be combined. The combination of images can be accomplished in a known HDR manner. Thus, in order to display the fused image on the display 130 of device 110, tone mapping can be performed, which maps one set of colors to another set of colors to approximate the appearance of the HDR image on display 130. This is necessary because mobile phone displays typically have a limited dynamic range that is insufficient to reproduce the full range of light intensity present in an HDR image.

[0051] A capture button 150 is also shown on the display 130, which is introduced by the software of the device 110 to give the user the feel and sensation of a button that can be pressed to capture an image. In this sense, capturing the image will be accompanied by storing a combined image 115A and 115B of scene 100 into the device's memory. Therefore, in Figure 1 At the illustrated time instance, only a preview of the image of scene 100 is shown, which has not yet been stored in the memory of device 110. This is also highlighted by displaying the word "preview" on display 130. However, it should be assumed that this is done for illustrative purposes only and is not intended to be shown on the actual display 130 of the actual device 110. As with modern mobile phones, a preview of the image to be captured is displayed on display 130 before the image is captured and permanently saved to the memory of device 110 and / or uploaded to cloud storage.

[0052] A dashed rectangle 140 is also shown on display 130, indicating the user's region of interest (ROI). It should be understood that the dashed line will not need to be shown on the actual display 130 of the actual device 110, but may be shown on the actual display; however, it is used here only for illustrative purposes. Region 140 is known to be of interest because the user has already performed a zoom-in action 145 within that region. The display 130 can be used as a user interface, indicating the zoom-in action by illuminating the sliding separation of the thumb and forefinger on the display 130. The user interface can be a capacitive or resistive touchscreen and can be sensitive to tactile and / or sensory contact with the user. In such embodiments, display 130 may include a multi-touch sensitive display. Alternatively, the region of interest 140 can also be obtained around a predefined area of ​​dual touches on the user interface. Thus, preset zooming can be performed based on dual touches. However, other user actions to define the region of interest are also contemplated. For example, drawing a circle within the region of interest may be sufficient to define it.

[0053] Figure 2 Examples Figure 1 The device 110 thus displays the region of interest on the display 130, therefore... Figure 1 The area within which it is magnified. In other words, the field of view has expanded from... Figure 1 The complete scene shown has been changed to, as follows: Figure 2 The scene shown is a portion of the image. It could also be said that the scaling factor has changed. Figure 1 In this context, the scaling factor can be one, while in... Figure 2 In this context, the scaling factor has been changed to a value greater than or equal to two. Again, the dashed lines here are for illustrative purposes only, to indicate how the image 225 displayed on monitor 130 is... Figure 1 The region of interest is shown in the example. As can be seen here, image 225 is very blurry. In fact, in Figure 1 The road, clearly visible in image 125, is almost indistinguishable in image 225. Furthermore, as... Figure 1 The mountains and clouds in the rural scene 100 shown are difficult to discern. Such a loss of resolution seems unacceptable because if the image 225 of the region of interest were captured in this way, it would be perceived by the user as poor image quality.

[0054] Since such resolution loss is unacceptable, embodiments of the present invention suggest adjusting at least one parameter of the image sensor to obtain, for example... Figure 1 At least one of the images 115A and 115B shown, these two images are used to present Figure 1 The combined image 125 and Figure 2The magnified region of interest is defined in the image. Therefore, the image sensor can initially operate in a low-power mode and can have already acquired at least one or both of images 115A and 115B at a low-power setting. For example, image 115A can be acquired using a first resolution, and image 115B can be acquired using a second resolution, where the first and second resolutions are different, and at least one of the first and second resolutions can be lower than the full resolution capability of the image sensor. Alternatively, one image may have a lower pixel count than the other. For example, image 115A may have been acquired with a high pixel count, while the second image 115B may have been acquired with a lower pixel count, for example, by merging more pixels. Therefore, it can also be said that the first image 115A can be acquired using the full resolution capability of the image sensor, while image 115B can be acquired at a lower resolution. The lower resolution can be half, a quarter, or any other fraction of the full resolution capability. However, it is also conceivable that each image can be acquired using any resolution setting of the image sensor, as long as they are different and as long as at least one is lower than the full resolution capability of the image sensor. This allows for a faster and more power-efficient combination of images 115A and 115B, which would be ideal if both were acquired at full resolution. This is due to the fact that less data must be processed when combining images 115A and 115B, which in turn reduces the power consumption required by the ISP or HP to combine images 115A and 115B. Figure 1 In this process, the combined image 125 still has sufficient resolution, even though at least one of images 115A and 115B was acquired at a resolution lower than the full resolution capability of the image sensor. However, once the user zooms in on it within the region of interest 140, and thus the field of view changes, the resulting combination of images 115A and 115B suffers an unacceptable loss of resolution due to the fact that at least one of images 115A and 115B was not acquired at full resolution. Because of this unacceptable loss of resolution, at least one setting of the image sensor needs to be adjusted to obtain two additional images for further fusion.

[0055] It is conceivable that device 110 always starts in a low-power mode setting, where a parameter with a value lower than the full capacity associated with that parameter is used to acquire at least one of the two images 115A and 115B. For example, if the parameter is the pixel count value used by the image sensor to acquire the image, then from the outset, a lower pixel count value can be used to acquire at least one of the two images to save power compared to the full pixel count capability of the image sensor. This may not be a problem, because the initial settings may no longer be sufficient, as the user is initially presented with the widest possible field of view in the preview, and only when the user interacts with that preview, for example, by zooming in to select a specific region of interest, and the parameters utilized in acquiring at least one of the images need to be adjusted to produce better results.

[0056] Figure 3 An example is shown with a display 130. Figure 1 Device 110. On display 130, a combined image 325 is shown, which is generated by combining at least two other images, but wherein, for at least one of the images, the associated parameters have been adjusted compared to those used for images 115A and 115B. In the embodiment example shown here, the pixel count values ​​used by the image sensor to acquire at least one of images 115A and 115B have been adjusted so that the full pixel count capability of the image sensor is used to acquire the two other images. Alternatively, full resolution is used to capture the two other images. This increases the power requirements of the ISP and / or HP, but results in a clear, high-resolution fused image 325 of the region of interest 140. Figure 3 What we see in the middle, when with Figure 2 Compared to acquiring two additional images at a higher resolution, the roads, mountains, and clouds of scene 100 are clearly visible in the resulting combined image 325, so that when the user now presses the virtual capture button 150, the combined image 325 is stored at a tolerable resolution.

[0057] Figures 1 to 3 The process illustrated can be described as an image capture technique where the parameters associated with the acquired image depend on a scaling factor characterizing the field of view. The field of view can also be referred to as the region of the image that the user wants to focus on. This focusing can be performed through user interaction, such as a zoom-in action on an image preview. The region of interest is therefore directly associated with the user interaction performed by the user (e.g., the zoom-in action). According to the technique described herein, the user's interaction with the image preview influences the parameters used to acquire the image (particularly images used for HDR fusion).

[0058] Although the example above describes a zoom-in action that increases the pixel count value of the other image used to obtain the fused image, the opposite operation can also be performed. In the case of a zoom-out action, if the resulting fused image still has a tolerable resolution, it can be determined that the other image can be obtained using a lower pixel count value for at least one of the other images. Using a lower pixel count value allows for power savings.

[0059] Figure 4 A decision graph 400 for adjusting parameters used to obtain images to be fused is illustrated according to one aspect of the invention. The decision graph begins at box 410, which can also be referred to as the initial starting point of the decision process according to the invention. In this box, the camera's image sensor outputs at least two images, each captured using at least one parameter. The at least one parameter may be associated with a pixel count value used to obtain the image. Furthermore, other parameters are conceivable, such as exposure time, gain, or other parameters among the settings described above used by the image sensor. For the sake of better understanding, only the pixel count value or resolution is referred to hereinafter, however, this should not be construed as limiting. After the image sensor has obtained at least two images, they are fused in box 420. Fusion can be performed pixel-by-pixel. Then, in box 430, the fused image is previewed. For example, the fused image can be displayed as... Figure 1 The device 110 is shown on display 130. If input is detected at box 440, the decision process proceeds to box 450. The input received in box 440 can be received from the user of device 110, for example, in the form of a zooming action performed on display 130. However, it is also conceivable that the input is received by software. For example, the field of view (e.g., the region of interest to which it should be zoomed) is provided by software for face recognition or for object tracking. It should be understood that the input can therefore originate from the user of device 110 or from software analyzing the image provided by device 110. At box 450, it is determined whether the fused image at the region of interest still has an acceptable resolution. Here, it can be determined whether the pixel count of the fused image in the region of interest is still above a threshold. The threshold can be set, for example, based on a identifiable resolution loss. For example, the user would identify blurring in the resulting image at the corresponding resolution or that it would not be visible to the user. The threshold can be associated, for example, with the amount of texture that is still visible in the region of interest. Furthermore, the threshold can also be associated with a scaling factor. Above a certain scaling factor, the fused image shown on the display may exhibit noticeable artifacts. Therefore, how to set the threshold can depend on the image sensor used, the initial parameters used to obtain the image, the display capability, and / or any combination thereof.

[0060] If the resolution of the fused image in the region of interest is still acceptable at box 450, the process returns to box 410 in the decision flow, and the loop begins again. This upper loop can also be referred to as the device's low-power mode. In this low-power mode, the image or at least one of the images obtained in box 410 can be associated with a parameter that does not use the full capabilities of the image sensor (e.g., the pixel counting capability of the image sensor). Not using the full capabilities of the image sensor (e.g., full resolution) allows for power-efficient fusion of the images at box 420. For example, the parameter associated with the first image could be full resolution, so the image sensor used to obtain the first image can be configured to use its full resolution capability to obtain the first image. The parameter associated with the second image could be a lower resolution, so the image sensor used to obtain the second image can be configured to use a resolution lower than its full resolution capability to obtain the second image. For example, the first parameter could be full resolution, and the second parameter could be half resolution. According to this example, the first image is obtained using the full resolution capability of the image sensor, while the second image is obtained using only half of the image sensor's resolution capability. This not only allows for power-efficient acquisition of images, since less data must be captured, but also allows for power-efficient fusion of images, since less data must be processed. Similarly, although only two images and two parameters are named here, it is possible to acquire and fuse more than two images, and each image can be associated with more than one parameter.

[0061] Low power mode can be used as a device (such as Figures 1 to 3 The device 110 shown is set to a default power mode to maintain low power consumption. This is also possible because, as long as no input is detected, the low power mode setting preferably ensures that the fused image at box 420 has sufficient quality without noticeable degradation or artifacts. Alternatively, the low power mode can be associated with parameter settings used by at least one image sensor that are lower than those used in normal power mode, which will be discussed further below.

[0062] If, at box 450, the resolution of the fused image in the region of interest is determined to be no longer acceptable, the decision process continues to box 460. In box 460, at least one parameter associated with at least one of the other images to be acquired is adjusted. For example, the settings of the image sensor regarding the resolution utilized for acquiring the third or fourth image can be increased compared to the settings used for the first and / or second image. Thus, at box 410, the first image can be acquired using the full resolution capability of the image sensor, while the second image is acquired using half the resolution capability of the image sensor. If this does not result in a fused image with an acceptable resolution in the region of interest, then at box 460, both the third and fourth images are acquired using the full resolution capability of the image sensor. Therefore, compared to acquiring a first image with full resolution and a second image with half resolution, now a third and fourth image, both with full resolution, are acquired. Thus, at least one parameter associated with one image is increased from half resolution to full resolution. Although only one parameter associated with one image is described for adjustment, more than one parameter can also be adjusted. Furthermore, corresponding parameters can also be adjusted for both the third and fourth images. The image obtained at box 460 is then fused in box 470. This will result in a different fused image than in box 420 because at least one associated parameter has been adjusted for at least one image. The fused image is then provided again as a preview in box 480. If the fused image produced at box 480 is acceptable, it can be captured and stored in memory and / or uploaded to the cloud. In this case, the decision-making process ends at box 480. However, if input is detected again in box 440, a decision regarding the acceptability of the resolution is again made in box 450. This lower decision loop can also be referred to as the normal power mode of the device because, in this case, the adjusted parameters result in the capture of more data and the need to process more data, as is the case in low power mode.

[0063] It should be understood that iterations can be made by referring to the lower decision loop of the normal power mode, specifically box 460, meaning that at least one parameter is further adjusted each time box 460 is encountered. For example, if at least one parameter is related to the resolution used by the image sensor to acquire the image, the resolution can be increased each time box 460 is encountered until the resolution of the fused image is acceptable or reaches the full capability of the image sensor. To avoid imposing too much overhead on the device, it should be assumed that the iterations may not exceed the power savings. Therefore, only a limited number of iterations can be allowed before adjusting the parameters to the full capability of the image sensor, since acquiring images and fusing them at the full capability of the image sensor may be more power-efficient than using more iterations.

[0064] It should be envisioned that all boxes except box 440, where input is detected, can operate automatically without any input. This allows for near real-time decision-making, enhancing the user experience. This means that if the user performs a zoom-in action (detected in box 440), this will result in a new preview of the region of interest with either unadjusted or adjusted parameters almost instantaneously, ensuring the user experiences no delay in the preview and a seamless user experience during image capture. To this end, the adjustments used in box 460 can be preset before image capture begins. Thus, the adjustments may already exist in the instructions of the ISP or HP's operating software, or they may be variable adjustments that can be set by the user. For example, if the parameters are resolution-related, the software instructions could be such that, in low-power mode, the full resolution capability of the image sensor is always used to acquire at least one image, while only half the resolution capability of the image sensor is used to acquire the other image. In normal power mode, the full resolution capability of the image sensor can be used for both images. However, other values ​​are also envisioned, and it is also envisioned that the user can set these corresponding values. Furthermore, it is also envisioned that these values ​​are directly related to the device's current power setting or the current remaining battery power. For example, if the device's power setting is configured to conserve as much power as possible or if battery power is already low, a lower value can be used. Conversely, if the device is in high-power mode or connected to a power source, a higher value can be used for the parameters from the outset. Thus, not only can the region of interest defined by user input change the parameters used to obtain the image for HDR image capture, but also the device's current power mode can be altered.

[0065] It should be envisioned that, although the above description describes the device switching from a low-power mode to a normal-power mode based on a zoom-in action, the reverse is also possible in the case of a zoom-out action, thus switching from a normal-power mode to a low-power mode. Therefore, if the user changes the field of view by zooming out, the current parameter settings can be adjusted using a lower setting of the image sensor's capabilities (e.g., using a lower resolution), resulting in less power consumption. This might be the case, for example, if reducing a parameter associated with at least one image still results in a fused image in which no noticeable artifacts are visible. For example, if the fused image is the result of two full-resolution images and the user zooms out, the fused image can also be obtained using a full-resolution image and a half-resolution image without any resolution loss. Therefore, at least one parameter associated with acquiring an image can be reduced. This again saves power because not only must less data be captured, but less data must be combined. Although only one parameter reduction is described, more than one parameter can be reduced, as in the above switching from low-power mode to normal-power mode. The reduction of at least one parameter can also be performed in a stepwise manner.

[0066] Figure 5 A detailed method 500 is shown, through which, such as Figure 1 The device of the mobile device shown (i.e., device 110) can implement image capture and fusion as described herein. In step 510, at least two first images of the scene are obtained, each of the at least two first images being associated with at least one parameter. In the embodiment example shown here, the first image is obtained at step 510A, and the second image is obtained at step 510B. As described above, the at least two obtained images can be used for HDR image capture. Each of the images can be associated with a parameter. As described above, the parameter can be a setting of the image sensor used to obtain at least one image. It is conceivable that different parameters can be used for each image. The parameter can be, for example, a pixel count value used by at least one image sensor to obtain the image. The pixel count value can also be associated with a merging mode used by the image sensor to capture the corresponding two images.

[0067] In step 520, the obtained at least two images are then fused to obtain at least one first fused image. In the embodiment example shown here, the first image and the second image are fused. The fusion of at least two images may include fusing at least two first images on a pixel-by-pixel basis.

[0068] The method may also optionally include step 530, in which the first blended image is provided as a preview. However, it is also conceivable to send the first image to another device and preview it there to a user. Previewing the first blended image includes displaying the image on the device itself or on the display of the device to which the first blended image is sent. Displaying the first blended image may include the ability to perform tone mapping to adapt the image to the display.

[0069] In step 540, input regarding the region of interest (ROI) is received. The ROI refers to a specific region of the first fused image. Using this information, the field of view can be determined at step 541. The ROI can be selected by the user of the device, for example, by marking the region on the device's touchscreen. Alternatively or additionally, the input can be a zoom-in or zoom-out action performed by the user, such as by moving two fingers apart or together on the device's touchscreen. However, the ROI can also be given in other ways, or defined through an automated process. For example, facial or object recognition software can define the ROI where a face or specific object is visible in the scene. The software can then provide input regarding the ROI. In a video sequence, the ROI can also be given by tracking people or objects. Thus, the received input can come from the user of the device and / or from the software.

[0070] In step 550, at least one of a third or fourth parameter used to obtain at least two second images in subsequent steps is adjusted. For example, the pixel count value used to obtain the image can be adjusted to obtain another fused image that still has sufficient resolution in the region of interest. This sufficient resolution can be defined by the amount of texture that is still visible. Other criteria can also be used to define when the parameters need to be adjusted. For example, the parameters can also be adjusted based on device power considerations. The parameters associated with the first and second images can be related to the third and fourth parameters. It can also be said that the third and fourth parameters are adjusted versions of the first and second parameters. Thus, the first parameter can be related to the third parameter, and the second parameter can be related to the fourth parameter. For example, the first image can be obtained at full resolution, where full resolution is indicated by the first parameter, and then the third image can also be obtained at full resolution indicated by the third parameter. The second image can be obtained at only half resolution indicated by the second parameter, and then the adjustment of the fourth parameter can cause the image capture device to obtain a fourth image with a resolution different from half the resolution used for the second image. For example, with the increase of parameters, the fourth image can be captured at a resolution higher than half the resolution (e.g., full resolution). With a reduced parameter, a fourth image can be captured at less than half the resolution (e.g., a quarter resolution). Therefore, in this example, only one parameter is adjusted. However, it is also conceivable that other correlations and relationships exist between parameters, and adjusting the parameter envisions all possible adjustments. In the context of resolution mentioned above, this refers to the corresponding pixel count value used by the image sensor.

[0071] As described above, the adjustment of at least one parameter can be based on power considerations. The device may default to a low-power mode and use lower values ​​for the parameters associated with the first and second images, as in normal power mode, where at least one parameter associated with the third or fourth image may be increased. Conversely, if the device is already operating in normal power mode and switches back to low power mode, at least one parameter associated with the third or fourth image may be decreased. Therefore, switching power modes can also lead to the adjustment of at least one parameter.

[0072] After adjusting at least one parameter in step 560, at least two second images are obtained, wherein at least one of the at least two images is associated with at least one adjusted parameter. In the embodiment example shown here, a third image is obtained at step 560A, and a fourth image is obtained at step 560B. If the parameter is related to the pixel count value used by the image sensor to obtain the image, the image sensor can use a different pixel count value setting than before the adjustment to capture the image after adjustment. This may involve the image sensor using a different pixel image for at least one of the second images compared to the first image, or using different pixel images for both of the at least two second images.

[0073] In step 570, at least two second images are then fused to obtain at least one second fused image. In the embodiment example shown here, a third and fourth image are fused. The second fused image differs from the first fused image in that at least one parameter associated with at least one of the second images is adjusted compared to obtaining at least two first images. Alternatively, the image sensor has applied different settings to at least one of the at least two second images compared to obtaining at least two first images.

[0074] The settings for the image sensor can be directly related to the device's power consumption. Therefore, a first power mode can be used when acquiring an image in step 510 and fusing at least two first images in step 520, compared to a second power mode used when acquiring an image in step 560 and fusing at least two second images in step 570. Thus, the first power mode can be associated with lower power consumption of the device, or vice versa, compared to the second power mode.

[0075] Figure 6 This is a block diagram illustrating the architecture of an image capture and processing system 600. The image capture and processing system 600 includes various components for capturing and processing images of a scene (e.g., an image of scene 100). The image capture and processing system 600 can be, for example, as shown below. Figures 1 to 3 This is part of the device 110 shown. The image capture and processing system 600 can capture individual images (or photographs), and / or can capture video including multiple images (or video frames) in a specific sequence. The lens 615A of the system 600 faces the scene and receives light from the scene. The lens 615A bends the light toward the image sensor 630. The light received by the lens 615A passes through an aperture controlled by one or more control mechanisms 620 and is received by the image sensor 630.

[0076] One or more control mechanisms 620 may control exposure, focus, and / or zoom based on information from image sensor 630 and / or image processor 650. One or more control mechanisms 620 may include multiple mechanisms and components; for example, control mechanism 620 may include one or more exposure control mechanisms 625A and one or more merging mode control mechanisms 625B. One or more control mechanisms 620 may also include additional control mechanisms besides those illustrated, such as controls for gain, flash, depth of field, and / or other image capture attributes.

[0077] The exposure control mechanism 625A of the control mechanism 620 can obtain the exposure settings. In some cases, the exposure control mechanism 625A stores the exposure settings in a memory register. Based on the exposure settings, the exposure control mechanism 625A can control the aperture size (e.g., aperture size or aperture order), the duration of aperture opening (e.g., exposure time or shutter speed), and the sensitivity of the image sensor 630 (e.g., ISO speed or film speed). The merging mode control mechanism 625B can control the merging mode of the application, such as how many pixels are combined together to form a superpixel.

[0078] Image sensor 630 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in the image generated by image sensor 630. In some cases, different photodiodes may be covered by different color filters, and thus light matching the color of the color filter covering the photodiode can be measured. For example, Bayer color filters include red, blue, and green color filters, where each pixel of the image is generated based on red light data from at least one photodiode covered by the red color filter, blue light data from at least one photodiode covered by the blue color filter, and green light data from at least one photodiode covered by the green color filter. Other types of color filters may use yellow, magenta, and / or cyan (also known as "emerald green") color filters as alternatives to or complements to red, blue, and / or green color filters. Some image sensors may have no color filters at all and may alternatively use different photodiodes (in some cases stacked vertically) throughout the pixel array. Different photodiodes in the pixel array can have different spectral sensitivity profiles, thus responding to light of different wavelengths. Monochrome image sensors may also lack color filters and therefore lack color depth.

[0079] In some cases, image sensor 630 may alternatively or additionally include opaque masks and / or reflective masks that block light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles, which can be used for phase detection autofocus (PDAF). Image sensor 630 may also include an analog gain amplifier for amplifying the analog signal output from the photodiodes and / or an analog-to-digital converter (ADC) for converting the analog signal output from the photodiodes (and / or amplified by the analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed with respect to one or more control mechanisms in control mechanism 620 may alternatively or additionally be included in image sensor 630. Image sensor 630 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal-oxide-semiconductor (CMOS), an N-type metal-oxide-semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0080] The image processor 650 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 654), one or more host processors (including host processor 652), and / or one or more other types of processors—not shown here. The host processor 652 may be a digital signal processor (DSP) and / or other types of processor. In some implementations, the image processor 650 is a single integrated circuit or chip (e.g., referred to as a system-on-a-chip or SoC) including the host processor 652 and ISP 654. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) ports), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G, or LTE, 5G, etc.), memory, and connectivity components (e.g., Bluetooth). ™ I / O ports may include any suitable input / output ports or interfaces according to one or more protocols or specifications, such as Inter-Integrated Circuit 2 (I2C) interface, Inter-Integrated Circuit 3 (I3C) interface, Serial Peripheral Interface (SPI) interface, Serial General Purpose Input / Output (GPIO) interface, Mobile Industrial Processor Interface (MIPI) (such as MIPI CSI-2 physical (PHY) layer ports or interfaces), Advanced High Performance Bus (AHB) bus, any combination thereof and / or other input / output ports.

[0081] Image processor 650 can perform multiple tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, image fusion to form HDR images, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. Image processor 650 can store image frames and / or processed images in random access memory (RAM) 640, read-only memory (ROM), cache, memory cells (e.g., system memory), another storage device, or some combination thereof.

[0082] Various input / output (I / O) devices 660 may be connected to the image processor 650. I / O devices 660 may include displays, keyboards, keypads, touchscreens, touchpads, touch-sensitive surfaces, printers, any other output devices, any other input devices, or some combination thereof. I / O 660 may include one or more ports, jacks, or other connectors that enable wired connections between system 600 and one or more peripheral devices, through which system 600 receives data from and / or sends data to one or more peripheral devices. I / O 660 may include one or more wireless transceivers that enable wireless connections between system 600 and one or more peripheral devices, through which system 600 receives data from and / or sends data to one or more peripheral devices. Peripheral devices may include any type of I / O device of the types discussed above, and once they are coupled to ports, jacks, wireless transceivers, or other wired and / or wireless connectors, they themselves may be considered I / O devices 660.

[0083] In some cases, the image capture and processing system 600 may be a single device. In other cases, the image capture and processing system 600 may be two or more independent devices, including at least one image capture device 605A (e.g., a camera) and an image processing device 606 (e.g., a computing device coupled to the camera). In some embodiments, the image capture and processing system 600 may also include multiple image capture devices, such as image capture device 605A and image capture device 605B shown herein, which may have the same or similar components. In some embodiments, at least one image capture device 605A and image processing device 606 may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some embodiments, at least one image capture device 605A and image processing device 606 may be disconnected from each other.

[0084] Figure 6 The image capture and processing system 600 is divided into two parts, namely at least one image capture device 605A and an image processing device 606. The at least one image capture device 605A includes a lens 615A, a control mechanism 620, and an image sensor 630. The image processing device 606 includes an image processor 650 (including an ISP 654 and a host processor 652), RAM 640, and I / O 660. In some cases, certain components illustrated in the image capture device 606 (such as the ISP 654 and / or the host processor 652) may be included in the image capture device 605A.

[0085] Image capture and processing system 600 may include electronic devices such as mobile or landline phones (e.g., smartphones, cellular phones, etc.), desktop computers, laptop or notebook computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, Internet Protocol (IP) cameras, or any other suitable electronic devices. In some examples, image capture and processing system 600 may include one or more wireless transceivers for wireless communication (such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, or some combination thereof). In some specific implementations, at least one image capture device 605A and image processing device 606 may be different devices. For example, at least one image capture device 605A may include a camera device, and image processing device 606 may include a computing device, such as a mobile phone, desktop computer, or other computing device.

[0086] Although the image capture and processing system 600 is shown as including certain components, those skilled in the art will understand that the image capture and processing system 600 may include more than [other components]. Figure 6 The components shown herein are additional components. Components of the image capture and processing system 600 may include software, hardware, or one or more combinations of software and hardware. For example, in some embodiments, components of the image capture and processing system 600 may include electronic circuitry or other electronic hardware, and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits); and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. Software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the image capture and processing system 600.

[0087] The host processor 652 can configure the image sensor 630 with new parameter settings (e.g., via an external control interface such as I2C, I3C, SPI, GPIO, and / or other interfaces). In an exemplary example, the host processor 652 can adjust the resolution settings used by the image sensor 630, such as a merging mode, based on an assessment of whether the region of interest still provides sufficient texture. The host processor 652 can also dynamically configure the parameter settings of the internal pipeline or modules of the ISP 654 to match the settings of one or more input images from the image sensor 630, so that the image data is processed correctly by the ISP 654. The processing (or pipeline) blocks or modules of the ISP 654 may include modules for lens (or sensor) noise correction, demosaicing, color conversion, correction or enhancement / suppression of image attributes, denoising filters, sharpening filters, etc. Each module of the ISP 654 may include a large number of tunable parameter settings. Additionally, modules can be interdependent because different modules may affect similar aspects of the image. For example, denoising and texture correction or enhancement may both affect the high-frequency aspects of the image. As a result, a large number of parameters are used by the ISP to generate the final image based on the captured raw image.

[0088] like Figure 4 and Figure 5 The techniques described in the context can be implemented in an image capture and processing system 600, which can be, for example, Figures 1 to 3 This is part of the device 110 shown. Thus, instructions 645 stored in memory 640 can cause host processor 652 and / or SP 654 to adjust parameters to be adjusted by image sensor 630 based on input received via I / O 660. For example, I / O 660 can detect, for example... Figure 1 The touchscreen 130 shown represents user input, and based on this input, it can identify, for example... Figure 1 The depicted region of interest 140 is shown in the image 125 displayed on the display 130. This region of interest 140 can be associated, for example, with magnification of a specific area of ​​the displayed image, causing a change in the field of view. Once the host processor 652 or ISP 654 determines that the resolution in the region of interest 140 is no longer sufficient, for example, that the scaling factor has reached a threshold, the host processor 652 or ISP 654 can adjust at least one setting used by the image sensor 630. This setting can be, for example, the pixel count value used by the image sensor 630 to obtain at least one image. Additionally or alternatively, the host processor 652 or ISP 654 can also influence settings used by the control mechanism 620.

[0089] It should be envisioned that the at least two images used herein to describe HDR fusion can be acquired by one image capture device 605A or multiple image capture devices 605A and 605B. Instructions 645 that cause the host processor 652 or ISP 654 to adjust the parameters used to acquire the at least two images can therefore adjust at least one of the settings used by the multiple image capture devices 605A or 605B.

[0090] The methods, systems, and devices discussed above are examples. Various implementations may omit, substitute, or add processes or components as appropriate. For example, in alternative configurations, the described methods may be performed in a different order than described, and stages may be added, omitted, and / or combined. Furthermore, features described with respect to certain implementations may be combined into various other implementations. Different aspects and elements of implementations may be combined in a similar manner.

[0091] In the above description, reference numerals have sometimes been used in combination with various terms. When terms are used in conjunction with reference numerals, this may be intended to refer to a specific element shown in at least one of the accompanying drawings. When terms are used without reference numerals, this may mean a term that is not generally limited to any particular drawing.

[0092] The terms “coupled” or “associated” and any variations thereof can indicate a direct or indirect connection between elements. For example, a first element coupled to a second element can be directly connected to the second element, or indirectly connected to the second element through another element.

[0093] The term "processor" should be interpreted broadly to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, etc. In some cases, "processor" can refer to application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), etc. The term "processor" can also refer to a combination of processing devices, such as a combination of a digital signal processor (DSP) and a microprocessor, multiple microprocessors, a combination of at least one microprocessor with a digital signal processor (DSP) core, or any other such configuration.

[0094] The term "memory" should be broadly interpreted to include any electronic component capable of storing electronic information. The term memory can refer to various types of processor-readable media, such as: Random Access Memory (RAM), Read-Only Memory (ROM), Non-Volatile Random Access Memory (NVRAM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable PROM (EEPROM), Flash Memory, magnetic or optical data storage, registers, etc. If the processor can read information from and / or write information to the memory, it is said that the memory is in electronic communication with the processor. Memory integrated with the processor communicates electronically with the processor.

[0095] The terms “instruction” and “code” should be interpreted broadly to include any type of computer-readable statement. For example, the terms “instruction” and “code” can refer to at least one of a program, routine, subroutine, function, procedure, etc. “Instruction” and “code” can comprise a single computer-readable statement or a number of computer-readable statements.

[0096] It should be noted that, where compatible, at least one of the features, functions, processes, components, elements, structures, etc., of any of the configurations described herein can be combined with at least one of the functions, processes, components, elements, structures, etc., of any other configuration described herein. In other words, any compatible combination of the functions, processes, components, elements, etc., described herein can be implemented based on the systems and methods disclosed herein.

[0097] The described configurations are presented to enable those skilled in the art to make or use the methods and other structures disclosed herein. The flowcharts, block diagrams, and other structures shown and described herein are merely examples, and other variations of these structures are also within the scope of this disclosure. Various modifications to these configurations are possible, and the general principles presented herein can be applied to other configurations. Therefore, this disclosure is not intended to be limited to the configurations shown above, but should be accorded the widest scope consistent with the principles and novel features disclosed herein in any way, including in the appended claims, which form part of the original disclosure.

[0098] Those skilled in the art will understand that information and signals can be represented using any of a variety of different techniques and arts. For example, data, instructions, commands, information, signals, bits, and symbols that may be mentioned throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0099] The apparatuses disclosed herein (e.g., any device configured to perform the techniques described herein) can be considered as any combination of hardware and software and / or firmware suitable for a given application. For example, elements of such apparatuses can be fabricated as electronic and / or optical devices residing, for example, on the same chip or within a chipset. An example of such apparatuses is a fixed or programmable array of logic elements (such as transistors or logic gates), and any of these elements can be implemented as one or more such arrays. Any two or more, or even all, of these elements can be implemented within the same one or more arrays. Such one or more arrays can be implemented within one or more chips (e.g., within a chipset comprising two or more chips).

[0100] One or more elements of various specific embodiments of the apparatus disclosed herein may be implemented, in whole or in part, as one or more instruction sets arranged to execute on one or more arrays of fixed or programmable logic elements, such as microprocessors, embedded processors, intellectual property (IP) cores, digital signal processors, FPGAs (Field-Programmable Gate Arrays), ASSPs (Application-Specific Standard Products), and ASICs (Application-Specific Integrated Circuits). Any element of the various implementations of the apparatus disclosed herein may also be embodied as one or more computers (e.g., including one or more arrays of one or more instruction sets or sequences, also referred to as "processors"), and any two or more, or even all, of these elements may be implemented within the same one or more such computers.

[0101] The processor or other components used for processing as disclosed herein can be manufactured as one or more electronic and / or optical devices residing between two or more chips, such as on the same chip or in a chipset. An example of such a device is a fixed or programmable array of logic elements, such as transistors or logic gates, and any of these elements can be implemented as one or more such arrays. Such arrays can be implemented within one or more chips (e.g., within a chipset comprising two or more chips). Examples of such arrays include fixed or programmable arrays of logic elements, such as microprocessors, embedded processors, IP cores, DSPs, FPGAs, ASSPs, and ASICs. The processor or other components used for processing as disclosed herein can also be embodied as one or more computers (e.g., machines including one or more arrays programmed to execute one or more instruction sets or sequences) or other processors. The processor described herein can be used to perform tasks or other instruction sets not directly related to a specific implementation of the methods disclosed herein, such as tasks related to another operation of a device or system in which the processor is embedded (e.g., an audio sensing device). A portion of the methods disclosed herein can also be executed by the processor of the audio sensing device, and another portion of the methods can be executed under the control of one or more other processors.

[0102] Those skilled in the art will understand that the various exemplary modules, logic blocks, circuits, tests, and other operations described in connection with the configurations disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. Such modules, logic blocks, circuits, and operations can be implemented or performed using a general-purpose processor, digital signal processor (DSP), ASIC or ASSP, FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to produce the configurations disclosed herein. For example, such configurations can be implemented at least partially as hardwired circuitry, circuit configurations fabricated into application-specific integrated circuits, or firmware loaded into non-volatile memory, or software programs loaded or added to data storage media as machine-readable code, such code being instructions executable by an array of logic elements such as a general-purpose processor or other digital signal processing unit. The general-purpose processor can be a microprocessor, but in alternatives, the processor can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Software modules may reside in non-transitory storage media such as RAM (Random Access Memory), ROM (Read-Only Memory), non-volatile RAM such as flash RAM (NVRAM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disks, removable disks, or CD-ROMs, or in any other form of storage media known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium may be integral with the processor. The processor and storage medium may reside in an ASIC, and the ASIC may reside in a user terminal. Alternatively, the processor and storage medium may reside as discrete components in a user terminal. The term "computer program product" refers to a computing device or processor combined with code or instructions (e.g., a "program") that can be executed, processed, or computed by the computing device or processor.

[0103] Note that the various methods disclosed herein can be executed by an array of logic elements (e.g., a processor), and the various elements of the apparatus described herein can be implemented as modules designed to execute on such an array. As used herein, the terms "module" or "submodule" can refer to any method, apparatus, device, unit, or computer-readable data storage medium that includes computer instructions (e.g., logical expressions) in the form of software, hardware, or firmware. It should be understood that multiple modules or systems can be combined into one module or system, and a module or system can be divided into multiple modules or systems to perform the same function. When implemented in software or other computer-executable instructions, the elements of a process are essentially code segments used to perform related tasks, such as routines, programs, objects, components, data structures, etc. The term "software" should be understood to include source code, assembly language code, machine code, binary code, firmware, macro code, microcode, any one or more sets or sequences of instructions executable by an array of logic elements, and any combination of these examples. Programs or code segments can be stored in a processor-readable medium or transmitted by computer data signals embodied in a carrier wave on a transmission medium or communication link.

[0104] Specific implementations of the methods, schemes, and techniques disclosed herein may also be tangibly embodied (e.g., in the tangible computer-readable features of one or more computer-readable storage media listed herein) as one or more instruction sets executable by a machine comprising an array of logic elements (e.g., a processor, microprocessor, microcontroller, or other finite state machine). The term "computer-readable medium" can include any medium capable of storing or transmitting information, including volatile, non-volatile, removable, and non-removable storage media. Examples of computer-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks or other magnetic storage, CD-ROM / DVD or other optical storage, hard disks or any other medium that can be used to store desired information, fiber optic media, radio frequency (RF) links, or any other medium that can be used to carry desired information and make it accessible. Computer signals can include any signal capable of propagating over a transmission medium such as an electronic network channel, fiber optic cable, air, electromagnetic field, or RF link. Code segments can be downloaded via a computer network such as the Internet or an intranet. In no event should the scope of this disclosure be construed as limited to such embodiments. Each of the tasks in the methods described herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. In typical applications of specific implementations of the methods disclosed herein, an array of logic elements (e.g., logic gates) is configured to perform one, more than one, or even all of the various tasks of the methods. One or more (potentially all) tasks can also be implemented as code (e.g., one or more instruction sets) embodied in a computer program product (e.g., one or more data storage media, such as disks, flash memory or other non-volatile memory cards, semiconductor memory chips, etc.), which can be read and / or executed by a machine (e.g., a computer) including an array of logic elements (e.g., a processor, microprocessor, microcontroller, or other finite state machine). Tasks in specific implementations of the methods disclosed herein can also be performed by more than one such array or machine. In these or other implementations, tasks can be performed within a device for wireless communication (e.g., a cellular telephone or other device with this communication capability). Such devices can be configured to communicate with circuit-switched and / or packet-switched networks (e.g., using one or more protocols such as VoIP). For example, such devices may include RF circuitry configured to receive and / or transmit encoded frames.

[0105] It is explicitly disclosed that the various methods disclosed herein can be performed by portable communication devices such as mobile phones, cell phones, headsets or portable digital assistants (PDAs), and the various devices described herein can be included within such devices.

[0106] In one or more exemplary embodiments, the operations described herein may be implemented using hardware, software, firmware, or any combination thereof. If implemented in software, such operations may be stored as one or more instructions or code on or transmitted via a computer-readable medium. The term “computer-readable medium” includes both computer-readable storage media and communication (e.g., transmission) media. By way of example, and not limitation, a computer-readable storage medium may include an array of storage elements, such as semiconductor memory (which may include, but is not limited to, dynamic or static RAM, ROM, EEPROM, and / or flash RAM), or ferroelectric, magnetoresistive, bidirectional, polymer, or phase-change memory; CD-ROM or other optical disc storage; and / or magnetic disk storage devices or other magnetic storage devices. Such storage media may store information in the form of computer-accessible instructions or data structures. Communication media may include any medium that can be used to carry desired program code in the form of instructions or data structures and that is computer-accessible, including any medium that facilitates the transfer of a computer program from one place to another. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and / or microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and / or microwave are included within the definition of media. As used herein, disks and optical discs include compact optical discs (CDs), laser discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0107] One or more elements of a specific embodiment of the apparatus described herein may be used to perform a task or to perform other sets of instructions not directly related to the operation of the apparatus, such as a task related to another operation of the device or system in which the apparatus is embedded. One or more elements of such a specific embodiment of the apparatus may also have a common structure (e.g., a processor for executing code portions corresponding to different elements at different times, a set of instructions executed to perform tasks corresponding to different elements at different times, or an arrangement of electronic and / or optical devices for performing operations of different elements at different times).

[0108] It should be understood that the claims are not limited to the precise configurations and components illustrated above. Various modifications, alterations, and variations can be made to the arrangement, operation, and details of the systems, methods, and apparatus described herein without departing from the scope of the claims.

[0109] The claim language or other language that states "at least one of" and / or "one or more of" in a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, the claim language stating "at least one of A and B" means A, B, or A and B. In another example, the claim language stating "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" and / or "one or more of" in a set does not limit the set to the items listed in the set. For example, the claim language stating "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

Claims

1. A method for image fusion, the method comprising: Obtain the first image of the scene associated with the first parameter, and Obtain a second image of the scene associated with the second parameter; The first image and the second image are merged to obtain a first merged image; Obtain a third image of the scene associated with the third parameter, and A fourth image of the scene associated with a fourth parameter is obtained, wherein at least one of the third parameter or the fourth parameter is adjusted, wherein the adjustment is at least partially based on the field of view of the first fused image; as well as The third image and the fourth image are merged to obtain a second merged image.

2. The method according to claim 1, further comprising: Preview the first fused image before obtaining the third and fourth images.

3. The method of claim 1, wherein the parameter associated with the image is the output pixel count value of the image.

4. The method of claim 3, wherein the output pixel count value associated with the image is related to a merging mode used to obtain the image.

5. The method according to claim 1, further comprising: Adjust at least one of the third parameter or the fourth parameter based on the power mode.

6. The method of claim 5, wherein in low power mode, at least one of the third parameter or the fourth parameter is adjusted to have a lower value than in normal power mode.

7. The method of claim 1, wherein the adjustment is based on determining whether a threshold is exceeded.

8. The method of claim 7, wherein the threshold is associated with power consumption.

9. The method of claim 1, wherein the first image and the second image and / or the third image and the fourth image were captured with different exposure values.

10. The method of claim 1, wherein the fusion of the first image and the second image comprises fusion of the first image and the second image on a pixel-by-pixel basis and / or the fusion of the third image and the fourth image comprises fusion of the third image and the fourth image on a pixel-by-pixel basis.

11. The method according to claim 1, further comprising: The user receives input associated with a region of interest in the first fused image, and the field of view is determined based on the region of interest.

12. The method of claim 11, wherein the input from the user is a zoom-in or zoom-out action.

13. The method of claim 1, further comprising storing the second fused image.

14. The method of claim 1, wherein obtaining comprises capturing the image using at least one image capturing device.

15. The method of claim 14, wherein the at least one image capturing device is configured to capture images with different pixel count values.

16. An apparatus for image capture and fusion, the apparatus comprising: At least one image capture device, said at least one image capture device being configured to: Obtain a first image of the scene associated with a first parameter, a second image of the scene associated with a second parameter, a third image of the scene associated with a third parameter, and a fourth image of the scene associated with a fourth parameter; At least one or more processors, wherein the at least one or more processors are configured to: The first image and the second image are merged to obtain a first merged image; At least one of the third parameter or the fourth parameter to be used to obtain the third image or the fourth image is adjusted, based at least in part on the field of view in the first fused image. as well as The third image and the fourth image are merged to obtain a second fused image; and At least one memory, the at least one memory being used to store the fused image.

17. The apparatus of claim 16, wherein the at least one image capturing device is configured to acquire an image having a different pixel count value, and the parameter associated with the image is related to the pixel count value used to acquire the corresponding image.

18. The apparatus of claim 17, wherein the pixel count value of the image is associated with a merging mode for capturing the image.

19. The apparatus of claim 16, wherein the adjustment is further based on the power mode of the apparatus.

20. The apparatus of claim 16, wherein the one or more processors are configured to adjust at least one of the parameters when a threshold is exceeded.

21. The apparatus of claim 20, wherein the threshold is associated with the power consumption of the apparatus.

22. The apparatus of claim 16, wherein the first image and the second image and / or the third image and the fourth image were captured at different exposure values.

23. The apparatus of claim 16, wherein the one or more processors are configured to fuse the first image and the second image on a pixel-by-pixel basis and / or fuse the third image and the fourth image on a pixel-by-pixel basis.

24. The apparatus of claim 23, wherein the one or more processors are further configured to: select a pixel from the first image or the second image for each pixel of the first fused image, and select a pixel from the third image or the fourth image for each pixel of the second fused image.

25. The apparatus of claim 23, wherein the first image is composed of sub-pixels of a superpixel image, and the second image is composed of other sub-pixels of the superpixel image, and the one or more processors are further configured to fuse the sub-pixels, and / or wherein the third image is composed of sub-pixels of a superpixel image, and the fourth image is composed of other sub-pixels of the superpixel image, and the one or more processors are further configured to fuse the sub-pixels.

26. The apparatus of claim 16, wherein the one or more processors are configured to receive input from a user associated with a region of interest in the first fused image, and to determine the field of view based on the region of interest.

27. The apparatus of claim 26, wherein the input from the user is a zoom-in or zoom-out action.

28. The device of claim 16, wherein the device is part of a handheld camera or mobile phone.

29. An apparatus for image fusion, the apparatus comprising: A component used to obtain a first image of the scene associated with a first parameter; A component for obtaining a second image of the scene associated with the second parameter; Components for fusing the first image and the second image to obtain a first fused image; Components for adjusting at least one of a third parameter or a fourth parameter to be used to obtain a third image and a fourth image, wherein the adjustment is at least partially based on the field of view in the first fused image; A component for obtaining the third image of the scene associated with the third parameter; A component for obtaining the fourth image of the scene associated with the fourth parameter; and A component for fusing the third image and the fourth image to obtain a second fused image.

30. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: Obtain the first image of the scene associated with the first parameter, and Obtain a second image of the scene associated with the second parameter; The first image and the second image are merged to obtain a first merged image; Obtain a third image of the scene associated with the third parameter, and A fourth image of the scene associated with a fourth parameter is obtained, wherein at least one of the third parameter or the fourth parameter is adjusted, wherein the adjustment is at least partially based on the field of view in the first fused image; as well as The third image and the fourth image are merged to obtain a second merged image.

Images