Method and apparatus for combining image frames captured using different exposure settings into a blended image

Abstract

A method for multi-frame blending includes: obtaining multiple image frames of a scene, the multiple image frames including a reference image frame captured with a shorter exposure time and multiple non-reference image frames captured with a longer exposure time; adjusting the brightness and color of the reference image frame to match the non-reference image frame; obtaining multiple motion maps corresponding to the multiple image frames respectively; blending the reference image frame with the non-reference image frame into a blended image based on the multiple motion maps; and performing contour reduction on the blended image to reduce contour artifacts in the blended image.

Description

Technical Field

[0001] The present disclosure relates to image capture systems and more particularly to methods and apparatus for combining image frames captured using different exposure settings into a blended image. Background Art

[0002] Many mobile electronic devices, such as smartphones and tablet computers, include cameras that can be used to capture both still and video images. Although convenient, cameras on mobile electronic devices often have many deficiencies. For example, captured images of action scenes are often prone to motion blur, which refers to the blurring or smearing of one or more moving objects in the scene. The amount of motion blur in the image can be reduced by reducing the exposure time used to capture the image, but unfortunately, this also increases the noise level of the captured image. As another example, images of certain scenes may be prone to contour artifacts, which refers to the generation of false contours in areas of the image with less image detail. Contour artifacts are particularly noticeable when they appear on a person's face or skin, or when they appear around an object that is set against a generally uniform background. Summary of the invention

[0003] Technical issues

[0004] Cameras in many mobile electronic devices may have many shortcomings. For example, captured images of action scenes are often prone to motion blur, and one possible solution (reducing the exposure time used to capture the image) can reduce motion blur but increase noise. Motion blur is particularly problematic in low-light environments because the exposure time for capturing images in low-light environments needs to be increased to ensure that a sufficient amount of light is sensed. As another example, images of certain scenes may be prone to contour artifacts, which are particularly noticeable when they appear on a person's face or skin, or when they appear around an object set against a generally uniform background. Reducing or minimizing contour artifacts is not an easy task.

[0005] Technical Solution

[0006] The present disclosure provides methods and apparatus for combining image frames captured using different exposure settings into a blended image.

[0007] According to one aspect of the present disclosure, a method for multi-frame blending is provided. The method includes: obtaining multiple image frames of a scene, the multiple image frames including a reference image frame captured with a shorter exposure time and multiple non-reference image frames captured with a longer exposure time; adjusting the brightness and color of the reference image frame to match the non-reference image frame; obtaining multiple motion maps corresponding to the multiple image frames respectively; blending the reference image frame with the non-reference image frame into a blended image based on the multiple motion maps; and performing contour reduction on the blended image to reduce contour artifacts in the blended image.

[0008] In an embodiment, the reference image frames are captured with a higher sensitivity than the non-reference image frames.

[0009] In an embodiment, the non-reference image frames include at least one image frame captured before the reference image frame and at least one image frame captured after the reference image frame.

[0010] In an embodiment, obtaining a plurality of image frames includes obtaining a plurality of image frames captured using a shorter exposure time, and selecting one image frame from the plurality of image frames captured using the shorter exposure time as a reference image frame.

[0011] In an embodiment, obtaining the plurality of image frames includes modifying the non-reference image frames to align with the reference image frames.

[0012] In an embodiment, adjusting the brightness and color of the reference image frame comprises performing a non-linear transformation on the reference image frame based on at least one histogram associated with at least one non-reference image frame.

[0013] In an embodiment, adjusting brightness and color of the reference image frame includes obtaining a histogram map based on differences between the reference image frame and the non-reference image frame, and modifying the reference image frame by mapping pixel values ​​of the reference image frame based on the at least one histogram map.

[0014] In an embodiment, obtaining the histogram map includes: obtaining a plurality of histogram maps respectively corresponding to the plurality of non-reference image frames, and combining the plurality of histogram maps to obtain the above-mentioned histogram map.

[0015] In an embodiment, obtaining the histogram map includes identifying motion pixels associated with motion from the reference image frame and the non-reference image frame, and obtaining the histogram map based on pixels other than the motion pixels in the reference image frame and the non-reference image frame.

[0016] In an embodiment, identifying motion pixels includes: identifying absolute differences between pixel values ​​from a reference image frame and a non-reference image frame, generating at least one histogram based on the absolute differences, identifying one or more motion thresholds based on the at least one histogram, and determining pixels having at least one absolute difference value exceeding at least one of the one or more motion thresholds as motion pixels.

[0017] In an embodiment, obtaining multiple motion maps includes: obtaining multiple non-reference motion maps based on differences between a reference image frame and multiple non-reference image frames, respectively, combining the multiple non-reference motion maps to obtain a combined non-reference motion map, and obtaining a reference motion map corresponding to the reference image frame based on the combined non-reference motion map, wherein the reference motion map and the combined non-reference motion map complement each other.

[0018] In an embodiment, the values ​​of the reference motion map are within a predetermined range.

[0019] In an embodiment, blending the reference image frame and the non-reference image frame into a blended image based on the plurality of motion maps comprises performing a weighted average of the reference image frame and the non-reference image frame by using the plurality of motion maps as weights.

[0020] In an embodiment, performing contour reduction on the mixed image includes: obtaining contour likelihoods for pixels of the mixed image based on the plurality of motion maps, the contour likelihoods indicating a degree of consistency of contributions of non-reference image frames to the mixed image, and modifying the mixed image by using the contour likelihoods.

[0021] In an embodiment, modifying the mixed image includes: obtaining a difference signal between the mixed image and a reference image frame, obtaining a filtered difference signal by performing at least one of edge intensity filtering and low-pass filtering on the difference signal, obtaining a contour signal based on the filtered difference signal and contour likelihood, and subtracting the contour signal from the mixed image.

[0022] In an embodiment, obtaining the difference signal includes identifying a luminance data difference between the reference image frame and the mixed image without identifying a chrominance data difference between the reference image frame and the mixed image.

[0023] In an embodiment, obtaining the filtered difference signal comprises: normalizing the difference signal by performing edge intensity filtering using pixels in a moving 3x3 window, and performing low pass filtering on the normalized difference signal using a 7x7 uniform filter to obtain the filtered difference signal.

[0024] In an embodiment, the method further comprises performing at least one of noise filtering and edge sharpening on the contour-reduced hybrid image.

[0025] According to another aspect of the present disclosure, an electronic device is provided. The electronic device includes at least one image sensor, at least one memory, and at least one processor. The at least one memory stores instructions. The at least one processor is configured to execute the stored instructions to control the at least one image sensor to obtain multiple image frames of a scene, the multiple image frames include reference image frames captured with a shorter exposure time and multiple non-reference image frames captured with a longer exposure time, adjust the brightness and color of the reference image frames to match the non-reference image frames, obtain multiple motion maps corresponding to the multiple image frames, mix the reference image frames with the non-reference image frames into a mixed image based on the multiple motion maps, and perform contour reduction on the mixed image to reduce contour artifacts in the mixed image.

[0026] According to another aspect of the present disclosure, a method for contour reduction is provided. The method includes obtaining a mixed image by combining a plurality of image frames including a reference image frame and a plurality of non-reference image frames, obtaining a plurality of motion maps respectively corresponding to the plurality of image frames, obtaining contour likelihoods of pixels of the mixed image based on the plurality of motion maps, the contour likelihood indicating a balanced degree of contribution of the non-reference image frames to the mixed image, and modifying the mixed image by using the contour likelihood.

[0027] According to another aspect of the present disclosure, a method for multi-frame blending includes obtaining at least two image frames of a scene. One image frame is associated with a shorter exposure time and a higher sensitivity and represents a reference image frame. At least another image frame is associated with a longer exposure time and a lower sensitivity and represents at least one non-reference image frame. The method also includes blending the reference image frame and the non-reference image frame into a blended image, such that (i) one or more motion regions of the blended image are more based on the reference image frame, and (ii) one or more static regions of the blended image are more based on the at least one non-reference image frame.

[0028] According to another aspect of the present disclosure, an electronic device includes at least one image sensor and at least one processor. The at least one processor is configured to use the at least one image sensor to obtain at least two image frames of a scene. One image frame is associated with a shorter exposure time and a higher sensitivity and represents a reference image frame. At least another image frame is associated with a longer exposure time and a lower sensitivity and represents at least one non-reference image frame. The at least one processor is also configured to blend the reference and non-reference image frames into a blended image such that (i) one or more motion regions of the blended image are more based on the reference image frame, and (ii) one or more static regions of the blended image are more based on the at least one non-reference image frame.

[0029] According to another aspect of the present disclosure, a method for contour reduction includes obtaining a hybrid image generated by combining a reference image frame and a plurality of non-reference image frames. The method also includes identifying locations where motion maps associated with the non-reference image frames consistently or inconsistently identify motion relative to the reference frame. The method also includes identifying pixels or regions in the hybrid image to which contour reduction is to be applied based on the locations where the motion maps inconsistently identify motion. In addition, the method includes applying contour reduction to the identified pixels or regions to generate a processed hybrid image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, wherein like reference numerals represent like parts:

[0031] Figure 1 An example network configuration including electronic devices according to an embodiment of the present disclosure is shown;

[0032] Figure 2 An example process for blending multiple image frames captured using different exposure settings according to an embodiment of the present disclosure is shown;

[0033] Figure 3 An example process for performing a multi-frame blending operation according to an embodiment of the present disclosure is shown;

[0034] Figure 4 An example process for performing a histogram matching operation according to an embodiment of the present disclosure is shown;

[0035] Figure 5 An example process for performing a histogram matching map generation operation according to an embodiment of the present disclosure is shown;

[0036] Figure 6 An example process for performing motion mapping post-processing operations according to an embodiment of the present disclosure is shown;

[0037] Figure 7 An example process for performing a contour reduction operation according to an embodiment of the present disclosure is shown;

[0038] Figure 8 An example edge strength filter according to an embodiment of the present disclosure is shown;

[0039] Fig. 9 is a flow chart illustrating an example method for blending multiple image frames captured using different exposure settings according to an embodiment of the present disclosure;

[0040] Fig.10 is a flow chart illustrating an example method for contour reduction in a mixed image according to an embodiment of the present disclosure;

[0041] Fig.11A and Fig. 11B illustrates example results obtained by blending multiple image frames captured using different exposure settings according to an embodiment of the present disclosure; and

[0042] Fig. 12A and Fig. 12B An example result obtained by performing contour reduction on a mixed image according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0043] Embodiments of the present disclosure will be described below with reference to the accompanying drawings. However, the embodiments of the present disclosure are not limited to specific embodiments, and should be interpreted as including all modifications, variations, equivalent devices and methods, and / or alternative embodiments of the present disclosure. In the description of the accompanying drawings, similar reference numerals are used for similar elements.

[0044] The terms "send," "receive," and "communicate" and their derivatives encompass direct and indirect communications. The terms "include" and "comprises" and their derivatives mean inclusion without limitation. The term "or" is inclusive, meaning and / or. The phrase "associated with" and its derivatives means include, included, interconnected with, contain, contained within, connected to or connected with, coupled to or coupled with, communicable with, cooperating with, intertwined, side by side, adjacent to, bound to or bound with, having, having the property of, having a relationship with, and the like.

[0045] Moreover, the various functions described below can be implemented or supported by one or more computer programs, each of which is composed of a computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data or parts thereof suitable for being implemented as suitable computer-readable program codes. The phrase "computer-readable program code" includes any type of computer code, including source code, object code and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard disk drive, a compact disk (CD), a digital video disk (DVD) or any other type of memory. "Non-transient" computer-readable media excludes wired, wireless, optical or other communication links that transmit transient electrical or other signals. Non-transient computer-readable media include media in which data can be permanently stored and media in which data can be stored and subsequently rewritten, such as rewritable optical disks or erasable storage devices.

[0046] As used herein, terms and phrases such as "having", "may have", "include", or "may include" a feature (similar to a number, function, operation, or component such as a part) indicate the presence of the feature, but do not exclude the presence of other features. Moreover, as used herein, the phrases "A or B", "at least one of A and / or B", or "one or more of A and / or B" may include all possible combinations of A and B. For example, "A or B", "at least one of A and B", and "at least one of A or B" may indicate all of the following: (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B. Further, as used herein, the terms "first" and "second" may modify various components without regard to importance and do not limit the components. These terms are used only to distinguish one component from another. For example, a first user device and a second user device may indicate user devices that are different from each other, without regard to the order or importance of the devices. Without departing from the scope of the present disclosure, a first component may refer to a second component, and vice versa.

[0047] It will be understood that when an element (such as a first element) is referred to as being (operably or communicatively) "coupled to" or "connected to" another element (such as a second element), it may be coupled to or connected to the other element directly or via a third element. In contrast, it will be understood that when an element (such as a first element) is referred to as being "directly coupled to" or "directly connected to" another element (such as a second element), there are no other elements intervening (such as a third element) between the element and the other element.

[0048] As used herein, the phrase "configured (or set) to" may be used interchangeably with the phrases "suitable for," "capable of," "designed to," "suitable for," "so that," or "capable of," depending on the circumstances. The phrase "configured (or set) to" does not inherently mean "specially designed in hardware to." On the contrary, the phrase "configured to" may mean that a device can perform a certain operation together with another device or component. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a general-purpose processor, which is a hardware device (such as a CPU or application processor) that can perform operations by executing one or more software programs stored in a memory device, or may refer to a special-purpose processor, which is a hardware device (such as an embedded processor) for performing operations.

[0049] The terms and phrases used herein are provided only to describe some embodiments of this document, rather than to limit the scope of other embodiments of the present disclosure. It should be understood that the singular forms "one", "an" and "the" include plural references unless the context clearly indicates otherwise. All terms and phrases used herein, including technical and scientific terms and phrases, have the same meanings as those commonly understood by those skilled in the art to which the embodiments of the present disclosure belong. It will also be further understood that those terms and phrases such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant art, and will not be interpreted as idealized or overly formal meanings, unless clearly defined here. In some cases, the terms and phrases defined herein may be interpreted as excluding embodiments of the present disclosure.

[0050] Examples of "electronic devices" according to embodiments of the present disclosure may include at least one of the following: smart phones, tablet personal computers (PCs), mobile phones, video phones, e-book readers, desktop PCs, laptop computers, netbook computers, workstations, personal digital assistants (PDAs), portable multimedia players (PMPs), MP3 players, mobile medical devices, cameras, or wearable devices (such as smart glasses, head-mounted devices (HDMs), electronic clothing, electronic bracelets, electronic necklaces, electronic accessories, electronic tattoos, smart mirrors, or smart watches). Definitions for some other words and units may be provided through this patent document. Those of ordinary skill in the art should understand that in many cases, if not most cases, these definitions apply to previous and future usages of the words and phrases defined in this way.

[0051] Nothing in this application should be interpreted as implying that any particular element, step, or function is an essential element that must be included in the scope of the claims. The scope of the subject matter protected by the patent is limited only by the claims. Applicants understand that any other terms used in the claims, including but not limited to "mechanism", "module", "device", "unit", "component", "element", "member", "device", "machine", "system", "processor" or "controller", refer to structures well known to technicians in the relevant field.

[0052] Discussed below Figures 1 to 12B Various embodiments of the present disclosure are described with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to these embodiments, and all changes and / or equivalents or substitutes thereto also belong to the scope of the present disclosure. Throughout the specification and the accompanying drawings, the same or similar reference symbols may be used to indicate the same or similar elements.

[0053] The present disclosure provides techniques for blending multiple image frames captured using different exposure settings. As described in more detail below, an image frame may be captured using a shorter exposure time and a higher sensitivity and may be used as a reference frame, while one or more additional image frames may be captured using a longer exposure time and a lower sensitivity (such as before, after, or before and after capturing the shorter exposure image frame). Compared to the longer exposure image frame, the shorter exposure image frame may have less image detail in areas where less or no motion occurs, but the shorter exposure image frame may have less or no motion blur in areas where motion occurs. Conversely, compared to the shorter exposure image frame, the longer exposure image frame may have more image detail in areas where less or no motion occurs, but the longer exposure image frame may have more motion blur in areas where motion occurs. Therefore, the techniques described in the present disclosure allow for combining multiple image frames (at least some of which are captured using different exposure settings) to produce a blended image with reduced / minimized motion blur and reduced / minimized noise. In various embodiments, this may be achieved by using histogram matching with effective motion pixel suppression to reduce or minimize photometric differences between image frames, and multiple histogram matches may be performed using different image frames from longer exposure image frames (if available) and shorter exposure image frames.

[0054] According to various embodiments described in the present disclosure, different image frames can be optimized for different desired characteristics, such as by (i) capturing one or more objects with reduced or minimized motion blur using a shorter exposure time, and (ii) increasing or maximizing image detail by reducing or minimizing noise using a longer exposure time. Higher quality regions from different image frames can be combined, while low quality regions of the image frames are primarily suppressed during the combination. In the context of reducing motion blur, this can include primarily using regions where motion occurs in shorter exposure image frames (referred to as “motion regions” of the image frames) and regions where less or no motion occurs in longer exposure image frames (referred to as “static regions” of the image frames). Transitions between regions from different image frames can be gracefully handled to improve transitions between regions in the blended image. It should be noted here that while motion blur reduction is one possible application of the multi-exposure image frame blending technique, other applications of the multi-exposure image frame blending technique are also possible.

[0055] The present disclosure also provides techniques for contour reduction in blended images, which may or may not be used with the multi-exposure image frame blending techniques described in the present disclosure. Techniques for contour reduction can be used with various methods in which multiple image frames are combined to produce a blended image, regardless of whether the multiple image frames are captured using common or different exposure settings. As described in more detail below, the blended image produced by combining multiple image frames in any manner is analyzed to isolate and suppress contour artifacts only in specific areas where contour artifacts are likely to occur. This is done so that contour artifacts are reduced or minimized in areas of the blended image that are associated with areas of motion, while avoiding the application of contour reduction to areas where contour artifacts are unlikely to occur, thereby avoiding unnecessary degradation of image quality. In this way, these contour reduction techniques can produce more pleasing scene images with fewer distracting artifacts.

[0056] Note that although these techniques are generally described below as being performed using a mobile electronic device, other electronic devices may also be used to perform or support these techniques. Therefore, these techniques may be used in various types of electronic devices. Furthermore, although the techniques described below are generally described as processing image frames when capturing a static image of a scene, the same or similar methods may also be used to support the capture of video images.

[0057] Figure 1 An example network configuration 100 including electronic devices according to an embodiment of the present disclosure is shown. Figure 1 The embodiment of the network configuration 100 shown is for illustrative purposes only. Other embodiments of the network configuration 100 may also be used without departing from the scope of the present disclosure.

[0058] According to an embodiment of the present disclosure, an electronic device 101 may be included in the network configuration 100. The electronic device 101 may be a device for capturing and processing images. The electronic device 101 may include at least one of the following: a bus 110, a processor 120, a memory 130, an input / output (I / O) interface 150, a display 160, a communication interface 170, or a sensor 180. In some embodiments, the electronic device 101 may not include at least one of these components, or at least one other component may be added. The bus 110 may include circuits for connecting the components 120-180 to each other and for transmitting communications (such as control messages and / or data) between the components.

[0059] The processor 120 may include one or more central processor units (CPUs), application processors (APs), or communication processors (CPs). The processor 120 is capable of controlling at least one of the other components of the electronic device 101 and / or performing communication-related operations or data processing. In some embodiments, the processor 120 may be a graphics processor unit (GPU). For example, the processor 120 may receive image data captured by at least one camera. The processor 120 may process the image data as discussed in more detail below to mix image frames captured using different exposure settings. The processor 120 may also or instead process the image data as discussed in more detail below to provide contour reduction.

[0060] The memory 130 may include volatile and / or non-volatile memory. For example, the memory 130 may store commands or data related to at least one other component of the electronic device 101. According to an embodiment of the present disclosure, the memory 130 may store software and / or programs 140. The programs 140 include, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and / or an application program (or "application") 147. At least a portion of the kernel 141, the middleware 143, or the API 145 may represent an operating system (OS).

[0061] The kernel 141 may control or manage system resources (such as bus 110, processor 120, or memory 130) for executing operations or functions implemented in other programs (such as middleware 143, API 145, or application 147). The kernel 141 provides an interface that allows the middleware 143, API 145, or application 147 to access the various components of the electronic device 101 to control or manage system resources. The application 147 may include one or more applications for image capture and image processing discussed below. These functions may be performed by a single application or by multiple applications, in which each application performs one or more of these functions. The middleware 143 may be used as a relay to allow the API 145 or application 147 to transmit data, for example, with the kernel 141. Multiple applications 147 may be provided. The middleware 143 is capable of controlling work requests received from the application 147, such as by allocating a priority for using the system resources (such as bus 110, processor 120, or memory 130) of the electronic device 101 to at least one of the multiple applications 147. The API 145 may be an interface that allows the application 147 to control functions provided from the kernel 141 or the middleware 143. For example, the API 145 may include at least one interface or function (such as a command) for archive control, window control, image processing, or text control.

[0062] The I / O interface 150 may be used as an interface capable of, for example, transmitting commands or data input from a user or other external devices to other components of the electronic device 101. The I / O interface 150 may also output commands or data received from other components of the electronic device 101 to the user or other external devices.

[0063] The display 160 may include, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot light emitting diode (QLED) display, a micro-electromechanical system (MEMS) display, or an electronic paper display. The display 160 may also be a depth perception display, such as a multi-focal display. The display 160 is capable of displaying, for example, various contents (such as text, images, videos, icons, or symbols) to the user. The display 160 may include a touch screen and may receive, for example, a touch, gesture, proximity, or hover input using an electronic pen or a user's body part.

[0064] The communication interface 170 can, for example, establish communication between the electronic device 101 and an external electronic device (such as the first electronic device 102, the second electronic device 104, or the server 106). For example, the communication interface 170 can be connected to the network 162 or 164 through wireless or wired communication to communicate with the external electronic device. The communication interface 170 can be a wired or wireless transceiver or any other component for transmitting and receiving signals (such as images).

[0065] The electronic device 101 may also include one or more sensors 180, which may measure physical quantities or detect the activation state of the electronic device 101, and convert the measured or detected information into electrical signals. For example, the one or more sensors 180 may include one or more buttons for touch input, one or more cameras, gesture sensors, gyroscopes or gyro sensors, air pressure sensors, magnetic sensors or magnetometers, acceleration sensors or accelerometers, grip sensors, proximity sensors, color sensors (such as red, green, and blue (RGB) sensors), biophysical sensors, temperature sensors, humidity sensors, light sensors, ultraviolet (UV) sensors, electromyography (EMG) sensors, electroencephalography (EEG) sensors, electrocardiography (ECG) sensors, infrared (IR) sensors, ultrasonic sensors, iris sensors, or fingerprint sensors. The sensor 180 may also include an inertial measurement unit, which may include one or more accelerometers, gyroscopes, and other components. Here, the sensor 180 may also include a control circuit for controlling at least one of the included sensors. Any of these sensors 180 may be located within the electronic device 101. One or more cameras may optionally be used in conjunction with at least one flash 190. Flash 190 represents a device, such as one or more LEDs, configured to generate illumination for use in image capture by electronic device 101 .

[0066] The first external electronic device 102 or the second external electronic device 104 may be a wearable device or a wearable device (such as an HMD) in which the electronic device can be installed. When the electronic device 101 is installed in the electronic device 102 (such as an HMD), the electronic device 101 may communicate with the electronic device 102 through the communication interface 170. The electronic device 101 may be directly connected to the electronic device 102 to communicate with the electronic device 102 without a separate network. The electronic device 101 may also be an augmented reality wearable device, such as glasses, which includes one or more cameras.

[0067] Wireless communication can use, for example, at least one of the following as a cellular communication protocol: Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), a fifth generation wireless system (5G), millimeter wave or 60 GHz wireless communication, wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro), or global system for mobile communications (GSM). Wired connections can include, for example, at least one of the following: universal serial bus (USB), high-definition multimedia interface (HDMI), recommended standard 232 (RS-232), or plain old telephone service (POTS). Network 162 may include at least one communication network, such as a computer network (like a local area network (LAN) or a wide area network (WAN)), the Internet, or a telephone network.

[0068] Each of the first and second external electronic devices 102 and 104 and the server 106 may be a device of the same or different type as the electronic device 101. According to some embodiments of the present disclosure, the server 106 may include a group of one or more servers. Moreover, according to some embodiments of the present disclosure, all or part of the operations performed on the electronic device 101 may be performed on another or more other electronic devices (such as the electronic devices 102 and 104 or the server 106). Further, according to some embodiments of the present disclosure, when the electronic device 101 performs some functions or services automatically or based on a request, instead of the electronic device 101 performing the function or service by itself or additionally, it may request another device (such as the electronic devices 102 and 104 or the server 106) to perform at least part of the functions associated therewith. Other electronic devices (such as the electronic devices 102 and 104 or the server 106) are capable of performing the requested function or additional function and transmitting the execution result to the electronic device 101. The electronic device 101 may provide the requested function or service by processing the received result as is or in addition. To this end, for example, cloud computing, distributed computing, or client-server computing technology may be used. Although Figure 1The electronic device 101 is shown to include a communication interface 170 for communicating with an external electronic device 104 or a server 106 via a network 162 , but according to some embodiments of the present disclosure, the electronic device 101 may operate independently without a separate communication function.

[0069] The server 106 may optionally support the electronic device 101 by performing or supporting at least one operation (or function) implemented on the electronic device 101. For example, the server 106 may include a processing module or processor that may support the processor 120 implemented in the electronic device 101.

[0070] although Figure 1 An example of a network configuration 100 including an electronic device 101 is shown, but the network configuration 100 may also be Figure 1 Various changes may be made. For example, network configuration 100 may include any number of each component in any suitable arrangement. In general, computing and communication systems have a variety of configurations, Figure 1 The scope of the present disclosure is not limited to any particular configuration. Figure 1 One operating environment is shown in which the various features disclosed in this patent document may be used, but these functions may be used in any other suitable system.

[0071] Figure 2 An example process 200 for blending multiple image frames obtained using different exposure settings according to an embodiment of the present disclosure is shown. For ease of explanation, Figure 2 The process 200 shown in FIG. 2 is described as involving the use of Figure 1 Electronics 101. However, Figure 2 Process 200 shown in FIG. 2 may be used in any suitable system with any other suitable electronic device.

[0072] Operation 202 may be performed by using at least one image sensor, where the at least one image sensor may represent Figure 1 One or more sensors 180 in at least one camera of the electronic device 101 shown in FIG. Operations 204, 206, and 208 may be performed by at least one processor, which may represent Figure 1 One or more processors 120 of the electronic device 101 shown in FIG.

[0073] refer to Figure 2In operation 202, the image sensor 180 may be used to capture preview image frames and image frame bursts associated with a scene. Any suitable image sensor 180 may be used herein. The preview image frames may represent images detected in real time. An image frame burst may be a plurality of image frames of a scene captured continuously within a short time interval. The preview image frames are output to a display, such as the display 160 of the electronic device 101, so that a user may view the preview image frames and determine whether and when to initiate capture of an image frame burst.

[0074] In operation 204, a preview image frame may also be provided for scene analysis. The processor 120 may identify a baseline exposure time (typically expressed as EV+0) to be used to capture an image frame of the scene. For example, in some embodiments, the processor 120 may determine an exposure time appropriate for the scene based on the light level of the scene through an automatic exposure (AE) algorithm. Based on the AE algorithm, the processor 120 may select an exposure time to attempt to balance the captured image frame in terms of brightness (meaning that the captured image frame has the fewest underexposed and overexposed pixels). Thus, the processor 120 may operate to identify a baseline exposure time based on the light level detected by the image sensor just before the actual frame capture operation occurs.

[0075] In operation 206, the determined baseline exposure time may be provided for exposure control. The processor 120 may identify the number of image frames to be captured and the exposure settings to be used when capturing the image frames. The exposure settings to be used for capture may be set differently for each image frame. In an embodiment, in order to support the multi-exposure image frame blending technique described below, the processor 120 may determine that the image sensor 180 should capture one or more image frames at a baseline exposure time (EV+0) and capture at least one image frame at a shorter exposure time, such as at 1 / 2 exposure time (EV-1), 1 / 4 exposure time (EV-2), or 1 / 8 exposure time (EV-3).

[0076] In various embodiments of the present disclosure, the processor 120 may determine the order in which to capture longer exposure image frames and shorter exposure image frames using the image sensor 180. In some embodiments, the processor 120 may determine that the image sensor 180 should capture at least one shorter exposure image frame before capturing one or more longer exposure image frames. In other embodiments, the processor 120 may determine that the image sensor 180 should capture at least one shorter exposure image frame after capturing one or more longer exposure image frames. In yet other embodiments, the processor 120 may determine that the image sensor 180 should capture at least one shorter exposure image frame between the capture of at least two longer exposure image frames, meaning that the shorter exposure image frame has a capture time that is between the capture times of the longer exposure image frames.

[0077] In various embodiments of the present disclosure, the sensitivity of the image sensor 180 may also be controlled by the processor 120 during image capture, i.e., using a higher ISO level or other higher sensitivity (to provide higher gain) for shorter exposure image capture, and using a lower ISO level or other lower sensitivity (to provide lower gain) for longer exposure image capture. In some embodiments, different ISO levels or other sensitivities may be used for captured image frames to have more similar or substantially equal brightness levels. In certain embodiments, the exposure time and sensitivity may be selected so that the ratio between the longer and shorter exposure times substantially matches the ratio between the higher and lower sensitivities.

[0078] When capturing bursts of image frames, processor 120 may control image sensor 180 to capture multiple image frames at a determined exposure time and in a determined order. To support other types of image capture operations, processor 120 may determine other exposure times and numbers of image frames to be captured. For example, processor 120 may also support capturing multiple image frames using a common exposure setting, which are then combined.

[0079] In operation 208, a plurality of image frames in a burst captured by the image sensor 180 are provided for multi-frame blending. The processor 120 processes the captured image frames to produce at least one final image of the scene with minimal or no motion blur. The plurality of image frames include a reference image frame captured using a shorter exposure time and a plurality of non-reference image frames captured using a longer exposure time.

[0080] As described in more detail below, when mixing image frames captured using different exposure settings, the processor 120 can identify motion regions in the reference image frame. The motion regions represent areas that are prone to blurring due to motion. Motion regions are particularly prone to blurring in low light because exposure time increases as light levels decrease. The processor 120 can compare the reference image frame with one or more non-reference image frames to identify motion regions. In an embodiment, if multiple non-reference image frames are available, the processor 120 can perform multiple histogram matching operations.

[0081] In various embodiments of the present disclosure, the processor 120 may also generate one or more motion maps based on the reference and non-reference image frames, and blend the reference and non-reference image frames based on histogram matching and the motion maps. The processor 120 may also or instead perform contour reduction in the blended image frames. Example embodiments of blending operations 208 and contour reduction operations are described below, although other implementations of blending operations 208 and contour reduction operations may also be used.

[0082] Figure 3An example process for performing a multi-frame mixing operation according to an embodiment of the present disclosure is shown. The process may correspond to Figure 2 The multi-frame mixing operation 208 in the process. The process can be performed by at least one processor, and the at least one processor represents Figure 1 One or more processors 120 of the electronic device 101 are shown.

[0083] refer to Figure 3 , a set 302 of image frames may be captured using at least one image sensor 180 such as in a camera of the electronic device 101. Here, the set 302 includes at least three image frames 304, 306, and 308, however, the set 302 may also include two or more than three image frames.

[0084] The set 302 may include a plurality of image frames captured using different exposure settings. The plurality of image frames may include reference image frames captured using a shorter exposure time and a plurality of non-reference image frames captured using a longer exposure time. For example, image frames 304 and 308 may be non-reference image frames captured using a longer exposure time (such as an EV+0 exposure time). Image frames 304 and 308 may be captured using a lower sensitivity. Relative to image frames 304 and 308, image frame 306 may be a reference image frame captured using a shorter exposure time (such as an EV-1, EV-2, or EV-3 exposure time). Relative to image frames 304 and 308, image frame 306 may be captured using a higher sensitivity. Here, image frame 306 having a shorter exposure time may be captured between the capture of longer exposure image frames 304 and 308.

[0085] In embodiments, other numbers of image frames may be captured and other numbers of exposure settings may be used. As an example, one or more image frames 304 having a longer exposure time and a lower sensitivity may be captured before capturing image frame 306 (without capturing any image frame 308 after image frame 306), or one or more image frames 308 having a longer exposure time and a lower sensitivity may be captured after capturing image frame 306 (without capturing any image frame 304 before image frame 306). As another example, multiple image frames 304 having a longer exposure time and a lower sensitivity may be captured before capturing image frame 306, and multiple image frames 308 having a longer exposure time and a lower sensitivity may be captured after capturing image frame 306. As a specific example, two or more longer exposure image frames 304 may be captured before shorter exposure image frames 306, and two or more longer exposure image frames 308 may be captured after shorter exposure image frames 306.

[0086] As yet another example, multiple shorter exposure image frames may be captured, and one shorter exposure image frame may be selected to be used as a reference image frame in subsequent processing. For example, a shorter exposure image frame having the least amount of motion blur may be selected from the multiple shorter exposure image frames as a reference frame. In the following discussion, it may be generally considered that multiple non-reference image frames 304 and 308 are processed, although those skilled in the art may readily make appropriate modifications if a different number of non-reference image frames (including a single non-reference image frame) is used.

[0087] Furthermore, it should be noted that in some cases, the longer exposure image frame 304 or 308 may actually be clearer or have better overall quality than the shorter exposure image frame 306. This may be caused, for example, by a sudden vibration or other motion during the capture of the shorter exposure image frame 306. When the shorter exposure image frame 306 is of poor quality (such as compared to a certain threshold or to the other image frames 304, 308), the longer exposure image frame 304 or 308 may be selected as a reference frame.

[0088] In operation 310, a collection 302 of image frames may be provided for image registration. The processor 120 may align the image frames 304, 306, and 308 and generate aligned image frames. Alignment may be required if the electronic device 101 is moved or rotated between image captures and causes the objects in the image frames 304, 306, and 308 to also move or rotate slightly, which is often the case in handheld devices. Here the image frames 304, 306, and 308 may be aligned both geographically and photometrically.

[0089] The processor 120 may modify one or more non-reference image frames 304 and 308 to align with the reference image frame 306. The aligned versions of the image frames include a reference image frame 312a (which represents an aligned version of the reference image frame 306) and one or more non-reference image frames 312b (which represent aligned versions of the one or more non-reference image frames 304 and 308). Note that here the reference image frame 306 may or may not be modified during alignment (meaning that the image frame 312a may match the image frame 306), and the one or more non-reference image frames 304 and 308 may represent the only image frames that are modified during alignment. Any suitable technique for aligning the image frames may be used, and various alignment techniques are well known in the art. In some embodiments, the processor 120 may align the image frames by using global oriented FAST and rotational BRIEF (ORB) features and local features in a block search, however other implementations of the image registration operation 310 may also be used.

[0090] In operation 314, the aligned image frames 312a-312b may be provided for histogram matching. The processor 120 may adjust the brightness and color of the reference image frame 312a to make the brightness and color of all image frames 312a-312b substantially equal.

[0091] In various embodiments of the present disclosure, the processor 120 may perform a non-linear transformation of the reference image frame based on a plurality of histograms associated with a plurality of non-reference image frames. The processor 120 may match the histogram of the reference image frame 312a with one or more histograms of one or more non-reference image frames 312b, such as by applying a suitable transformation function to the reference image frame 312a.

[0092] As described in more detail below, in various embodiments of the present disclosure, the processor 120 may obtain a histogram map based on the difference between the reference image frame and the non-reference image frame, and modify the reference image frame by mapping the pixel values ​​of the reference image frame based on the histogram map. In some embodiments of the present disclosure, the processor 120 may obtain multiple histogram matching maps by using different image frames in the non-reference image frame 312b and the reference image frame 312a (assuming that there are multiple non-reference image frames 312b) to identify the photometric difference between the image frames 312a-312b. The processor 120 may combine the multiple histogram maps and modify the reference frame based on the combined histogram map. In some embodiments of the present disclosure, the processor 120 may identify motion pixels associated with motion from the reference image frame and the non-reference image frame, and obtain a histogram map based on pixels other than the motion pixels in the reference image frame and the non-reference image frame.

[0093] Histogram matching may result in the generation of a pre-processed aligned reference image frame 316. After image registration and histogram matching, the scene content in the non-reference image frame 312b and the pre-processed reference image frame 316 may closely match, except for moving objects in the scene.

[0094] An example embodiment of a histogram matching operation 314 is described below, however other implementations of the histogram matching operation 314 may also be used. In operation 318, one or more aligned non-reference image frames 312b and pre-processed aligned reference image frames 316 may be provided for frame fusion. The processor 120 may compare each non-reference image frame 312b with the reference image frame 316. For each comparison between a non-reference image frame and a reference image frame, the processor 120 may generate a motion map 320 that identifies local differences between the two compared image frames. Since the image frames being compared here have been previously aligned, the local differences between the two compared image frames indicate motion that has occurred between the two image frames.

[0095] In some embodiments, the values ​​of the motion map 320 may indicate the magnitude of the motion associated with each pixel. In an embodiment, the motion map may have a larger value for pixels determined to be associated with motion. In another embodiment, the motion map may have a smaller value for pixels determined to be associated with motion. Any suitable technique may be used to generate a motion map or other data identifying differences between image frames, and a variety of techniques are known in the art. As a specific example, the frame blending operation 318 may generate each motion map 320 using techniques disclosed in U.S. Patent Application Publication No. 2018 / 0192098 (which is incorporated herein by reference in its entirety).

[0096] In some embodiments, the processor 120 may generate n motion maps 320 for n image frames 312b, 316. The motion maps 320 may include one or more non-reference motion maps and reference motion maps. By comparing the non-reference image frame 312b with the reference image frame 316, a non-reference motion map may be generated for each non-reference image frame 312b. For example, if there are three original image frames (one longer exposure, one shorter exposure, and another longer exposure), the processor 120 may generate two non-reference motion maps, each of which is based on a comparison of the shorter exposure image frame with one of the two longer exposure image frames. If there are five original image frames (two longer exposures, one shorter exposure, and another two longer exposures), the processor 120 may generate four non-reference motion maps, each of which is based on a comparison of the shorter exposure image frame with one of the four longer exposure image frames. Another motion map, i.e., a reference motion map, may be associated with the reference image frame 316 itself. In an embodiment, since there will be no differences when comparing image frame 316 to itself, the reference motion map may be populated with values ​​indicating no motion was detected. For example, the reference motion map may have the same value, such as a minimum or maximum value, for all pixels of reference image frame 316.

[0097] In operation 322, the motion map 320 generated by frame fusion can be provided for motion mapping post-processing. The processor 120 can modify the reference motion map associated with the reference image frame 316 to support improved blending of the reference and non-reference image frames 316, 312b. For example, the processor 120 can analyze the motion map 320 and determine how to modify the reference motion map associated with the reference image frame 316 so as to reduce or minimize the contribution of the reference image frame 316 when blending the image frames. In other words, the processor 120 can reduce or minimize the image data from the shorter exposure image frame 306 used in the final image, which helps to improve or maximize the noise reduction in the final image (because the image frame 306 has a shorter exposure time and therefore has more noise than the other image frames 304 and 308).

[0098] Motion map post-processing may result in generation of a post-processed motion map 324 representing a modified reference motion map associated with reference image frame 316. One example implementation of motion map post-processing operation 322 is described below, although other implementations of motion map post-processing operation 322 may also be used.

[0099] In operation 326, the reference image frame 316, one or more non-reference image frames 312b, and motion maps 320 and 324 are provided for multi-frame blending. The processor 120 can combine the reference image frame 316 and the one or more non-reference image frames 312b based on the motion maps 320 and 324. For example, the values ​​in the motion maps 320 and 324 can be used as weights, where a larger value indicates that the image data contributes more to the blended image, and a smaller value indicates that the image data contributes less to the blended image (and vice versa). The blended image 328 generated by multi-frame blending can have less or no motion blur for any motion area as defined by the reference image frame 316, and can have more image details and less noise for any static area as defined by one or more non-reference image frames 312b.

[0100] Any suitable technique for combining two or more image frames based on motion mapping, such as by using a weighted combination of pixel values ​​from image frames 312 b and 316 , where the weights are defined by the motion mapping, may be used for multi-frame blending. In some embodiments, processor 120 may blend image data as follows:

[0101] Y blend = (Y 1 * W 1 + ... + Y n * W n ) / (W 1 + ... + W n)(1)

[0102] U blend = (U 1 * W 1 + ... + U n * W n ) / (W 1 + ... + W n ) (2)

[0103] V blend = (V 1 * W 1 + ... + V n * W n ) / (W 1 + ... + W n ) (3)

[0104] Here, Y 1 -Y n represents the brightness pixel value from the n image frames 312b and 316 being combined, U 1 -U n and V 1 -V n represents the chrominance pixel values ​​from the n image frames 312b and 316 being combined. 1 -W n represents the weights from the motion maps 320 and 324 associated with the n image frames 312b and 316 being combined. For example, for five input image frames, assume that the third image frame has a shorter exposure time. In this example, W 1 , W 2 , W 4 and W 5 The non-reference motion map 320 associated with the non-reference image frame 312b may be represented by W 3 The reference motion map 324 associated with the reference image frame 316 may be represented. blend , U blend and V blend represents the luminance and chrominance pixel values ​​contained in the blended image 328. Note, however, that other blending operations may occur here.

[0105] In operation 330, the mixed image 328 is provided for contour reduction. The processor 120 may estimate the likelihood of contour artifacts being formed and apply contour reduction only to those areas of the mixed image 328 where contour artifacts are likely to be present. The contour reduction may help reduce the likelihood of contour reduction (which may reduce image quality) being applied to areas of the mixed image 328 where contour reduction is not desired. The contour reduction may result in the generation of a processed mixed image 332.

[0106] When the contribution of the non-reference image frame 312b to the mixed image 328 is unbalanced, contour artifacts are likely to appear in areas. Based on this, the processor 120 can estimate the likelihood of contour artifacts. In some embodiments, the processor 120 can estimate the likelihood of contour artifacts by identifying any areas in the mixed image 328 where one non-reference image frame 312b contributes a lot of information to the mixed image 328, while the other non-reference image frame 312b does not. An example implementation of the contour reduction operation 330 is described below, however other implementations of the contour reduction operation 330 may also be used.

[0107] The processed mixed image 332 may undergo one or more post-processing operations to further improve the quality of the mixed image 332. For example, in operation 334, the processed mixed image 332 may undergo noise filtering and edge enhancement. The processor 120 may remove spatial noise and improve edge sharpness in the processed mixed image 332. Various techniques for edge enhancement and noise filtering are well known in the art. In some embodiments, noise filtering may represent a multi-scale denoising process guided by motion mapping 320, 324.

[0108] Figure 3 The output of the illustrated process is at least one final image 336 of the scene. Final image 336 may represent a processed blend of original image frames 304, 306, and 308. Final image 336 may have less or no motion blur in areas of motion, where the amount of motion blur in final image 336 may depend primarily on the amount of motion blur contained in original reference image frame 306. Furthermore, final image 336 may have improved image detail in stationary areas, which may depend primarily on the amount of image detail in original non-reference image frames 304 and 308. Figure 4 An example process for performing histogram matching according to an embodiment of the present disclosure is shown. The process may correspond to Figure 3 The histogram matching operation 314 in the process. The process can be performed by at least one processor, which can represent Figure 1 One or more processors 120 of the electronic device 101 are shown.

[0109] refer to Figure 4, in operations 402 and 404, a plurality of histogram matching maps may be generated. The processor 120 may generate a map from a source image frame to a target image frame. In this case, the source frame for both map generation operations 402 and 404 is the reference image frame 312a, and the target image frame in each map generation operation 402 and 404 is one of the two non-reference image frames 312b, which are denoted as image frame 312b 1 and 312b 2 to distinguish between different non-reference image frames 312b. This results in the generation of multiple histogram maps 406a-406b, which represent multiple samples of the photometric relationship between the reference image frame 312a and the non-reference image frame 312b. In some embodiments, each map generation operation 402 and 404 can generate a histogram map for each channel (Y, U and V) of the image frame.

[0110] Although two histogram matching map generation operations 402 and 404 are shown here to produce two histogram maps 406a-406b, this is based on the assumption that there are three image frames being processed (including two non-reference image frames). If there are more than two non-reference image frames, the processor 120 can generate more than two histogram matching maps. Moreover, if there is only one non-reference image frame, the processor 120 can generate one histogram matching map. An example implementation of the histogram matching map generation operation 402 or 404 is described below, but other implementations of the histogram matching map generation operations 402 and 404 may also be used.

[0111] In some embodiments, the processor 120 may select a predetermined number of non-reference frames from all non-reference frames and generate a histogram map for the selected non-reference image frames, rather than generating a histogram map for each non-reference frame. For example, even if there are more than three non-reference image frames, the processor 120 may select two non-reference image frames from these non-reference image frames and generate a histogram map for the selected non-reference image frames. In some embodiments, a non-reference image frame captured before the reference image frame and another non-reference image frame captured after the reference image frame are selected to generate a corresponding histogram map. In operation 408, the processor 120 may combine the histogram maps 406a-406b into a single combined histogram map 410. Any suitable technique for combining a plurality of histogram maps 406a-406b and generating a combined histogram map 410 may be used. In some embodiments, the processor 120 may determine the average value of the histogram maps 406a-406b to generate a combined histogram map 410. Since the histogram maps 406a-406b generally represent monotonically increasing functions, their averages may also be monotonically increasing functions. Note, however, that other techniques may be used to combine multiple histogram maps.

[0112] In some embodiments, the histogram maps 406a-406b and the combined histogram map 410 may include both a forward mapping and a reverse mapping, wherein one mapping involves mapping the first image frame to the second image frame and the other mapping involves mapping the second image frame to the first image frame. If only one histogram matching map is generated (such as when only one non-reference image frame is processed), the map combination operation 408 may be omitted or skipped, and the output of the histogram matching map generation operation 402 or 404 may be used as or instead of the combined histogram map 410.

[0113] In operation 412, the reference image frame 312a and the combined histogram map 410 may be provided for a forward mapping application. The processor 120 may apply the combined histogram map 410 (or a forward mapping of the combined histogram map 410) to the reference image frame 312a to modify the reference image frame 312a and generate a pre-processed reference image frame 316. The processor 120 may use any suitable technique to modify the reference image frame 312a based on the combined histogram map 410.

[0114] In some embodiments, processor 120 may use a lookup table that maps original luminance and / or chrominance pixel values ​​from reference image frame 312 a to new luminance and / or chrominance pixel values ​​for pre-processed reference image frame 316, where the lookup table is defined by combined histogram map 410. The resulting pre-processed reference image frame 316 may have a histogram that more closely matches the histogram of one or more non-reference image frames 312 b to help more closely match the brightness and color of image frames 312 b and 316.

[0115] Figure 5 An example process for performing histogram matching map generation according to an embodiment of the present disclosure is shown. The process may correspond to Figure 4 The histogram matching map generation operation 402 or 404 in the process. The process can be performed by at least one processor, which can represent Figure 1 One or more processors 120 of the electronic device 101 are shown.

[0116] refer to Figure 5, in operations 502 and 504, the image frames input, i.e., reference image frame 312a and a non-reference image frame 312b, can be provided for down sampling. Processor 120 can down sample the image data of the image frame to produce down sampled image frames 506a-506b. Any suitable amount of down sampling can be used here, such as down sampling to half, one quarter or one eighth of the original image data. Down sampling can allow to reduce the image data to be processed in subsequent operations, which can help speed up subsequent processing. Since the histogram map generation is a statistical operation, down sampling can be performed and the influence on the generated resulting histogram map is very small. However, down sampling is not necessarily needed here, and it can be omitted or skipped.

[0117] In operation 508, the downsampled image frames 506a-506b can be provided for histogram generation. The processor 120 can determine the absolute difference between the downsampled image frames 506a-506b (such as on a pixel-by-pixel basis) and generate at least one histogram 510 based on these absolute differences. The histogram 510 identifies how many times each absolute difference has occurred between the downsampled image frames 506a-506b. In an embodiment, a histogram can be generated for each channel of the image frame. For example, the processor 120 can generate a Y channel histogram based on the absolute difference of the brightness pixel value, and generate a U channel histogram and a V channel histogram based on the absolute difference of the chrominance pixel value.

[0118] In operation 512, histogram 510 may be provided for motion threshold generation. Processor 120 may identify one or more thresholds 514 based on histogram 510, such as a Y threshold, a U threshold, and a V threshold. Each threshold 514 defines a point above which a pixel value is considered to correspond to a motion pixel. In other words, threshold 514 may be used to identify which pixels in an image frame are associated with one or more moving objects in a scene. Processor 120 may use any suitable technique to identify threshold 514.

[0119] In some embodiments, processor 120 may generate threshold values ​​Y for luma and chroma pixel values ​​as follows MotTh , U MotTh and V MotTh :

[0120] Y MotTh = C * median(imhist(abs(Y ref – Y non-ref )))(4)

[0121] U MotTh = C * median(imhist(abs(U ref – U non-ref)))(5)

[0122] V MotTh = C * median(imhist(abs(V ref – V non-ref )))(6)

[0123] Here, Y ref , U ref and V ref Indicates the luminance and chrominance pixel values ​​of the reference image frame 506a, Y non-ref , U non-ref and V non-ref indicates the luminance and chrominance pixel values ​​of the non-reference image frame 506b, and abs(x) represents the absolute value of x. Therefore, imhist(abs(Y ref – Y non-ref )) represents the value in the histogram 510 generated in the histogram generating operation 508 based on the brightness value of the image frame, and imhist(abs(U ref – U non-ref )) and imhist(abs(V ref – V non-ref )) represents the value in the histogram 510 generated in the histogram generation operation 508 based on the chrominance value of the image frame. Moreover, median(·) represents a function that determines the median of the brightness or chrominance histogram values ​​of the histogram 510 generated in the histogram generation operation 508. In addition, C represents a multiplier for defining multiples of the median (assuming motion is occurring at this point). In some cases, the value of C can be 3, but other positive real numbers may also be used. However, the threshold value 514 can be determined in other ways, such as by using a technique that does not rely on a certain multiple of the median.

[0124] In operation 516, the downsampled image frames 506a-506b and the threshold 514 may be provided for motion pixel removal. When pixels in the downsampled image frames 506a-506b have values ​​that exceed their associated threshold 514, the processor 120 may remove these pixels from the downsampled image frames 506a-506b. In other words, when pixels are determined to be associated with motion, the processor 120 may remove or otherwise disregard these pixels.

[0125] In some embodiments, this results in the generation of a processed downsampled image frame 518, which contains image data associated only with non-excluded pixels (not associated with motion) in the image frames 506a-506b. The values ​​of the pixels associated with motion in the image frames 506a-506b can be assigned zero, which excludes the use of these pixels in the future. In other embodiments, the pixels of the image frames 506a-506b may not be modified. Instead, a motion pixel flag map indicating whether each pixel is determined to be a motion pixel may be generated, and the motion pixel flag map is provided together with the downsampled image frames 506a-506b for use in generating a histogram map.

[0126] In some embodiments using three thresholds 514 generated using equations (4)-(6) above, the processor 120 may determine whether each pixel of the image frames 506a-506b satisfies any of the following conditions:

[0127] abs(Y ref – Y non-ref ) > Y MotTh (7)

[0128] abs(U ref – U non-ref ) > Y MotTh (8)

[0129] abs(V ref – V non-ref ) > Y MotTh (9)

[0130] If at least one of these conditions is met, the associated pixel may be excluded when generating the histogram map. In an embodiment, a pixel in the image frame may be removed if one, two or all of the conditions in equations (7)-(9) are met.

[0131] In operation 520, the processed downsampled image frame 518 or the downsampled image frame and the motion pixel flag map may be provided for histogram generation. The processor 120 may generate one or more histogram maps 406a or 406b associated with the image frame 518. The processor 120 may create one or more histograms of pixel values ​​from each processed downsampled image frame 518 and match their counts to establish a relationship between two image frames (a non-reference image frame and a reference image frame).

[0132] During the generation of one or more histogram maps 406a or 406b, the processor 120 may not consider any pixels associated with motion because these pixels have been previously removed. In some embodiments, when generating the processed downsampled image frame 518, the processor 120 may not use pixels with a value of zero to create one or more histograms. In other embodiments, when a motion pixel flag map is generated, the processor 120 may not use pixels indicated as motion pixels by the motion pixel flag map to create one or more histograms. In some embodiments, the processor 120 may generate forward and reverse maps for each channel (Y, U, and V) of the downsampled image frame 518.

[0133] Removing motion pixels can greatly enhance the robustness of the histogram map 406a or 406b. Figure 4 As shown, to further enhance robustness, different non-reference image frames 312 (if available) may be used to perform multiple histogram matching map generation operations 402 and 404. In some embodiments, the non-reference image frames may include one or more non-reference image frames 312b captured before the reference image frame 316. 1 and one or more non-reference image frames 312b captured after the reference image frame 316 2 Ideally, the histogram maps 406a-406b generated from these image frames would be identical, but in practice, in many scenarios, there may often be slight differences between the histogram maps 406a-406b. Instead of using only one of the histogram maps 406a-406b, the histogram maps 406a-406b may be combined to provide a more reliable mapping between the multiple non-reference image frames 312b and the reference image frame 316.

[0134] Figure 6 An example process for performing motion mapping post-processing according to an embodiment of the present disclosure is shown. The process may correspond to Figure 3 The motion mapping post-processing operation 322 is performed during the process. The processing can be performed by at least one processor, which can represent Figure 1 One or more processors 120 of the electronic device 101 are shown.

[0135] As described above, the motion map 320 generated in the frame fusion operation 318 may include n motion maps for the n image frames being combined. The motion map 320 may include a reference motion map for the reference image frame 316 itself, which may be populated with a value indicating no motion (because the reference image frame 316 has no motion relative to itself). The reference motion map is represented as Figure 6The motion map 320 may also include n-1 non-reference motion maps for n-1 non-reference image frames 312b, wherein each of these non-reference motion maps is generated by comparing one non-reference image frame 312b with the reference image frame 316. These non-reference motion maps are represented as Figure 6 Motion map 320b in.

[0136] exist Figure 3 In a motion map post-processing operation 322 of the process, the processor can evaluate the motion map 320 to minimize the contribution of the reference frame 316, thereby maximizing noise reduction. In areas of motion, the reference image frame 316 will still contribute the most because the non-reference motion maps 320b for the non-reference image frames 312b will indicate the presence of local motion (which will limit or even eliminate their contribution to the blended image 328).

[0137] refer to Figure 6 In operation 606, the non-reference motion map 320b for the non-reference image frame 312b may be provided for motion map averaging. The processor 120 may average or otherwise combine the weights or other values ​​contained in the non-reference motion map 320b. This results in the generation of a combined motion map 608, which may provide a better measure of the overall motion level for each pixel in the non-reference image frame 312b. For example, if there are five image frames being processed (one reference image frame 316 and four non-reference image frames 312b), the processor 120 may determine the average value of the weights for each pixel in the non-reference motion map 320b as follows:

[0138] NRMM avg = (W 1 + W 2 + W 4 + W 5 ) / 4 (10)

[0139] Where W 1 , W 2 , W 4 and W 5 represents the value of the non-reference motion map 320b for a given pixel of the non-reference image frame 312b, NRMM avg represents W from the non-reference motion map 320b 1 , W 2 , W 4 and W 5 However, other techniques can be used to combine motion maps instead of calculating the average.

[0140] It should be noted here that Figure 6Assume that multiple non-reference image frames 312b are used to generate the combined multiple motion maps 320b. If a single non-reference image frame 312b is processed, the motion map comparison operation 606 may be omitted or skipped.

[0141] In operation 610, the reference motion map 320a for the reference image frame 316 and the combined motion map 608 for the non-reference image frame 312b may be provided for motion map comparison. The processor 120 may adjust the values ​​in the reference motion map 320a for the reference image frame 316 based on the combined motion map 608 for the non-reference image frame 312b, thereby generating a post-processed motion map 324. For example, for any pixel identified in the combined motion map 608 as having little or no motion, the processor 120 may minimize the contribution of the reference image frame 316 to the blended image 328. Conversely, for any pixel identified in the combined motion map 608 as having motion, the processor 120 may maximize the contribution of the reference image frame 316 to the blended image 328. The minimization of the reference image frame 316 may be limited by a minimum threshold 612, and the maximization of the reference image frame 316 may be limited by a maximum threshold 614. The specific thresholds 612 and 614 used herein may be adjusted in any suitable manner, such as based on an image quality assessment.

[0142] In some embodiments, processor 120 may generate the values ​​in post-processed motion map 324 as follows:

[0143] RMM upd = max[Th max – min(NRMM avg , Th max ), Th min ](11)

[0144] Here, RMM upd represents the update value contained in the post-processed motion map 324, Th min Indicates the minimum threshold 612, Th max Indicates the maximum threshold 614.

[0145] In a specific embodiment, assume that the non-reference motion map 320b for the non-reference image frame 312b contains pixel values ​​ranging between 0 and 255 (inclusive), and assume that the reference motion map 320a for the reference image frame 316 contains all pixel values ​​of 255. The minimum threshold 612 may be set to 32, and the maximum threshold 614 may be set to 128. Therefore, the pixel values ​​in the post-processed motion map 324 will range between 32 and 128 (inclusive).

[0146] In some embodiments, the values ​​of the combined motion map 608 and the post-processed motion map 324 may be complementary to each other. For a given pixel, if the value of the combined motion map 608 is small, it may indicate that the pixel is associated with motion. Therefore, the value of the post-processed motion map 324 may be determined to be large to maximize the contribution of the reference image frame 316. On the other hand, if the value of the combined motion map 608 is large, it may indicate that the pixel has little or no motion. Therefore, the value of the post-processed motion map 324 may be determined to be small to minimize the contribution of the reference image frame 316.

[0147] It should be noted here that in some implementations, the values ​​contained in the post-processed motion map 324 may not depend on the original values ​​contained in the motion map 320a. For example, the above formula (11) defines the RMM in the post-processed motion map 324 based on the average value contained in the combined motion map 604 and the two thresholds 612 and 614. upd The values ​​of motion map 320a are not referenced without reference to the original values ​​contained in motion map 320a. Therefore, in some embodiments, frame blending operation 318 may not generate motion map 320a, but instead a post-processed motion map 324 may be generated based on motion map 320b generated in frame blending operation 318 through motion map post-processing operation 322. However, other embodiments may be used in which the values ​​of motion map 320a are modified or otherwise taken into account when generating post-processed motion map 324. In addition, note that motion map 320b is not modified here, but other embodiments may be used in which motion map 320b is modified in motion map post-processing operation 322.

[0148] Figure 7 An example process for performing contour reduction according to an embodiment of the present disclosure is shown. The process may correspond to Figure 3 The process may be performed by at least one processor, which may represent a contour reduction operation 330. Figure 1 One or more processors 120 of the electronic device 101 are shown.

[0149] As described above, in the contour reduction operation 330, the processor 120 may estimate the likelihood of the contour artifact occurring and apply contour reduction only in the area where the contour artifact is likely to occur, so that the image quality in other areas of the mixed image 328 is not affected. When the contribution of the non-reference image frame 312b to the mixed image 328 is unbalanced, the contour artifact is likely to occur. Based on this, the processor 120 may estimate the likelihood of the contour artifact.

[0150] refer to Figure 7In operation 702, a plurality of motion maps 320b associated with a plurality of non-reference image frames 312b may be provided for blend level comparison. The processor 120 may compare the contributions of different non-reference image frames 312b to a blended image 328, wherein the blended image is defined by or otherwise based on the non-reference motion maps 320b for the non-reference image frames 312b. The processor 120 may therefore compare the non-reference motion maps 320b and determine for local regions or pixels whether one or more of the non-reference motion maps 320b indicate no motion, while one or more other non-reference motion maps 320b indicate motion. This results in the generation of a comparison level 704, which identifies (possibly on a per-region or per-pixel basis) whether the non-reference motion maps 320b consistently identify motion.

[0151] In some embodiments, the processor 120 may process the non-reference motion map 320b and generate a comparison level 704 by identifying, for each pixel, a contribution ratio between the non-reference image frames 312b as follows:

[0152] ContEq = max(MotMap) / [mean(MotMap) * NumMM] (12)

[0153] Here, ContEq represents a comparison level, which is a contribution ratio between non-reference image frames 312b for a certain area or pixel. When the non-reference image frames 312b contribute equally to the mixed image 328, the ratio has a relatively low value, and when the non-reference image frames 312b contribute unequally to the mixed image 328, the ratio has a relatively high value. When the non-reference motion map 320b for the non-reference image frame 312b is uniformly zero for the area or pixel, the value of the ContEq ratio may also be zero. Moreover, max(MotMap) represents the maximum value for the area or pixel in the set of non-reference motion maps 320b, and mean(MotMap) represents the average value for the area or pixel in the set of non-reference motion maps 320b. In addition, NumMM represents the number of non-reference motion maps 320b processed.

[0154] In operation 706, a comparison level 704 may be provided for contour likelihood estimation. The processor 120 may analyze the comparison level 704 to determine whether contour artifacts are likely to form in any given pixel or region of the mixed image 328. As mentioned above, the comparison level 704 may identify whether the non-reference motion map 320b consistently identifies motion. If the non-reference motion map 320b does not consistently identify motion, the confidence level for blending the associated pixel or region is low, and it is considered likely that contour artifacts will occur. Note that blending may not be disabled here, even if contour artifacts may be formed. It has been found that contour signals are primarily contained in the low-frequency luminance (Y) channel, so blending may be enabled, and contour artifacts may be reduced or minimized by filtering the luminance channel as described below. Therefore, the processor 120 may use the comparison level 704 to identify situations where the motion map 320b does not consistently identify motion for pixels or regions of the mixed image 328. This results in the generation of a contour flag 708, which indicates whether contour artifacts are likely (such as on a per-pixel or per-region basis).

[0155] In some embodiments, processor 120 may process comparison level 704 to identify profile likelihoods as follows:

[0156] CR en = min[max((ContEq – CR offset ), 0), CR thr ](13)

[0157] Here, CR en represents the contour flag 708, which is the likelihood of contour artifacts and can serve as an enable signal for contour reduction. offset is a tuning parameter related to the offset applied to the contribution ratio between the non-reference image frames 312b. thr is a tuning parameter related to the minimum likelihood required before determining that there may be contouring artifacts, and its value can be 0.25 or some other positive real number.

[0158] In operation 710, the reference image frame 316 and the blended image 328 (or perhaps just the luminance data from the reference image frame 316 and the blended image 328) may be provided for difference determination. The processor 120 may identify the difference between the luminance data in the reference image frame 316 and the blended image 328. This results in the generation of an image difference 712, which effectively identifies a Y channel update for comparison with the reference image frame 316 to generate the blended image 328.

[0159] In operation 714, the reference image frame 316 may also be provided for edge intensity filtering. The processor 120 may identify edge locations of one or more objects in the reference image frame 316. This results in the generation of an edge indicator 716, which may indicate whether each pixel or region of the image frame is associated with an object edge. The processor 120 may use any suitable technique to identify edge locations, such as by using the techniques disclosed in U.S. Patent Application Publication No. 2018 / 0192098 (which is incorporated herein by reference above). An example embodiment of the edge intensity filtering operation 714 is described below, however other implementations of the edge intensity filtering operation 714 may also be used.

[0160] In some embodiments, processor 120 may generate edge indicator 716 as follows:

[0161] ESFY ref = max[ESF(Y ref ), C esf ] (14)

[0162] Here, ESFY ref represents edge indicator 716. Also, ESF(Y ref ) represents the brightness data (Y ref ). In addition, C esf represents a tuning parameter, which in some embodiments may have a value of 10 or some other positive real number.

[0163] In operation 718, the image difference 712 and the edge indicator 716 may be provided for normalization. The processor 120 may normalize the image difference 712 based on the edge indicator 716. This results in generating a normalized image difference 720. By normalizing the operation 718, the contour reduction operation 330 may generally avoid applying contour reduction in areas where the edges of one or more objects in the scene may be located (and in areas of the image where actual desired boundaries or contours are expected to be found).

[0164] In some embodiments, processor 120 may process image difference 712 and edge indicator 716 as follows:

[0165] Norm diff = C esf * Y diff / ESFY ref (15)

[0166] Here, Y diff Represents image difference 712, Norm diff 720 represents the normalized image difference.

[0167] In operation 722, the normalized image difference 720 may be provided for filtering. The processor 120 may filter the normalized image difference 720 and produce a filtered normalized image difference 724. The filtering operation 722 may help remove high frequency components from the normalized image difference 720 because, as mentioned above, contour artifacts may reside primarily at lower frequencies in the luminance channel. Therefore, the processor 120 may perform a low pass filtering operation or other filtering operation to remove high frequency components from the normalized image difference 720. The processor 120 may use any suitable filtering technique, such as using a 7x7 uniform filter.

[0168] In some embodiments, processor 120 may process normalized image difference 720 as follows:

[0169] Y diffLPF = Norm diff ** LPF 7x7 (16)

[0170] Here, Y diffLPF represents the filtered normalized image difference 724 after low pass filtering, LPF 7x7 Represents the function of a 7x7 uniform filter. Of course, other uniform, non-uniform or other filters can also be used here.

[0171] In operation 726, the mixed image 328, the contour marker 708, and the filtered normalized image difference 724 may be provided to a contour reduction application. The processor 120 may use the contour marker 708 and the filtered normalized image difference 724 to reduce or minimize contour artifacts in the mixed image 328. For example, the processor 120 may subtract the filtered normalized image difference 724 (or a scaled version of the filtered normalized image difference 724) from the mixed image 328, but only for pixels or areas identified by the contour marker 708 as likely to have contour artifacts. As another example, the processor 120 may obtain a contour signal based on the filtered normalized image difference 724 and the contour marker 708, and subtract the contour signal from the mixed image 328. This results in the generation of a processed mixed image 332, or at least brightness data of the processed mixed image 332.

[0172] In some embodiments, processor 120 may generate brightness data for processed mixed image 332 as follows:

[0173] Y upd = Y blend – (Y diffLPF *CR en / CR thr ) (17)

[0174] Here, Y blend represents the brightness data of the mixed image 328, Y upd Y represents the brightness data of the processed mixed image 332. diffLPF *CR en / CR thr is determined as the contour signal to be eliminated from the mixed image 328. According to Formula 17, if CR en If the value is zero for a region or pixel, there will be no pixel update so that the pixel or region of the blended image 328 matches the corresponding pixel or region of the processed blended image 332. Also, if CR en If the value is non-zero for a region or pixel, contour reduction is applied to the region or pixel, so that the low frequency content of the processed mixed image 332 becomes lower compared to the low frequency content of the mixed image 328.

[0175] Figure 8 An example edge intensity filter according to an embodiment of the present disclosure is shown. Specifically, Figure 8 The edge strength filter in can be obtained by Figure 7 The edge strength filter operation 714 is used during the process.

[0176] By using an edge intensity filter, the processor 120 can detect changes in the brightness values ​​of adjacent pixel pairs 802. These brightness changes can indicate edges or textures in the image frame. Here, the processor 120 can detect changes in the brightness values ​​of each pixel pair 802 within a moving 3x3 pixel window 804. More specifically, the processor 120 can compare the pixels 802 in each group of two pixels relative to each other about the center pixel 802 in the window 804. This results in a comparison of the pixels 802 along the horizontal axis, the vertical axis and the two diagonal axes in the window 804. In addition, the pixel window 804 can be moved to cover all or substantially all of the image frames so as to perform brightness comparisons on all or substantially all of the image frames.

[0177] In this way, the processor 120 can detect edges (defined by large changes in brightness values ​​between compared pixels 802) that are arranged vertically, horizontally, or diagonally within the window 804. This can be referred to as "non-directional" edge detection. In some embodiments, the comparison of pixels 802 can be performed by adding the absolute values ​​of the differences in the Y values ​​of the pixels 802, and the resulting value can indicate how strongly the edge is detected. However, any other suitable edge detection mechanism can be used here.

[0178] although Figure 2 An example of a process 200 for blending multiple image frames is shown. Figures 3 to 8 Shows Figure 2The operation example in process 200 is shown, but it can be Figures 2 to 8 For example, although shown as a sequence of steps, Figures 2 to 8 The various operations shown may overlap, occur in parallel, occur in a different order, occur any number of times, or be omitted. Figures 3 to 8 The specific operations shown are only examples, and other techniques may also be used to perform Figures 3 to 8 Each operation shown.

[0179] Figures 2 to 8 The operations shown may be implemented in the electronic device 101 in any suitable manner. For example, in some embodiments, Figures 2 to 8 The operations shown may be implemented or supported using one or more software applications or other software instructions executed by the processor 120 of the electronic device 101. Figures 2 to 8 At least some of the operations shown may be implemented or supported using dedicated hardware components. Figures 2 to 8 The operations illustrated may be performed using any suitable hardware or any suitable combination of hardware and software / firmware instructions.

[0180] The above Figures 2 to 8 The operations shown are described as being performed using a specific implementation of the electronic device 101, but many modifications may be made based on the electronic device 101 actually used. For example, the electronic device 101 may include or use a single camera or multiple cameras. If a single camera is used, multiple image frames of the scene may be captured sequentially, such as in one or more fast bursts. If multiple cameras are used, it is possible to capture multiple image frames concurrently or in an overlapping manner, such as by capturing multiple image frames of the scene simultaneously, but using different cameras to capture with different camera exposure times. Some specific examples of multi-camera settings may include using wide-angle and telephoto cameras, wide-angle and monofocal cameras, or wide-angle, telephoto and monofocal cameras. Additional processing may be performed to handle baseline differences between cameras. If desired, multiple image frames of the scene may still be captured sequentially using at least one of the multiple cameras. Assuming that the geometry of the multiple cameras is known in advance, the geometry may be used to help align the image frames captured by the cameras or perform other functions.

[0181] As another example, the electronic device 101 is described above as performing various operations using YUV image data. However, data in other domains (such as RGB or Bayer data) may also be used or processed. As a third example, the techniques described in this document may be combined with other image processing algorithms in a software library used by, for example, the electronic device 101. A user of the electronic device 101 may select different image processing algorithms, such as based on a specific situation or based on user preferences. As a fourth example, the electronic device 101 may use the above techniques to support capturing still images or video sequences of scenes. The capture settings of the video sequence may be alternately varied to provide image frames with different exposure times, and the high frame rate capture capability may help to mask the frame rate reduction caused by frame selection. As a fifth example, the electronic device 101 may include an inertial measurement unit as described above, and the inertial measurement unit may provide input for the hybrid operation 208.

[0182] In addition, it should be noted that the present disclosure describes (i) the multi-frame mixing technique and (ii) the contour reduction technique. However, it is not required that these techniques should be used together in the same electronic device 101. The electronic device 101 can be used to implement the multi-frame mixing technique or the contour reduction technique (but not necessarily both). Of course, these two techniques can also be implemented in the same electronic device 101.

[0183] Fig. 9 is a flow chart illustrating an example method 900 for blending multiple image frames captured using different exposure settings according to an embodiment of the present disclosure. For ease of explanation, Fig. 9 The method 900 shown in FIG. 1 is described as involving using Figure 1 electronic device 101. However, Fig. 9 The illustrated method 900 may also be used in any suitable system with any other suitable electronic device.

[0184] refer to Fig. 9In step 902, an electronic device may be used to capture multiple image frames of a scene. The processor 120 of the electronic device 101 may receive a capture request and control the image sensor 180 to capture two or more image frames 304, 306, 308 of the scene using different exposure settings. The image frames captured here may include at least one image frame 304, 308 generated using a longer exposure time and a lower sensitivity, and at least one image frame 306 generated using a shorter exposure time and a higher sensitivity. The shorter exposure image frame 306 may be used as a reference image frame, and the longer exposure image frames 304, 308 may be used as one or more non-reference image frames. In some embodiments, the ratio between the longer and shorter exposure times may substantially match the ratio between the higher and lower sensitivities, so that different image frames 304, 306, 308 have substantially equal overall brightness. As mentioned above, the image frames may be captured in various orders, such as capturing a longer exposure image frame before and / or after a shorter exposure image frame. Moreover, as mentioned above, multiple shorter exposure image frames may be captured here, and one of these shorter exposure image frames may be selected as a reference image frame.

[0185] In step 904, the image frames may be aligned. The processor 120 of the electronic device 101 may modify one or more non-reference image frames 304, 308 to align the non-reference image frames 304, 308 with the reference image frame 306, thereby generating aligned image frames 312a-312b.

[0186] In step 906, a non-linear transformation of the reference image frame may be performed using histogram matching. The processor 120 of the electronic device 101 may adjust the brightness and color of the reference image frame 306 to match the non-reference image frames 304, 308. For example, the processor 120 may perform a non-linear transformation of the reference image frame 306 based on at least one histogram associated with at least one non-reference image frame.

[0187] In some embodiments, the processor 120 of the electronic device 101 may generate one or more histogram maps 406a-406b, wherein each histogram map 406a-406b may be generated based on the difference between the aligned reference image frame 312a and an aligned non-reference image frame 312b. If multiple non-reference image frames 312b are available, the processor 120 of the electronic device 101 may generate multiple histogram maps 406a-406b and combine the multiple histogram maps 406a-406b.

[0188] In an embodiment, the processor 120 of the electronic device 101 may identify motion pixels associated with motion and obtain a histogram map based on pixels other than the motion pixels. For example, the processor 120 may identify an absolute difference between pixel values ​​from an aligned reference image frame 312a and an aligned non-reference image frame 312b, and generate at least one histogram 510 based on the absolute difference. The processor 120 may identify one or more motion thresholds 514 based on the at least one histogram 510, and determine a pixel having at least one absolute difference exceeding at least one of the one or more motion thresholds as a motion pixel.

[0189] The processor 120 of the electronic device 101 may modify the reference image frame 312a based on the histogram maps 406a-406b to more closely match the non-reference image frame 312b, for example, in brightness and color, thereby generating a pre-processed aligned reference image frame 316. For example, the processor 120 may map pixel values ​​of the reference image frame 312a based on the histogram maps 406a-406b.

[0190] In step 908, a motion map that identifies portions of the image frame that are susceptible to motion blur may be generated and the motion map may be preprocessed. The processor 120 of the electronic device 101 may obtain a plurality of motion maps that correspond to the plurality of image frames, respectively. For example, the processor 120 may generate one or more non-reference motion maps 320b that identify the difference (motion) between each of the one or more non-reference image frames 312b and the reference frame 316. The processor 120 of the electronic device 101 may modify the reference motion map 320a to generate a post-processed motion map 324, or the processor 120 of the electronic device 101 may generate the motion map 324, in either case based on an average or other combination of the non-reference motion maps 320b (or based on a single non-reference motion map 320b if only one non-reference motion map 320b is available).

[0191] In step 910, the image frames may be blended using a motion map to generate a blended image. The processor 120 of the electronic device 101 may blend the reference image frame 316 with one or more non-reference image frames 312b based on the motion map 324 to generate a blended image 328. The processor 120 may combine regions with less or no motion in one or more non-reference image frames 312b with regions containing motion in the reference image frame 316. For example, the processor 120 may perform a weighted average on the reference image frame 316 and the non-reference image frame 312b by using the motion map 324 as a weight. This may reduce or minimize motion blur in the blended image 328 while increasing image detail and reducing noise in the blended image 328.

[0192] In step 912, contour reduction is applied to the mixed image. The processor 120 of the electronic device 101 can identify areas where the motion map 320b does not consistently identify motion. In some embodiments, the processor 120 of the electronic device 101 can identify filtered normalized differences between the reference image frame 316 and the mixed image 328 at locations where no edges are expected based on the reference image frame 316. The processor 120 of the electronic device 101 can subtract the filtered normalized differences or a scaled version of the filtered normalized differences from the mixed image 328 at locations where the motion map 320b does not consistently identify motion to produce a processed mixed image 332.

[0193] Note that this form of contour reduction relies on the presence of multiple motion maps 320b for multiple non-reference image frames 312b, and therefore this form of contour reduction may not be used in the case of only a single non-reference image frame 312b. In this case, contour reduction may be omitted, or other forms of contour reduction may be used.

[0194] Any desired post-processing of the blended image may occur in step 914. The processor 120 of the electronic device 101 may perform spatial noise filtering and edge enhancement.

[0195] In step 916, the post-processed output is a final image of the scene, which can be stored, output, or used in some manner. For example, the processor 120 of the electronic device 101 can display the final image of the scene on the display 160 of the electronic device 101. The processor 120 of the electronic device 101 can save the final image of the scene to a camera roll stored in the memory 130 of the electronic device 101. The processor 120 of the electronic device 101 can attach the final image of the scene to a text message, email, or other communication to be sent from the electronic device 101. Of course, the final image of the scene can be used in any other or additional ways.

[0196] although Fig. 9 One example of a method 900 for blending multiple image frames obtained using different exposure settings is shown, but may be used for Fig. 9 Various changes may be made. For example, although shown as a series of steps, Fig. 9 The various steps in may overlap, occur in parallel, occur in a different order, occur any number of times, or be omitted.

[0197] Fig.10 1 is a flow chart illustrating an example method 1000 for contour reduction in a mixed image according to an embodiment of the present disclosure. For ease of explanation, Fig.10 The method 1000 is described as involving using Figure 1electronic device 101. However, Fig.10 The method 1000 shown may also be used in any suitable system with any other suitable electronic device. The method 1000 may correspond to Fig. 9 Step 912 of the method.

[0198] refer to Fig.10 In step 1002, the mixing levels for the plurality of image frames may be compared based on the motion maps of the plurality of image frames. The processor 120 of the electronic device 101 may compare the non-reference motion maps 320b for the non-reference image frames 312b with each other to identify pixels or regions in which the non-reference motion maps 320b do not consistently identify motion. As a specific example, the processor 120 of the electronic device 101 may generate a ratio based on the maximum value and the average value of the non-reference motion map 320b, for example, for each pixel or region.

[0199] The result of the comparison may be used to determine the likelihood of contour artifacts in step 1004. The processor 120 of the electronic device 101 may compare the ratios or other results of the aforementioned comparisons to identify pixels or regions where contour artifacts may exist.

[0200] In step 1006 , a difference between the reference image frame and the mixed image may be identified. The processor 120 of the electronic device 101 may identify a difference between the mixed image 328 and the reference image frame 316 in at least brightness data.

[0201] In step 1008, edge strength filtering is performed using the reference image frame. The processor 120 of the electronic device 101 may compare pairs of pixels in the reference image frame 316 that are opposite about a center pixel in a moving 3x3 window.

[0202] In step 1010, the difference between the reference image frame and the mixed image may be normalized based on the result of the edge intensity filtering. The processor 120 of the electronic device 101 may divide the difference in at least brightness data between the mixed image 328 and the reference image frame 316 by the likelihood that the difference may be at the edge of an object.

[0203] In step 1012, the obtained normalized image difference may be filtered. The processor 120 of the electronic device 101 may apply a 7x7 uniform filter or other filter to the normalized image difference. By applying the filter, the processor 120 may filter out high frequency components from the normalized image difference.

[0204] In step 1014, contour reduction is applied to the mixed image to produce an updated mixed image. For example, the processor 120 of the electronic device 101 subtracts the filtered normalized image difference or a scaled version of the filtered normalized image difference from the mixed image 328, but only at locations where contour artifacts may occur (this is based on the comparison of the motion map 320b associated with the non-reference image frame 312b). As another example, the processor 120 can obtain a contour signal based on the filtered normalized image difference 724 and the contour marker 708, and subtract the contour signal from the mixed image 328.

[0205] In step 1016, the updated mixed image may be stored, output, or otherwise used. The processor 120 of the electronic device 101 may display, store, or transmit the updated mixed image. The processor 120 of the electronic device 101 may perform any desired post-processing on the updated mixed image, such as spatial noise filtering and edge enhancement.

[0206] although Fig.10 An example of a method 1000 for blending images with fewer contours is shown, but may be used for Fig.10 Various changes may be made. For example, although shown as a series of steps, Fig.10 The various steps in may overlap, occur in parallel, occur in a different order, occur any number of times, or be omitted.

[0207] Fig.11A and Fig. 11B An example result obtained by blending multiple image frames captured using different exposure settings according to an embodiment of the present disclosure is shown.

[0208] like Fig.11A As shown, image 1100 of a scene captured using standard image capture techniques includes two vehicles and a person on a bicycle. The vehicles are moving, and image 1100 contains a large amount of motion blur due to the movement of the vehicles. The motion blur is bad enough to make it very difficult (if not impossible) to read the license plates of the vehicles. Fig. 11B As shown, an image 1102 of the same scene is captured using the multi-exposure multi-frame blending technique described above, such as the case where a shorter exposure image frame is captured between longer exposure image frames. It can be seen here that image 1102 contains less motion blur, and it is now much easier to read the license plate of the vehicle, while minimizing the noise caused by the short exposure. Therefore, the above process can be used to combine image frames captured using different exposure settings to effectively reduce motion blur and improve image details in the resulting blended image.

[0209] although Fig.11A and Fig. 11BExamples of results that can be obtained by mixing multiple image frames captured using different exposure settings are shown, but various changes can be made to these figures. For example, these figures are merely examples of the types of results that can be obtained using the methods described in this disclosure. Obviously, the image of the scene can vary greatly, and the results obtained using the methods described in this patent document can also vary greatly depending on the situation.

[0210] Fig. 12A and Fig. 12B An example result obtained by performing contour reduction on a mixed image according to an embodiment of the present disclosure is shown.

[0211] like Fig. 12A As shown, a scene image 1200 generated using a standard image blending technique includes a ceiling fan that is rotating. Because the ceiling fan is rotating and there is a generally uniform background, contour artifacts 1202 are generated in the image 1200. These contour artifacts 1202 cause the shadow of the ceiling fan blade to appear to include multiple different parts, which is not the case with real shadows. Fig. 12B 1, an image 1204 of the same scene is generated using the contour reduction techniques described above. Here, it can be seen that image 1204 contains few contour artifacts (if any contour artifacts) in the shadow of the ceiling fan blade, making the shadow appear more natural against a generally consistent background. Therefore, the above process can be used to effectively reduce contour artifacts in a blended image generated using multiple image frames.

[0212] although Fig. 12A and Fig. 12B Examples of results obtained by performing contour reduction on a mixed image are shown, but various changes may be made to these figures. For example, these figures are merely examples of the types of results that can be obtained using the methods described in this disclosure. Obviously, images of scenes can vary greatly, and the results obtained using the methods described in this patent document can also vary greatly depending on the situation.

[0213] Although the present disclosure has been described with reference to various exemplary embodiments, various changes and modifications may occur to those skilled in the art. The present disclosure is intended to encompass such changes and modifications as fall within the scope of the appended claims.

Claims

1. A method for multi-frame mixing, the method comprising: obtaining a plurality of image frames of a scene, the plurality of image frames comprising a reference image frame captured using a shorter exposure time and a higher sensitivity and a plurality of non-reference image frames captured using a longer exposure time and a lower sensitivity; adjusting brightness and color of the reference image frame to match the plurality of non-reference image frames; Obtaining a plurality of motion maps corresponding to the plurality of image frames respectively, the plurality of motion maps comprising a reference motion map corresponding to the reference image frame and at least one non-reference motion map corresponding to the plurality of non-reference image frames; Based on the reference motion map and the at least one non-reference motion map, blending the reference image frame with the plurality of non-reference image frames into a blended image, wherein at least one motion region of the blended image is more based on the reference image frame, and at least one static region of the blended image is more based on the plurality of non-reference image frames; as well as Contour reduction is performed on the mixed image to reduce contour artifacts in the mixed image. 2 . The method of claim 1 , wherein the plurality of non-reference image frames include at least one image frame captured before the reference image frame and at least one image frame captured after the reference image frame.

3. The method according to claim 1, wherein obtaining the plurality of image frames comprises: obtaining a plurality of image frames captured using a shorter exposure time; as well as One of the plurality of image frames captured using a shorter exposure time is selected as the reference image frame.

4. The method according to claim 1, wherein obtaining the plurality of image frames comprises: The plurality of non-reference image frames are modified to be aligned with the reference image frame.

5. The method according to claim 1, wherein adjusting the brightness and color of the reference image frame comprises: obtaining a histogram map based on differences between the reference image frame and the plurality of non-reference image frames; The reference image frame is modified by mapping pixel values ​​of the reference image frame based on the histogram map.

6. The method of claim 5, wherein obtaining the histogram map comprises: Obtaining a plurality of histogram maps corresponding to the plurality of non-reference image frames respectively; as well as The plurality of histogram maps are combined to obtain the histogram map.

7. The method of claim 5, wherein obtaining the histogram map comprises: identifying motion pixels associated with motion from the reference image frame and the plurality of non-reference image frames; as well as The histogram map is obtained based on pixels other than the motion pixels in the reference image frame and the plurality of non-reference image frames.

8. The method of claim 7, wherein identifying the motion pixels comprises: identifying absolute differences between pixel values ​​from the reference image frame and the plurality of non-reference image frames; generating at least one histogram based on the absolute differences; identifying one or more motion thresholds based on the at least one histogram; as well as A pixel having at least one absolute difference value exceeding at least one of the one or more motion thresholds is determined as the motion pixel.

9. The method of claim 1 , wherein obtaining the plurality of motion maps comprises: obtaining a plurality of non-reference motion maps based on differences between the reference image frame and the plurality of non-reference image frames, respectively; combining the plurality of non-reference motion maps to obtain a combined non-reference motion map; as well as obtaining a reference motion map corresponding to the reference image frame based on the combined non-reference motion map, The reference motion map and the combined non-reference motion map complement each other, so that: if the value of the pixel in the combined non-reference motion map is smaller, the value of the pixel in the reference motion map is larger; if the value of the pixel in the combined non-reference motion map is larger, the value of the pixel in the reference motion map is smaller.

10. The method of claim 1, wherein blending the reference image frame and the plurality of non-reference image frames into a blended image comprises: A weighted averaging of the reference image frame and the non-reference image frame is performed by using the reference motion map as a weight of the reference image frame and using the at least one non-reference motion map as a weight of the plurality of non-reference image frames.

11. The method of claim 1 , wherein performing contour reduction on the mixed image comprises: identifying locations, relative to the reference image frame, where motion maps associated with the plurality of non-reference image frames consistently or inconsistently identify motion; as well as Based on the motion map inconsistently identifying locations of motion, pixels or areas in the mixed image are identified to which contour reduction is applied.

12. The method of claim 11, wherein performing contour reduction on the mixed image further comprises: identifying an image difference in at least brightness data between the reference image frame and the mixed image; normalizing the image difference based on an object edge identified in the reference image frame to generate a normalized image difference; filtering the normalized image difference to generate a filtered normalized image difference; as well as The filtered normalized image difference or a scaled version of the filtered normalized image difference is subtracted from the mixed image only for pixels or regions identified in the mixed image to which contour reduction is to be applied.

13. The method of claim 12, wherein filtering the normalized image difference comprises: Perform low pass filtering on the normalized image difference using a 7x7 uniform filter.

14. The method according to claim 1, further comprising: At least one of noise filtering and edge sharpening is performed on the contour-reduced mixed image.

15. An electronic device, comprising: at least one image sensor; at least one memory storing instructions; as well as at least one processor configured to execute the stored instructions to: controlling the at least one image sensor to obtain a plurality of image frames of a scene, the plurality of image frames comprising a reference image frame captured using a shorter exposure time and a higher sensitivity and a plurality of non-reference image frames captured using a longer exposure time and a lower sensitivity; adjusting brightness and color of the reference image frame to match the plurality of non-reference image frames; Obtaining a plurality of motion maps corresponding to the plurality of image frames respectively, the plurality of motion maps comprising a reference motion map corresponding to the reference image frame and at least one non-reference motion map corresponding to the plurality of non-reference image frames; blending the reference image frame with the plurality of non-reference image frames into a blended image based on the reference motion map and the at least one non-reference motion map, wherein at least one motion region of the blended image is more based on the reference image frame and at least one static region of the blended image is more based on the plurality of non-reference image frames; as well as Contour reduction is performed on the mixed image to reduce contour artifacts in the mixed image.

16. A method for contour reduction, the method comprising: obtaining a hybrid image by combining a plurality of image frames including a reference image frame and a plurality of non-reference image frames; identifying locations, relative to the reference image frame, where motion maps associated with the plurality of non-reference image frames consistently or inconsistently identify motion; identifying pixels or regions in the hybrid image to which contour reduction is to be applied based on the motion map inconsistently identifying locations of motion; as well as Contour reduction is applied to the identified pixels or regions to generate a processed hybrid image.

17. The method of claim 16, wherein applying contour reduction to the mixed image further comprises: identifying an image difference in at least brightness data between the reference image frame and the mixed image; normalizing the image difference based on an object edge identified in the reference image frame to generate a normalized image difference; filtering the normalized image difference to generate a filtered normalized image difference; as well as The filtered normalized image difference or a scaled version of the filtered normalized image difference is subtracted from the mixed image only for pixels or regions identified in the mixed image to which contour reduction is to be applied.

Images