System and method for motion warping using multiple exposure frames

Abstract

A method is provided, comprising: obtaining a first image frame and a plurality of second image frames of a scene, each of the plurality of second image frames having a different exposure time than an exposure time of the first image frame; generating optical flow information using at least one convolutional neural network, wherein the optical flow information is indicative of motion information of at least one object in the scene; determining at least one parameter for a motion correction process based on the optical flow information; performing the motion correction process on at least one of the first image frame and the plurality of second image frames to generate first intermediate data, wherein the motion correction process uses the identified at least one parameter; and generating a target image frame from the first intermediate data.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application US16 / 837,785, filed April 1, 2020, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] Embodiments of this disclosure relate to image capture systems. More specifically, embodiments of this disclosure relate to systems and methods for motion distortion using multi-exposure frames. Background Technology

[0004] Many mobile electronic devices, such as smartphones and tablets, include cameras that can be used to capture both still and video images. While convenient, cameras on mobile devices typically have several drawbacks. For example, cameras on mobile devices often perform poorly in low light. While increasing the exposure time can increase the amount of light collected by the image sensor, it also increases the risk of blurry images due to object and camera movement. One approach to address blur is to capture multiple image frames of the scene and then combine the “best” portions of the frames to produce a blended image. However, generating a blended image based on a set of image frames with different exposures is a challenging process, especially for dynamic scenes. Summary of the Invention

[0005] Embodiments of this disclosure provide methods and electronic devices for preventing blurring in captured images. Furthermore, embodiments of this disclosure provide methods and electronic devices for motion distortion using multi-exposure frames.

[0006] According to embodiments of this disclosure, a method is provided, comprising: using at least one image sensor of an electronic device to obtain a first image frame and a plurality of second image frames of a scene, each of the plurality of second image frames having an exposure time different from that of the first image frame; using at least one processor to generate optical flow information by using at least one convolutional neural network, wherein the optical flow information indicates motion information of at least one object in the scene; using at least one processor to determine at least one parameter for motion correction processing based on the optical flow information; using at least one processor to perform motion correction processing on at least one of the first image frame and the plurality of second image frames to generate first intermediate data, wherein the motion correction processing uses the identified at least one parameter; and using at least one processor to generate a target image frame based on the first intermediate data.

[0007] Further, according to an embodiment of the present disclosure, wherein performing the motion correction processing comprises: encoding, using the at least one processor, each of the first image frame and the second image frame by using a convolutional neural network to generate corresponding feature maps; and aligning, using the at least one processor, the encoded features of the feature map corresponding to the first image frame and the encoded features of the feature map corresponding to the second image frame.

[0008] Further, according to an embodiment of the present disclosure, wherein the optical flow information comprises at least one optical flow map, and wherein aligning the encoded features of the feature map corresponding to the first image frame and the encoded features of the feature map corresponding to the second image frame comprises: generating, using the at least one optical flow network, at least one optical flow map representing a pixel-wise difference between at least one pair of frames among the first image frame and the second image frame; and performing a warping operation on at least one of the feature maps using the at least one optical flow map, and wherein generating the target image frame comprises: decoding, using the at least one processor, the feature maps with the aligned encoded features by using the convolutional neural network to generate the target image frame of the scene.

[0009] Further, according to an embodiment of the present disclosure, the exposure time of each second image frame is shorter than the exposure time of the first image frame.

[0010] Further, according to an embodiment of the present disclosure, the exposure time of one of the second image frames is longer than the exposure time of the first image frame; and the exposure time of another of the second image frames is shorter than the exposure time of the first image frame.

[0011] Further, according to an embodiment of the present disclosure, the method further comprises: concatenating the feature maps with the aligned encoded features before decoding the feature maps with the aligned encoded features.

[0012] Further, according to an embodiment of the present disclosure, the first image frame is used as a reference frame and the second image frames are used as non-reference frames.

[0013] Further, according to an embodiment of the present disclosure, the convolutional neural network comprises a generative adversarial network.

[0014] Further, according to an embodiment of the present disclosure, generating the optical flow information comprises: generating the optical flow information by using an optical flow network, wherein determining the at least one parameter comprises: generating, using the optical flow information, a blur kernel indicating a motion direction of the first image frame; and refining, using the at least one processor, the blur kernel by using the convolutional neural network.

[0015] Further, according to an embodiment of the present disclosure, performing the motion correction processing comprises: performing an occlusion-aware deconvolution operation using the refined blur kernel and the occlusion mask; and generating the target image frame comprises: rendering a static background of the target image frame.

[0016] In addition, according to an embodiment of the present disclosure, an exposure time of each of the second image frames is shorter than an exposure time of the first image frame.

[0017] In addition, according to an embodiment of the present disclosure, the optical flow network determines optical flow between the plurality of second image frames to generate a blur kernel.

[0018] In addition, according to an embodiment of the present disclosure, an occlusion mask for the second image frames is estimated before generating the target image frame of the scene.

[0019] In addition, according to an embodiment of the present disclosure, there is provided an electronic device including at least one image sensor; and at least one processor configured to: obtain, from the at least one image sensor, a first image frame and a plurality of second image frames of a scene, each of the plurality of second image frames having a different exposure time than an exposure time of the first image frame; generate, using at least one convolutional neural network, optical flow information, wherein the optical flow information indicates motion information of at least one object in the scene; determine at least one parameter for a motion correction process based on the optical flow information; perform the motion correction process on at least one of the first image frame and the plurality of second image frames to generate first intermediate data, wherein the motion correction process uses the determined at least one parameter; and generate a target image frame from the first intermediate data.

[0020] In addition, according to an embodiment of the present disclosure, there is provided a non-transitory machine-readable medium containing instructions that, when executed, cause at least one processor of an electronic device to: obtain, using at least one image sensor of the electronic device, a first image frame and a plurality of second image frames of a scene, each of the plurality of second image frames having a different exposure time than an exposure time of the first image frame; generate, using at least one convolutional neural network, optical flow information, wherein the optical flow information indicates motion information of at least one object in the scene; determine at least one parameter for a motion correction process based on the optical flow information; perform the motion correction process on at least one of the first image frame and the plurality of second image frames to generate first intermediate data, wherein the motion correction process uses the determined at least one parameter; and generate a target image frame from the first intermediate data.

[0021] Other technical features can be readily ascertainable by one of ordinary skill in the art from the following drawings, descriptions, and claims. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and other aspects, features, and advantages of some embodiments of the present disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0023] Figure 1 An example network configuration including an electronic device according to an embodiment of the present disclosure is illustrated.

[0024] Figure 2 An example process for motion warping using multiple-exposure image frames is shown according to an embodiment of the present disclosure;

[0025] Figure 3 An example process for training a convolutional neural network is shown according to an embodiment of the present disclosure;

[0026] Figure 4 An example process for generating training data for a training process of a convolutional neural network is shown according to an embodiment of the present disclosure; Figure 3 An example process for generating training data for a training process of a convolutional neural network is shown according to an embodiment of the present disclosure;

[0027] Figure 5 An example process for motion warping using multiple-exposure image frames is shown according to an embodiment of the present disclosure;

[0028] Figure 6 Another example process for training a convolutional neural network is shown according to an embodiment of the present disclosure;

[0029] Figure 7 An example process for denoising using multiple image frames is shown according to an embodiment of the present disclosure;

[0030] Figure 8A And Figure 8B Examples of advantages that can be achieved using embodiments of the present disclosure are shown;

[0031] Figure 9 An example method for motion warping using multiple image frames is shown according to an embodiment of the present disclosure;

[0032] Figure 10 An example process for removing motion blur using multiple-exposure image frames is shown according to an embodiment of the present disclosure;

[0033] Figure 11A And Figure 11B Further examples of advantages that can be achieved using embodiments of the present disclosure are shown; and

[0034] Figure 12 An example method for removing motion blur using multiple-exposure image frames is shown according to an embodiment of the present disclosure.

[0035] Figure 13 FIG. 1 is a diagram showing an image processing method using an electronic device according to an embodiment of the present disclosure.

[0036] Figure 14 FIG. 2 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] According to an embodiment of the disclosure, a method is provided, including: obtaining, using at least one image sensor of an electronic device, a first image frame and a plurality of second image frames of a scene, each of the plurality of second image frames having an exposure time different from an exposure time of the first image frame; generating, using at least one processor, optical flow information by using at least one convolutional neural network, wherein the optical flow information indicates motion information of at least one object in the scene; determining, using the at least one processor, at least one parameter for a motion correction process based on the optical flow information; performing, using the at least one processor, the motion correction process on at least one of the first image frame and the plurality of second image frames to generate first intermediate data, wherein the motion correction process uses the identified at least one parameter; and generating, using the at least one processor, a target image frame based on the first intermediate data.

[0038] Before undertaking the detailed description below, it can be advantageous to set forth definitions of certain terms and phrases used throughout this disclosure. The terms "transmit," "receive," and "communicate," and variations thereof, encompass both direct and indirect communication. The terms "include" and "comprise," and variations thereof, mean "without limitation." The term "or" is inclusive, meaning and / or. The phrase "associated with," and variations thereof, means includes, is included in, interconnects with, contains, is contained within, connects to or with, couples to or with, is communicable with, cooperates with, interleafs with, is interposed between, is ligated to or with, has, has properties of, has relations with, or relations to, and / or the like.

[0039] Moreover, various functions described below can be implemented or supported by one or more computer programs, each of which is formed from computer readable program code and embodied in a computer readable medium. The terms "application" and "program" refer to one or more computer programs, software components, sets of instructions, routines, functions, objects, classes, instances, related data, or a portion thereof, suitable for implementation on a suitable computer readable program code. The phrase "computer readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer readable medium" includes any type of medium capable of being accessed by a computer, such as read only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory. A non-transitory computer readable medium includes a medium that is not capable of being written to, and a medium that is capable of being written to and then overwritten with new data.

[0040] As used herein, terms or phrases such as "have," "has," "can have," "including," or "may include," with respect to a feature (e.g., a number, a function, an operation, or a component such as a part) or a term or phrase similar thereto are used to indicate the presence of that feature, but do not preclude the presence or addition of other features. Also, as used herein, the phrase "A or B," "at least one of A and / or B," or "one or more of A and / or B" can include all possible combinations of A and B. For example, "A or B," "at least one of A and B," or "at least one of A or B" can indicate all of (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B. Also, as used herein, the terms "first" and "second" can modify various components regardless of importance and do not limit the components. The terms are used only to distinguish a component from another component with respect to description. For example, a first user device and a second user device can indicate different user devices from each other, regardless of importance or order of the devices. A first component can be denoted by a second component, and vice versa, without departing from the scope of the disclosure.

[0041] It should be understood that when an element (e.g., a first element) is referred to as being "coupled to" or "connected to" another element (e.g., a second element), it can be directly coupled or connected to the other element or can be indirectly coupled or connected to the other element via a third element. On the contrary, it should be understood that when an element (e.g., a first element) is referred to as being "directly coupled to" or "directly connected to" another element (e.g., a second element), there are no other elements (e.g., a third element) interposed therebetween.

[0042] As used herein, the phrase "configured (or set) to" can be used interchangeably with the phrases "adapted to," "has the capacity to," "designed to," "suits for," "made to," or "capable of," depending on the context. The phrase "configured to" does not mean "specifically designed in hardware to" on its own. Rather, the phrase "configured to" can mean that a device can perform an operation with or without having another device or component. For example, a general-purpose processor (e.g., CPU or application processor) that can execute one or more software programs stored in a memory device to perform an operation, or a special-purpose processor (e.g., embedded processor) that is designed to perform the operation, can be configured to perform the operation.

[0043] The terminology and phraseology used herein is provided for the purpose of describing some embodiments of the present disclosure and is not intended to limit the scope of other embodiments of the present disclosure. It will be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. All terms and phrases used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present embodiments pertain. It will be further understood that terms and phrases used herein have meanings commonly understood by those of ordinary skill in the art unless otherwise explicitly defined herein. In some cases, terms and phrases with apparently similar definitions in different contexts can be intended to be different. All publications, patent applications, patents, and other references mentioned herein are incorporated by reference for the purposes of describing and disclosing items into which the citations are directed to provide additional description of problems sought to be solved by the present embodiments.

[0044] Examples of the "electronic device" according to the embodiments of the present disclosure can include at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical appliance, a camera, or a wearable device (e.g., smart glasses, a head-mounted device (HMD), an electronic apparel, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, a smart mirror, or a smart watch). Other examples of the electronic device include a smart home appliance. Examples of the smart home appliance can include at least one of a television, a digital versatile disc (DVD) player, an audio player, a refrigerator, an air conditioner, a cleaner, an oven, a microwave oven, a washing machine, a dryer, an air cleaner, a set-top box, a home automation control panel, a security control panel, a TV box (e.g., SAMSUNG HOMESYNC, APPLE TV, or GOOGLE TV), a smart speaker or a speaker with integrated digital assistant (e.g., SAMSUNG GALAXY HOME, APPLE HOMEPOD, or AMAZON ECHO), a game console (e.g., XBOX, PLAYSTATION, or NINTENDO), an electronic dictionary, an electronic key, a camcorder, or an electronic photo frame. Still other examples of the electronic device include at least one of various medical devices (e.g., a multi-functional portable medical measuring device (e.g., a blood glucose measuring device, a heart rate measuring device, or a body temperature measuring device), a magnetic resonance angiography (MRA) device, a magnetic resonance imaging (MRI) device, a computed tomography (CT) device, an imaging device, or an ultrasonic device), a navigation device, a global positioning system (GPS) receiver, an event data recorder (EDR), a flight data recorder (FDR), a car infotainment device, marine electronic equipment (e.g., marine navigation equipment or a gyro compass), avionics, security equipment, a vehicle head unit, an industrial or home robot, an automatic teller machine (ATM), a point of sale (POS) device, or an Internet of Things (IoT) device (e.g., a light bulb, various sensors, a gas or water meter, a sprinkler, a fire alarm, a thermostat, a street light, a toaster, a fitness equipment, a hot water tank, a heater, or a boiler). Other examples of the electronic device include at least a part of furniture or a building / structure, an electronic board, an electronic signature receiving device, a projector, and various measuring devices (e.g., a device for measuring water, electricity, gas, or electromagnetic waves). It is noted that the electronic device according to various embodiments of the present disclosure can be one or a combination of the above-listed devices. According to some embodiments of the present disclosure, the electronic device can be a flexible electronic device. The electronic device disclosed herein is not limited to the above-listed devices and can include new electronic devices according to technological development.

[0045] In the following description, in accordance with various embodiments of the present disclosure, an electronic device is described with reference to the drawings. As used herein, the term "user" can represent a person or another device (e.g., an artificial intelligence electronic device) using the electronic device.

[0046] Definitions for other certain words and phrases are provided throughout this disclosure. Those of ordinary skill in the art will understand that in many, if not most instances, such definitions apply to not only the

[0047] The description in this application should not be interpreted as implying any particular element, step, or function is essential to the practice of the claims. The scope of the patent subject matter is defined only by the claims. Moreover, these claims may comprise a variety of steps implemented in any manner and in any order unless otherwise indicated by the language of the claims.

[0048] The subject matter discussed in the following description relates to Figures 1 to 12 and embodiments of the present disclosure. It is to be understood, however, that the present disclosure is not limited to these embodiments and that all changes and / or equivalents or alternatives falling within the scope of the present disclosure are intended to be embraced by the appended claims. Throughout the specification and the drawings, like or similar elements are referred to using like or similar designations.

[0049] As mentioned above, many mobile electronic devices (e.g., smartphones and tablets) include cameras that can be used to capture still and video images, but these cameras have a number of shortcomings. For example, these cameras generally have poor performance in low light conditions. While the amount of light collected by the image sensor can be increased by increasing the exposure time, this also increases the risk of producing blurred images due to subject and camera motion. One approach to solving blur is called multi-frame imaging, which captures multiple image frames of a scene and then combines the "best" portions of the image frames to produce a blended image. Multi-frame imaging plays an important role in tasks such as image deblurring, high dynamic range (HDR) imaging, noise reduction, and the like.

[0050] When a scene contains moving objects, processing motion in multiple image frames becomes very challenging. One approach to address this issue involves detecting each moving region and excluding that region during multi-frame fusion. This approach can enable regions near a moving object to have the same quality as when a single image frame is used. Another approach is to register one or more non-reference image frames based on optical flow and then fuse the registered image frames. However, such registration can be error-prone and can result in inaccurate optical flow estimates. Small deviations in optical flow can adversely affect continuous sharp edges in non-reference image frames.

[0051] Embodiments of the present disclosure provide various techniques for motion warping in which one or more extracted features of a non-reference image frame are aligned with features in a reference image frame according to optical flow between the image frames. As described in greater detail below, motion warping techniques according to embodiments of the present disclosure can be flexibly incorporated into generative networks to align extracted features prior to inputting them to a decoding layer. Embodiments of the present disclosure can be advantageously used in motion deblurring, HDR imaging, multi-frame denoising, and other imaging applications.

[0052] It is noted that while the techniques described below are generally described as being performed using a mobile electronic device, other electronic devices can also be used to perform or support these techniques. Thus, these techniques can be used in various types of electronic devices. Moreover, while the techniques described below are generally described as processing image frames when capturing still images of a scene, the same or similar approaches can be used to support the capture of video images.

[0053] Figure 1 An example network configuration 100 including an electronic device according to embodiments of the present disclosure is shown. Figure 1 The illustrated embodiment of the network configuration 100 is for illustration only. Other embodiments of the network configuration 100 can be used without departing from the scope of the present disclosure.

[0054] According to embodiments of the present disclosure, an electronic device 101 is included in the network configuration 100. The electronic device 101 can include at least one of a bus 110, a processor 120, a memory 130, an input / output (I / O) interface 150, a display 160, a communication interface 170, or a sensor 180. In some embodiments, the electronic device 101 can not include at least one of the components, or can further include at least one other component. The bus 110 includes a circuit for connecting the components 120 to 180 to each other to transmit communications (e.g., control messages and / or data) between the components.

[0055] The processor 120 includes one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor 120 can perform control on at least one other component of the electronic device 101 and / or perform an operation or data processing related to communication. In some embodiments, the processor 120 can be a graphic processing unit (GPU). For example, the processor 120 can receive image data captured by at least one camera during a capture event. The processor 120 can process the image data using a convolutional neural network, among others, as discussed in more detail below, to perform motion twist, etc.

[0056] The memory 130 can include at least one of a volatile memory and a non-volatile memory. For example, the memory 130 can store commands or data related to at least one other component of the electronic device 101. According to an embodiment of the disclosure, the memory 130 can store at least one of software or a program 140. The program 140 includes, for example, at least one of a kernel 141, middleware 143, an application programming interface (API) 145, or an application program (or "application") 147. At least a portion of the kernel 141, the middleware 143, or the API 145 can be denoted as an operating system (OS).

[0057] The kernel 141 can control or manage system resources (e.g., the bus 110, the processor 120, or the memory 130) used to execute operations or functions implemented in other programs (e.g., the middleware 143, the API 145, or the application program 147). The kernel 141 provides an interface that allows the middleware 143, the API 145, or the application 147 to access individual components of the electronic device 101 to control or manage system resources. The application 147 includes one or more applications for image capture and image processing as discussed below. These functions can be performed by a single application or multiple applications each performing one or more of these functions. For example, the middleware 143 can act as a relay to allow the API 145 or the application 147 to communicate data with the kernel 141. Multiple applications 147 can be provided. The middleware 143 can control a work request received from the application 147, for example, by assigning priority for use of system resources (e.g., the bus 110, the processor 120, or the memory 130) of the electronic device 101 to at least one of the multiple applications 147. The API 145 is an interface that allows the application 147 to control functions provided from the kernel 141 or the middleware 143. For example, the API 145 includes at least one interface or function (e.g., a command) for file control, window control, image processing, or text control.

[0058] The I / O interface 150 serves, for example, as an interface for transmitting a command or data input from a user or other external device to other components of the electronic device 101. The I / O interface 150 can also output a command or data received from other components of the electronic device 101 to the user or other external device.

[0059] The display 160 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot light emitting diode (QLED) display, a microelectromechanical systems (MEMS) display, or an electronic paper display. The display 160 can also be a depth perception display, such as a multi-focal display. The display 160 is capable of displaying, for example, various contents (e.g., text, images, videos, icons, or symbols) to a user. The display 160 can include a touch screen and can receive, for example, touch, gesture, proximity, or hovering input using an electronic pen or a user's body part.

[0060] The communication interface 170 is capable of establishing communication between the electronic device 101 and an external electronic device (e.g., the first electronic device 102, the second electronic device 104, or the server 106), for example. For example, the communication interface 170 can be connected with the network 162 or the network 164 through wireless or wired communication to communicate with the external electronic device. The communication interface 170 can be a wired or wireless transceiver or any other component for transmitting and receiving signals (e.g., images).

[0061] The electronic device 101 also includes one or more sensors 180 that can measure physical quantities or detect an activation state of the electronic device 101 and convert the measured or detected information into an electrical signal. For example, the one or more sensors 180 include one or more cameras or other image sensors for capturing images of a scene. The sensors 180 can also include one or more buttons for touch input, gesture sensors, gyroscopes or gyro sensors, barometric sensors, magnetic sensors or magnetometers, acceleration sensors or accelerometers, grip sensors, proximity sensors, color sensors (e.g., red, green, blue (RGB) sensors), biophysical sensors, temperature sensors, humidity sensors, illuminance sensors, ultraviolet (UV) sensors, electromyography (EMG) sensors, electroencephalogram (EEG) sensors, electrocardiogram (ECG) sensors, infrared (IR) sensors, ultrasonic sensors, iris sensors, or fingerprint sensors. The sensors 180 can also include an inertial measurement unit, which can include one or more accelerometers, gyroscopes, and other components. In addition, the sensors 180 can include a control circuit for controlling at least one of the sensors included herein. Any of these sensors 180 can be located in the electronic device 101. The one or more cameras or other image sensors can be used optionally in conjunction with at least one flash 190. The flash 190 represents a device configured to generate illumination for use in image capture by the electronic device 101, e.g., one or more LEDs.

[0062] The first external electronic device 102 or the second external electronic device 104 can be a wearable device or a wearable device (e.g., an HMD) of a mountable electronic device. When the electronic device 101 is mounted in the electronic device 102 (e.g., an HMD), the electronic device 101 can communicate with the electronic device 102 through the communication interface 170. The electronic device 101 can be directly connected with the electronic device 102 to communicate with the electronic device 102 without involving a separate network. The electronic device 101 can also be an augmented reality wearable device (e.g., glasses) including one or more cameras.

[0063] The wireless communication can use at least one of, for example, Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), a 5th generation wireless system (5G), millimeter wave or 60GHz wireless communication, wireless USB, Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), a Universal Mobile Telecommunication System (UMTS), Wireless Broadband (WiBro), or a Global System for Mobile Communications (GSM) as a cellular communication protocol. The wired connection can include at least one of, for example, a Universal Serial Bus (USB), a High Definition Multimedia Interface (HDMI), Recommended Standard-232 (RS-232), or a Plain Old Telephone Service (POTS). The network 162 or the network 164 includes at least one communication network, for example, a computer network (e.g., a Local Area Network (LAN) or a Wide Area Network (WAN)), the Internet, or a telephone network.

[0064] The first external electronic device 102 and the second external electronic device 104 and the server 106 can each be a device of a same or different type from the electronic device 101. According to an embodiment of the disclosure, the server 106 includes a group of one or more servers. Also, according to an embodiment of the disclosure, all or some of the operations performed on the electronic device 101 can be performed on one or more other electronic devices (e.g., the electronic devices 102 and 104 or the server 106). Furthermore, according to an embodiment of the disclosure, when the electronic device 101 is to perform some functions or services automatically, or upon request, the electronic device 101, instead of or in addition to performing, by itself, the functions or services, can request another device (e.g., the electronic devices 102 and 104 or the server 106) to perform at least some of the functions associated therewith. The other electronic device (e.g., the electronic devices 102 and 104 or the server 106) can perform the requested functions or additional functions and transmit the results of the performance to the electronic device 101. The electronic device 101 can provide the requested functions or services by processing the received results as is, or additionally. To that end, a cloud computing, distributed computing, or client-server computing technology can be used, for example. Although Figure 1 Although it is illustrated that the electronic device 101 includes the communication interface 170 for communication with the external electronic device 104 or the server 106 via the network 162 or the network 164, the electronic device 101 can operate independently without additional communication functions according to an embodiment of the disclosure.

[0065] The server 106 can optionally support the electronic device 101 by performing or supporting at least one of the operations (or functions) implemented in the electronic device 101. For example, the server 106 can include a processing module or a processor that can support the processor 120 implemented in the electronic device 101.

[0066] Although Figure 1An example of a network configuration 100 including an electronic device 101 is shown, but other configurations are possible. Figure 1 Various changes can be made. For example, network configuration 100 can include any number of various components arranged in any suitable manner. Typically, computing and communication systems have a wide variety of configurations, and Figure 1 This disclosure is not intended to limit the scope to any particular configuration. Furthermore, although… Figure 1 An operating environment is shown that can use the various features disclosed in this disclosure, but these features can be used in any other suitable system.

[0067] Figure 2 An example processing 200 for motion distortion using multi-exposure image frames according to an embodiment of the present disclosure is shown. For ease of illustration, Figure 2 The process 200 shown is described as involving the use of Figure 1 Electronic device 101. However, any other suitable electronic device and any suitable system can be used. Figure 2 The process shown is 200.

[0068] like Figure 2 As shown, electronic device 101 receives or acquires multiple image frames 202, 203, 204 of a scene captured at approximately the same time by at least one camera or other image sensor 180 of electronic device 101. Image frame 202 (also referred to as "long frame I") L The image frame 202 was captured using a long exposure time, which is the same as or longer than the automatically determined "normal" exposure time. Because of the long exposure time, one or more moving objects in image frame 202 may have a blurred feature. Image frames 203 to 204 (also referred to as "short frames I") were captured using a long exposure time, which is the same as or longer than the automatically determined "normal" exposure time. s1 "and "short frame I" s2 The images were captured using a short exposure time, which is at least shorter than both normal and long exposure times. Because of the short exposure time, image frames 203 to 204 record one or more moving objects with less or no motion blur. Note that, as used herein, the terms “short,” “normal,” and “long” are relative to each other and can represent any suitable exposure time, as long as “normal” is longer than “short” and “long” is as long as or longer than “normal.” In many cases, “normal” refers to an automatically determined exposure time that results in minimal underexposed and / or overexposed areas in the image and is often referred to as the EV-0 exposure time.

[0069] Because of the short exposure times, the two short frames 203-204 record any moving objects with less motion blur or no motion blur (relative to the image frame 202). In contrast, the long exposure time of the long frame 202 results in any moving objects likely to be blurred. To correct for the blur in the processing 200, the electronic device 101 uses the long frame 202 as a reference frame and the short frames 203-204 as non-reference frames. The electronic device 101 aligns any moving objects in the short frames 203-204 with any corresponding moving objects in the long frame 202. However, as described in more detail below, the alignment is performed on encoded features rather than on the raw image frames. That is, instead of moving the image frames directly, the electronic device 101 first encodes each of the short frames 203-204 into feature space and then aligns the features. This gives more robust motion compensation that does not distort the image features, resulting in less error.

[0070] The electronic device 101 inputs the three image frames 202, 203, 204 to respective encoder networks 210. Each of the encoder networks 210 is based on a convolutional neural network structure. Convolutional neural network structures generally represent a type of deep artificial neural network that is commonly used for analyzing images. Each encoder network 210 is composed of multiple convolutional layers 212. Each of the convolutional layers 212 represents a layer of convolutional neurons that operate to apply a convolution operation that simulates the response of individual neurons to visual stimuli. Each neuron generally applies some function (typically by differentially weighting different input values) to its input values to generate output values. Each encoder network 210 is shown here as including three convolutional layers 212, but each encoder network 210 can include a different number of convolutional layers. In some embodiments, each encoder network 210 includes or is part of a generative adversarial network (GAN). The output of each encoder network 210 is a feature map that includes multiple features (e.g., 64, 128, 256, or other number of features). In the processing 200, the encoder networks 210 are used to encode the image frames 202, 203, 204 into feature space. L (x, y), F s1 (x, y), and F s2 (x, y) represent the encoded feature maps for the image frame 202, the image frame 203, and the image frame 204, respectively.

[0071] The electronic device 101 also provides the non-reference short frames 203-204 to the optical flow network 206 for optical flow estimation. Optical flow estimation is a technique for pixel-wise prediction of motion in images over time. In the process 200, the optical flow network 206 receives the short frames 203-204 and determines the motion between the short frames 203-204. The optical flow network 206 includes any suitable function, process, or algorithm for determining motion between image frames. In some embodiments, the optical flow network 206 can include a neural network, e.g., a convolutional neural network. The output of the optical flow network 206 is an optical flow map 208, which is a matrix of values representing the pixel-wise x and y differences (referred to as Ax and Ay) between the short frames 203-204.

[0072] In the process 200, the encoded feature maps F s1 (x, y) and F s2 (x, y) are aligned to the feature map F L (x, y) generated from the long frame 202. To accomplish the alignment, the electronic device 101 performs warp operations 214-215. Each warp operation 214-215 receives the optical flow map 208 generated from the optical flow network 206 and applies a translation operation to the corresponding encoded feature map F s1 (x, y) or F s2 (x, y). The amount of translation is determined by the Ax and Ay values in the optical flow map 208. In some embodiments, each warp operation 214-215 is downscaled to match the downsizing from the image to the encoded feature map. The warp operations 214-215 result in translated feature maps F and

[0073] At this step in the process 200, the three feature maps representing the moving object, i.e., F and F L (x, y) are aligned. The electronic device 101 uses a concatenator operation 216 to combine the aligned features and inputs the combined features to the decoder network 220. The concatenated features encode the aligned sharp information from the short frames 203-204 to the motion-blurred region of the long frame 202 to enable the decoder network 220 to generate natural sharp edges.

[0074] Similar to the encoder network 210, the decoder network 220 is based on a convolutional neural network structure. In some embodiments, the decoder network 220 includes or is part of a GAN. As Figure 2 shown, the decoder network 220 is composed of a plurality of residual blocks 222, a plurality of transposed convolutional layers 224, and standard convolutional layers 226. While Figure 2Nine residual modules 222, two transposed convolutional layers 224, and one standard convolutional layer 226 are shown, but other embodiments of the decoder network 220 may include other numbers of residual modules, transposed convolutional layers, and standard convolutional layers.

[0075] The decoder network 220 is trained to combine the feature maps F of the long frame 202. L and feature maps F of short frames 203 to 204 s1 and F s2 And generate clear frames 230 (referred to as I) target The trained decoder network 220 optimizes the image quality of the sharp frame 230, making it as blurry as possible or non-blurry. In some embodiments, the decoder network 220 (particularly the residual module 222) performs a blending operation to generate the sharp frame 230. The sharp frame 230 can then be output as a final image for viewing, storage, or other image processing.

[0076] Although Figure 2 An example of motion distortion processing 200 using multi-exposure image frames is shown, but it is possible to perform... Figure 2 Various modifications can be made. For example, although process 200 is shown as using two short image frames, another embodiment of this disclosure may use more than two short image frames. Furthermore, the operation of process 200 may be performed by any suitable component of electronic device 101 or other device (including processor 120 of electronic device 101).

[0077] Figure 3 An example process 300 for training a convolutional neural network according to an embodiment of the present disclosure is shown. Using process 300, a convolutional neural network 305 can be trained to adapt to motion distortion using multi-exposure image frames. For ease of illustration, process 300 is described as involving the use of Figure 1 Electronic devices 101 and in Figure 2 Processing 300 describes some of the components. However, processing 300 can be used with any other suitable device and any other suitable convolutional neural network architecture.

[0078] During training process 300, electronic device 101 trains convolutional neural network 305, which includes encoder network 210, decoder network 220, or a combination thereof. Training process 300 is performed to help define what a sharp frame 230 should look like. To perform training process 300, electronic device 101 obtains training data comprising one or more long frames 202 and multiple short frames 203 to 204 associated with each long frame 202. If the training data only includes long and short frames, it may be difficult or impossible to define the ground truth because the long and short frames may not represent the true situation. Therefore, electronic device 101 also obtains a sharp frame 310 of the ground truth associated with each long frame 202. (This is discussed in conjunction with...) Figure 4 Describe the method for obtaining training data (including long frames 202, short frames 203 to 204, and clear real-world frames 310).

[0079] like Figure 3 As shown, the training process 300 uses a GAN training structure including a generator 315 and a discriminator 320. The generator 315 receives input information and generates the desired output, in this case represented by the recovered clear frame 230. The discriminator 320 determines whether the desired output is believable or correct. The electronic device 101 performs the functions of the generator 315 and the discriminator 320 to train the convolutional neural network 305 in an end-to-end manner. In some embodiments, the discriminator 320 may include multiple functions 325, including convolutional layers, normalization functions, and rectified linear unit (ReLU) functions.

[0080] Electronic device 101 trains convolutional neural network 305 in an iterative manner to generate restored sharp frames 230. Ideally, during multiple iterations of the entire training process 300, convolutional neural network 305 generally improves the generation of restored sharp frames 230, thereby producing real-world sharp frames 310.

[0081] For each iteration of the entire training process 300, the electronic device 101 can execute a loss calculation function, which calculates the total loss function used for the convolutional neural network 305. This loss helps guide the updating of the weights of the convolutional neural network 305. For example, the electronic device 101 can calculate the total loss function according to the following formula:

[0082] L=λL conteent +L GAN

[0083] Where L is the total loss function, and λ is the set of training parameters used to balance content loss and GAN loss, λL content It is a loss of content, and L GANThis is the GAN loss. Content loss is a measure of the difference between the real-world sharp frame 310 and the restored sharp frame 230. In some embodiments, content loss can be L1 or L2 loss. GAN loss can be any standard adversarial loss. For example, such as... Figure 3 As shown, the GAN loss can be the Wasserstein distance, which is the distance metric (or loss metric) used in Wasserstein GAN.

[0084] Based on the calculated loss, electronic device 101 adjusts convolutional neural network 305 by updating the weights used by convolutional neural network 305. For example, electronic device 101 may change the weights used in generator 315, discriminator 320, or other parameters of convolutional neural network 305. Once the updated weights are determined, electronic device 101 performs another iteration for training convolutional neural network 305. The overall goal of training process 300 is to reduce or minimize the value of the loss function.

[0085] Although Figure 3 An example of processing 300 for training a convolutional neural network is shown, but it is possible to modify... Figure 3 Various modifications can be made. For example, the convolutional neural network 305 can be trained in any other suitable way (with or without involving the use of generators and discriminators).

[0086] Figure 4 An embodiment of the present disclosure is shown for generating [something] for [something]. Figure 3 Example processing 400 of the training data in training processing 300. Specifically, processing 400 can be used to generate long frames, short frames, and real-world clear frames to be used as training data. For ease of illustration, processing 400 is described as involving the use of Figure 1 Electronic devices 101 and Figure 3 Processing 300. However, processing 400 can be trained using any other suitable device and any other suitable network.

[0087] like Figure 4 As shown, the image sensor (e.g., a high-speed camera) of electronic device 101 captures multiple video frames 402 in burst mode within a short time period (e.g., 100-150 milliseconds). The video frames 402 depict the motion of at least one object within the burst time. According to embodiments of this disclosure, the video frames 402 can be captured at 240 frames per second.

[0088] Electronic device 101 selects two frames from video frames 402 captured at different times as short frames 203 to 204. Figure 4In this case, the end frames of the captured video frames 402 are selected as the short frames 203-204. However, according to another embodiment of the disclosure, other frames of the captured video frames 402 can be selected as the short frames 203-204. The electronic device 101 also selects one of the captured video frames 402 at or near the midpoint of the burst as the real-world sharp frame 310.

[0089] To obtain the long frame 202, which includes the actual motion blur portion, the electronic device 101 blends together multiple image frames. To obtain a larger sample for blending that includes more representation of the object motion, the electronic device 101 generates a plurality of interpolated frames 404 (shown in Figure 4 as shorter lines) by differencing pairs of adjacent video frames 402. As Figure 4 shown, once the interpolated frames 404 are generated over a time span (e.g., 50-100 milliseconds) in the burst, the electronic device 101 averages the video frames 402 and the interpolated frames 404 corresponding to the video frames 402 over the time span to create a synthesized long frame 202 with the actual motion blur.

[0090] The resulting long frame 202, short frames 203-204, and real-world sharp frame 310 can be used as one set of training data for training the process 300. Additional training data can be generated by shifting the sample to the left or right by one or more of the captured video frames 402. More additional training data can be generated by repeating the process 400 with additional captured video frames 402. Figure 4

[0091] While Figure 4 one example of a process 400 for generating training data is shown, various changes can be made to Figure 4 the process 400. For example, the number of interpolated frames and captured video frames included in the long frame can be greater than or less than the number shown. Figure 4

[0092] Figure 5 An example process 500 for motion warping using multiple exposure image frames to produce an HDR frame according to an embodiment of the disclosure is shown. For ease of explanation, the process 500 is described as involving the electronic device 101 and a plurality of components that are the same as or similar to the components shown in Figure 1 . However, the process 500 can be used with any other suitable device and components. Figure 2

[0093] As Figure 5 ​​​As shown, electronic device 101 receives or acquires multiple image frames 501, 502, 503 of a scene captured at approximately the same time by at least one camera or other image sensor 180 of electronic device 101. Image frame 501 of the scene (also referred to as "long frame I") long The image frame 502 (also known as "Normal Frame I") was captured using a long exposure time. norma The image frame 503 (also known as "Short Frame I") was captured using a normal exposure time. short The image is captured using a short exposure time. According to embodiments of this disclosure, long frame 501 is advantageous for capturing dark or shadowy areas, normal frame 502 captures details of the subject in the scene, and short frame 503 captures bright areas. Electronic device 101 uses normal frame 502 as a reference frame and long frame 501 and short frame 503 as non-reference frames.

[0094] Electronic device 101 inputs each of the three image frames 501, 502, and 503 into the corresponding encoder networks 511, 512, and 513, which can be connected to... Figure 2 The encoder network 511, 512, and 513 are the same as or similar to encoder network 210. As an example, the components of encoder networks 511, 512, and 513 can be the same as those of encoder network 210, but the trained parameters of encoder networks 511, 512, and 513 can be different from those of encoder network 210. The output of each encoder network 511, 512, and 513 is for that exposure level (F... long F norma F short The feature map includes multiple identified features.

[0095] Electronic device 101 also provides a non-reference long frame 501 and a reference normal frame 502 to a first optical flow network 505 for optical flow estimation. Similarly, electronic device 101 provides a non-reference short frame 503 and a reference normal frame 502 to a second optical flow network 506 for optical flow estimation. Optical flow network 505 determines an optical flow map 508 from long frame 501 to normal frame 502, and optical flow network 506 determines an optical flow map 509 from short frame 503 to normal frame 502. Each of optical flow networks 505 and 506 can be coupled with... Figure 2 The optical flow network 206 is the same as or similar to it.

[0096] In order to generate feature maps F based on long frame 501 and short frame 503 long F short Compare with the feature map F of normal frame 502 respectively normal Alignment, the electronic device 101 performs twisting operations 514 to 515, which can be with Figure 2The warp operations 214 to 215 are the same or similar to the warp operations 114 to 115. The warp operations 514 to 515 result in translated features maps that are aligned with each other. The electronic device 101 combines the aligned features using a concatenator operation 516 and then inputs the combined features to a decoder network 520.

[0097] The decoder network 520 is trained to select features from different feature maps F 1ong , F normal , F short associated with different exposure levels and blend the features to generate an HDR image frame 530 (referred to as I HDR ). According to embodiments of the present disclosure, the decoder network 520 applies an HDR blending algorithm to the feature maps to generate the HDR image frame 530 with reduced or minimal blurring because the feature maps are well aligned. The decoder network 520 can use any suitable blending technique or algorithm to generate the HDR image frame 530 based on the multiple feature maps.

[0098] Although Figure 5 processing 500 is shown using multiple exposure image frames for motion warping to produce an HDR frame, various changes can be made. Figure 5 For example, in the processing 500, more than one long frame or short frame can be used as non-reference frames.

[0099] Figure 6 Another example processing 600 for training a convolutional neural network is shown according to embodiments of the present disclosure. Using the processing 600, a convolutional neural network 605 can be trained for motion warping using multiple exposure image frames to produce an HDR frame. For ease of illustration, the processing 600 is described as involving the electronic device 101 using Figure 1 and some of the components depicted in the processing 500 of Figure 5 . However, the processing 600 can be used with any other suitable device and any other suitable convolutional neural network structure.

[0100] During the training processing 600, the electronic device 101 trains the convolutional neural network 605, which includes the encoder networks 511, 512, 513, the decoder network 520, or a combination thereof. To perform the training processing 600, the electronic device 101 obtains training data, which includes one or more long frames 501, one or more normal frames 502, and one or more short frames 503. According to embodiments of the present disclosure, at least some of the training data can be generated using the processing 400 of Figure 4 or a similar processing.

[0101] As Figure 6As shown, the training process 600 uses a GAN training structure that includes a generator 615 and a discriminator 620. These can be the same as or similar to the corresponding components of the electronic device 101 and can be the same as or similar to the components shown in FIG. 6A. The challenge in HDR is to create one or more ground truth HDR frames 610 to be used to define the content loss. This is addressed by performing the training process 600 in two stages. Figure 3

[0102] Similar to the training process 500, in a first stage (“Stage I”), the electronic device 101 trains the convolutional neural network 605 (training the generator and the discriminator together) in an end-to-end manner using both the GAN loss (e.g., the Wasserstein distance) and the content loss. The frames 501, 502, 503 that are used as training samples in the first stage are selected to have only static scenes, as it is easier to create ground truth HDR frames 610 from static scenes. In a second stage (“Stage II”), the electronic device 101 fine-tunes the convolutional neural network 605 using only the GAN loss. In the second stage, the frames 501, 502, 503 that are selected as training samples include scenes that contain moving objects. Because the second stage of the training process 600 uses only the GAN loss (without the content loss), the ground truth HDR frames 610 do not need to be used for fine-tuning.

[0103] Although the training process 600 is described as being performed by the electronic device 101, the training process 600 can be performed by any other suitable device. Figure 6 One example of a process 700 for denoising using multiple image frames is shown in FIG. 7. The process 700 can be used for denoising when a scene contains moving objects. For ease of illustration, the process 700 is described as involving the electronic device 101 and multiple components that can be the same as or similar to the components shown in FIG. 6A. However, the process 700 can be used with any other suitable device and components. Figure 6

[0104] Figure 7 One example of a process 700 for denoising using multiple image frames is shown in FIG. 7. The process 700 can be used for denoising when a scene contains moving objects. For ease of illustration, the process 700 is described as involving the electronic device 101 and multiple components that can be the same as or similar to the components shown in FIG. 6A. However, the process 700 can be used with any other suitable device and components. Figure 1 Figure 2 As shown, the training process 600 uses a GAN training structure that includes a generator 615 and a discriminator 620. These can be the same as or similar to the corresponding components of the electronic device 101 and can be the same as or similar to the components shown in FIG. 6A. The challenge in HDR is to create one or more ground truth HDR frames 610 to be used to define the content loss. This is addressed by performing the training process 600 in two stages.

[0102]

[0105] As shown, the training process 600 uses a GAN training structure that includes a generator 615 and a discriminator 620. These can be the same as or similar to the corresponding components of the electronic device 101 and can be the same as or similar to the components shown in FIG. 6A. The challenge in HDR is to create one or more ground truth HDR frames 610 to be used to define the content loss. This is addressed by performing the training process 600 in two stages. Figure 7 ​​As shown, the electronic device 101 receives or obtains multiple image frames 701, 702, 703 of a scene captured at approximately the same time instant using at least one camera or other image sensor 180 of the electronic device 101. Each of the image frames 701, 702, 703 is “noisy,” meaning that there is noise in each of the image frames 701, 702, 703. The three image frames 701, 702, 703 can have the same exposure level. The electronic device 101 selects the image frame 702 as the reference frame from the image frames 701, 702, 703. The image frame 702 can be selected because it is the sharpest image among the image frames 701, 702, 703 (i.e., the image frame 702 is the frame captured at the middle time), or it can be selected randomly. In general, any desired selection method can be used here. The electronic device 101 uses the other image frames 701 and 703 as non-reference frames.

[0106] The electronic device 101 inputs each of the three image frames 701, 702, 703 to a respective encoder network 711, 712, 713, which can be the same as or similar to the encoder network 210 of Figure 2 . The output of each encoder network 711, 712, 713 is a feature map corresponding to the image frame (F1, F2, F3), where the feature map includes a plurality of identified features.

[0107] To align the feature maps F1, F3 generated from the image frames 701 and 703, respectively, with the feature map F2 from the reference image frame 702, the electronic device 101 performs warping operations 714-715, which can be the same as or similar to the warping operations 214-215 of Figure 2 . The warping operations 714-715 result in translated feature maps that are aligned with each other. The electronic device 101 combines the aligned features using a concatenator operation 716 and inputs the combined features to a decoder network 720. The decoder network 720 is trained to select features from the different feature maps F1, F2, F3 and mix the features to generate a noise-free image frame 730 (or at least a noise-reduced image). According to embodiments of the present disclosure, a training process similar to the training process 600 of Figure 6 may be performed because there can be no good ground truth images available.

[0108] While Figure 7 one example of a process 700 for denoising using multiple image frames is shown, various changes can be made to the process 700. For example, more than two noisy frames can be used as non-reference image frames in the process 700. Figure 7

[0109] Figure 8A and Figure 8B ​Examples of advantages that can be achieved using embodiments of the present disclosure are shown. More specifically, Figure 8A and Figure 8B A comparison between an image 801 of a scene captured using conventional image processing and an image 802 of the same scene captured using one of the embodiments described above is depicted. In Figure 8A image 801 is captured and processed using conventional image operations. From Figure 8A it can be seen that image 801 exhibits significant blurring near the subject's arm 803 and shirt logo 804. In contrast, Figure 8B image 802 in is captured and processed using deblurring operations as described above. The resulting image 802 provides superior results to image 801, particularly near the subject's arm 803 and shirt logo 804. In images 801-802, the subject's face is blurred for privacy.

[0110] While Figure 8A and Figure 8B show examples of advantages that can be achieved using embodiments of the present disclosure, various changes can be made thereto. For example, images of many scenes under different lighting conditions can be captured, but these figures do not limit the scope of the present disclosure. These figures are merely used to represent an example type of advantages shown that can be obtained using the techniques described above. Figure 8A and Figure 8B For example, images of many scenes under different lighting conditions can be captured, but these figures do not limit the scope of the present disclosure. These figures are merely used to represent an example type of advantages shown that can be obtained using the techniques described above.

[0111] Figure 9 An example method 900 for motion warping using multiple image frames according to embodiments of the present disclosure is shown. For ease of explanation, Figure 9 the method 900 shown is described as involving the electronic device 101 using Figure 1 However, the method 900 shown can be utilized with any other suitable electronic device and in any suitable system. Figure 9

[0112] As Figure 9 ​As shown, at step 902, a first image frame and a plurality of second image frames of a scene are obtained using at least one image sensor of an electronic device. Each of the second image frames has an exposure time different from an exposure time of the first image frame. This can include, for example, the processor 120 of the electronic device 101 receiving a capture request; and causing a camera (sensor 180) to capture a plurality of image frames, e.g., image frames 202, 203, 204 or image frames 501, 502, 503. According to an embodiment of the present disclosure, each of the image frames has an exposure time shorter than the exposure time of the first image frame. For example, the exposure times of the short frames 203-204 are shorter than the exposure time of the long frame 202. According to another embodiment of the present disclosure, one of the second image frames has an exposure time longer than the exposure time of the first image frame, and another of the second image frames has an exposure time shorter than the exposure time of the first image frame. For example, the exposure time of the long frame 501 is longer than the exposure time of the normal frame 502, and the exposure time of the short frame 503 is shorter than the exposure time of the normal frame 502.

[0113] At step 904, each of the first image frame and the second image frames is encoded to generate a corresponding feature map. In some embodiments, the encoding is performed using a convolutional neural network, which can include a generative adversarial network. This can include, for example, the electronic device 101 encoding each of the image frames 202, 203, 204 using the encoder network 210, or encoding each of the image frames 501, 502, 503 using the encoder networks 511, 512, 513.

[0114] At step 906, at least one optical flow network is used to generate at least one optical flow map representing a pixel-wise difference between at least one pair of frames among the first image frame and the second image frames. This can include, for example, the electronic device 101 using the optical flow network 206 to generate the optical flow map 208, or using the optical flow networks 505-506 to generate the optical flow maps 508-509. At step 908, a warping operation is performed on at least one of the feature maps using the at least one optical flow map. This can include, for example, the electronic device 101 performing the warping operations 214-215 using the optical flow map 208, or performing the warping operations 514-515 using the optical flow maps 508-509.

[0115] At step 910, the feature maps with aligned encoded features are concatenated. This can include, for example, the electronic device 101 performing the concatenator operation 216 or the concatenator operation 516. At step 912, the feature maps with aligned encoded features are decoded using a convolutional neural network to generate a target image frame of the scene. This can include, for example, the electronic device 101 decoding the aligned feature maps using the decoder network 220 or the decoder network 520 to generate the clear frame 230 or the HDR image frame 530.

[0116] Although Figure 9 One example of a process 900 for motion warping using multiple image frames is shown, but various changes can be made. Figure 9 For example, although shown as a series of steps, various steps in Figure 9 may overlap, occur in parallel, occur in a different order, or occur any number of times.

[0117] Blind motion deblurring is another type of deblurring operation that can be performed for images captured in dim lighting. Blind motion deblurring is typically most successful when the non-uniform blur kernel is accurately estimated. Some methods use three consecutive blurry frames and attempt to determine the kernel for each pixel by bilinearly interpolating the optical flow between two adjacent blurry frames. However, such methods often cannot accurately estimate the optical flow between two blurry frames. In addition, such methods can not successfully recover background regions that are occluded by a moving object. To address these and other issues, the following embodiments include two or more clear short-exposure frames captured before or after the target long-exposure frame to make kernel estimation much more accurate.

[0118] Figure 10 An example process 1000 for removing motion blur using multiple-exposure image frames according to an embodiment of the present disclosure is shown. For ease of illustration, Figure 10 The process 1000 shown in Figure 1 is described as involving the use of the electronic device 101 and multiple components that are the same as or similar to the components shown in Figure 2 However, the process 1000 can be used with any other suitable devices and components.

[0119] As shown in Figure 10 , the electronic device 101 receives or obtains multiple image frames 1002, 1003, 1004 of a scene captured at approximately the same time using at least one camera or other image sensor 180 of the electronic device 101. An image frame 1002 of the scene (also referred to as a "long frame I L ") is captured using a long exposure time. Image frames 1003-1004 of the scene (also referred to as "short frame I s1 " and "short frame I s2 ", respectively) are captured using short exposure times. As with some of the embodiments described above, the long frame 1002 includes blurry portions.

[0120] The electronic device 101 provides the image frames 1002, 1003, 1004 to an optical flow network 1006 for optical flow estimation. The electronic device 101 uses the optical flow network 1006 to compute the optical flow (motion) between two short frames 1003-1004, and to compute a blur kernel 1008 of the approximate motion direction of the long frame 1002. The blur kernel 1008 represents a deconvolution kernel, which is a set of spatially varying filter kernels that can be used to sharpen the image. Each blur kernel 1008 can be computed by interpolating the optical flow at each pixel. For example, assume that the optical flow at pixel (x, y) is (u, v). The optical flow network 1006 can interpolate the optical flow into a blur kernel k by:

[0121]

[0122] After generating the blur kernel 1008, the electronic device 101 provides the blur kernel 1008 to a kernel refinement network 1010 to refine the blur kernel 1008. The kernel refinement network 1010 is a convolutional neural network for refining the blur kernel 1008 to fit the accurate exposure time window of the long frame 1002. The kernel refinement network 1010 can include a plurality of convolutional layers (e.g., 3x3 convolutional layers) followed by a plurality of transposed convolutional layers (e.g., 3x3 transposed convolutional layers). The output of the kernel refinement network 1010 is a plurality of refined kernels 1012. In some embodiments, the kernel refinement network 1010 can be trained using a re-blur loss by forcing the true case sharp image when convolved with the estimated kernel to be identical to the blurred input. In some cases, this can be expressed in the following way:

[0123]

[0124] where I sharp is the true case sharp image, k is the estimated kernel, and I blur is the blurred input.

[0125] The electronic device 101 can also estimate an occlusion mask 1014 for each of the short frames 1003-1004. Each occlusion mask 1014 estimates background objects that are partially occluded by a moving object. The occlusion masks 1014 will be applied to the short frames 1003-1004 to recover the background that is missing in the long frame 1002 as discussed below. The electronic device 101 can use any suitable method for occlusion mask estimation.

[0126] The electronic device 101 also performs an occlusion-aware deconvolution operation 1016. The occlusion-aware deconvolution operation 1016 is performed to recover blurred objects by applying a refined kernel 1012 to long frame 1002, and to recover occluded background regions by applying an occlusion mask 1014 to short frames 1003 to 1004. For example, suppose image frames 1002, 1003, and 1004 (hereinafter referred to as I) L I s1 and I s2 With occlusion masks 1014 (hereinafter referred to as M1 and M2) as input, the motion blur problem can be modeled as follows:

[0127] I L =I sharp *K+M1⊙I s1 +M2⊙I s2

[0128] Because the blur kernel k and the occlusion masks M1 and M2 are fixed, only the sharp image (hereinafter referred to as I) is visible. sharp It needs to be restored, which can be represented as follows:

[0129] I' L =I L -M1⊙I s1 -M2⊙I s2

[0130] I' L =I sharp *K

[0131] Used to estimate I sharp The objective function can be expressed as follows, where the primal-dual update can be used to optimally compute...

[0132]

[0133] Furthermore, the electronic device 101 performs static background rendering 1018 to render a static background for a clear image. For example, based on the kernel of each pixel, the electronic device 101 can estimate the static background mask M, and in long frame 1002 (I L ) and the estimated Alpha blending is applied between the input and the recovered, sharp image frame 1020 to transfer the high-quality background from the input.

[0134] Although Figure 10 An example of a process 1000 for removing motion blur using multi-exposure image frames is shown, but it is possible to modify... Figure 10 Various changes can be made. For example, in processing 1000, more than two short frames can be used as non-reference frames.

[0135] Figure 11A and Figure 11B Examples of advantages that can be achieved using embodiments of this disclosure are shown. More specifically, Figure 11A and Figure 11B Image 1101 depicts a scene captured using conventional image processing and using... Figure 10 A comparison between images 1102 of the same scene captured in the disclosed embodiments. Figure 11A In this process, traditional image manipulation techniques are used to capture and process image 1101. From Figure 11A As can be seen, image 1101 exhibits significant blurring near the subject's hands and legs. Conversely, Figure 11B Image 1102 was captured and processed using the deblurring operation described above. The resulting image 1102 provides a better result than image 1101, especially near the subject's hands and legs. In images 1101 and 1102, the subject's face is blurred for privacy.

[0136] Although Figure 11A and Figure 11B Examples of advantages that can be achieved using embodiments of the present disclosure are shown, but various changes can be made to these figures. For example, images of many scenes under different lighting conditions can be captured, but these figures do not limit the scope of the present disclosure. These figures are only used to illustrate examples of types of advantages that can be obtained using the techniques described above.

[0137] Figure 12 An example method 1200 for removing motion blur using multi-exposure image frames according to an embodiment of the present disclosure is shown. For ease of illustration, Figure 12 The method 1200 shown is described as involving the use of Figure 1 Electronic device 101. However, any other suitable electronic device and any suitable system can be used. Figure 12 Method 1200 is shown.

[0138] like Figure 12 As shown, at step 1202, a first image frame and a plurality of second image frames of the scene are acquired using at least one image sensor of the electronic device. Each of the second image frames has an exposure time different from that of the first image frame. This may include, for example, the processor 120 of the electronic device 101 receiving a capture request; and causing a camera (sensor 180) to capture a plurality of image frames, for example, image frames 1002, 1003, and 1004. According to embodiments of the present disclosure, the exposure time of each second image frame is shorter than the exposure time of the first image frame. For example, the exposure time of short frames 1003 to 1004 is shorter than the exposure time of long frame 1002.

[0139] At step 1204, an optical flow network is used to generate a blur kernel indicating the motion direction of the first image frame. This may include, for example, the electronic device 101 using an optical flow network 1006 to generate a blur kernel 1008. At step 1206, a convolutional neural network is used to refine the blur kernel. This may include, for example, the electronic device 101 using a kernel refinement network 1010 to generate a refined kernel 1012 based on the blur kernel 1008. At step 1208, an occlusion mask is estimated for the second image frame. This may include, for example, the electronic device 101 estimating an occlusion mask 1014 for short frames 1003 to 1004.

[0140] At step 1210, a target image frame of the scene is generated using a refined blur kernel and an occlusion mask for the second image frame. This may include, for example, the electronic device 101 generating a sharp image frame 1020 using a refined kernel 1012 and an occlusion mask 1014. In some embodiments, the target image frame is generated by the following steps: (i) performing an occlusion-aware deconvolution operation using a refined blur kernel and an occlusion mask; and (ii) rendering the static background of the target image frame, for example, as described for the occlusion-aware deconvolution operation 1016 and the background rendering operation 1018.

[0141] Although Figure 12 An example of a method 1200 for removing motion blur using multiple exposure image frames is shown, but more can be applied to it. Figure 12 Make various changes. For example, although shown as a series of steps, Figure 12 The steps in the process can overlap, occur in parallel, occur in different orders, or occur any number of times.

[0142] Figure 13 This is a diagram illustrating an image processing method using an electronic device according to an embodiment of the present disclosure.

[0143] The image processing method according to an exemplary embodiment of this disclosure can be... Figure 1 The electronic device 101 shown or Figure 14 The illustrated electronic device 1400 performs this function. In this disclosure, for ease of description, a method described is used... Figure 14 The illustrated electronic device 1400 performs an embodiment of the image processing method according to an embodiment of the present disclosure, but this is merely for ease of description. The image processing method according to the exemplary embodiments can be performed by various types of electronic devices.

[0144] In step 1302, the electronic device uses an image sensor to generate a first image frame and a plurality of second image frames. The first image frame and the plurality of second image frames correspond to a scene. Each of the second image frames has an exposure time different from that of the first image frame. According to one embodiment, each of the second image frames has a shorter exposure time than the first image frame. According to another embodiment, one of the second image frames has a shorter exposure time than the first image frame, while another of the second image frames has a longer exposure time than the first image frame. According to one embodiment, the plurality of second image frames includes two image frames.

[0145] According to one embodiment, in response to a shutter release signal, the electronic device acquires a first image frame and a plurality of second image frames. For example, the electronic device can capture the frames in a continuous shooting manner, in the order of the second image frames, the first image frames, and the second image frames.

[0146] Next, in step 1304, the electronic device generates optical flow information using at least one convolutional neural network. The optical flow information represents the motion information of at least one subject in the scene. According to one embodiment, the at least one convolutional neural network corresponds to... Figure 2 The optical flow network 206, and the optical flow information can correspond to Figure 2 The motion flow diagram 208. According to another embodiment, at least one neural network corresponds to... Figure 5 Optical flow networks 505 and 506, and the optical flow information corresponds to Figure 5 Optical flow diagrams 508 and 509. According to another embodiment, at least one neural network corresponds to… Figure 10 The optical flow network 1006, and the optical flow information corresponds to Figure 10 Optical flow or motion between two short frames, 1003 and 1004.

[0147] Next, in step 1306, the electronic device determines at least one parameter for motion correction processing based on optical flow information. According to one embodiment, the motion correction processing may correspond to... Figure 2 The motion distortion operations 214 and 215 are performed, and the parameters used for motion correction processing can correspond to the translation amount determined by the Δx and Δy values. According to another embodiment, the motion correction processing can correspond to... Figure 5 The motion distortion operations 514 and 515 are performed, and the parameters used for motion correction processing can correspond to the translation amount determined by the Δx and Δy values. According to another embodiment, the motion correction processing can correspond to... Figure 10 The occlusion-aware deconvolution 1016 is used, and the parameters used for motion correction processing can correspond to a refined kernel 1012 and an occlusion mask.

[0148] Next, in step 1308, the electronic device generates first intermediate data by performing a motion correction process on the first image frame or the second image frame.

[0149] According to embodiments of the present disclosure, the first intermediate data can be a feature map or an image on which the motion correction process is performed. In Figure 2 the embodiments described in Figure 5 , the first intermediate data can be concatenated features input to the decoder network 520. In Figure 10 the embodiments described in , the first intermediate data can be data output by the occlusion-aware deconvolution 1016 and input to the static background rendering 1018. According to embodiments of the present disclosure, the electronic device generates a feature map by using at least one convolutional neural network, and performs a motion correction process on the feature map. According to another embodiment of the present disclosure, the first intermediate data can be an image estimated using an estimation function. By performing a motion correction process on the feature map or the image frame, the electronic device generates motion-corrected first intermediate data before the decoding process.

[0150] In the motion correction process, the electronic device uses the parameters for the motion correction process determined in step 1306. According to one embodiment, the first intermediate data is a translated feature map F s1 (x+1 / 2Δx, y+1 / 2Δy) and F s2 (x-1 / 2Δx, y-1 / 2Δy). According to another embodiment, the first intermediate data can be a translated feature map described in the embodiments disclosed in Figure 5 . According to another embodiment, the first intermediate data can be an estimated clear image I sharp described in the embodiments disclosed in Figure 10 .

[0151] Next, in step 1310, the electronic device generates a target image frame from the first intermediate data. According to one embodiment, the process of generating the target image frame can correspond to the process of the decoder network 220 of Figure 2 . According to another embodiment, the process of generating the target image frame can correspond to the process of the decoder network 520 of Figure 5 . According to another embodiment, the process of generating the target image frame can correspond to the static background rendering process 1018 of Figure 10 .

[0152] Figure 14 is a block diagram of an electronic device according to embodiments of the present disclosure.

[0153] According to an embodiment of the disclosure, the electronic device 1400 includes an image sensor 1410 and a processor 1420.

[0154] The image sensor 1410 photoelectrically converts incident light to generate image frame data. The image sensor 1410 generates a first image frame and a plurality of second image frames. The image sensor 1410 can capture the first image frame and the plurality of second image frames with exposure times set based on a control signal CON received from the processor 1420. According to one embodiment, the image sensor 1410 can correspond to the sensor 180 of FIG. 1. Figure 1

[0155] The processor 1420 controls overall operations of the electronic device 1400. The processor 1420 can include one or more processors. The processor 1420 obtains a first image frame and a plurality of second image frames of a scene. Each of the plurality of second image frames from at least one image sensor has an exposure time different from an exposure time of the first image frame. The processor 1420 generates optical flow information using at least one convolutional neural network. The optical flow information indicates motion information of at least one object in the scene. The processor 1420 determines at least one parameter for a motion correction process based on the optical flow information. Also, the processor 1420 performs the motion correction process on at least one of the first image frame and the plurality of second image frames to generate first intermediate data. The motion correction process uses the determined at least one parameter. Also, the processor 1420 generates a target image frame based on the first intermediate data. According to one embodiment, the processor 1420 can correspond to the processor 120 of FIG. 1. Figure 1

[0156] It should be noted that while various operations are described above as being performed using one or more devices, these operations can be implemented in any suitable manner. For example, each function in the electronic device 101 or the server 106 can be implemented or supported using one or more software applications or other software instructions executed by at least one processor 120 of the electronic device 101 or the server 106. In other embodiments, at least some functions in the electronic device 101 or the server 106 can be implemented or supported using dedicated hardware components. In general, the operations of each device can be performed using any suitable hardware or any suitable combination of hardware and software / firmware instructions.

[0157] While the disclosure has been described with reference to various example embodiments, it will be understood that various changes and modifications can be suggested by those skilled in the art. It is intended that the disclosure encompass such changes and modifications as fall within the scope of the appended claims.​​

Claims

1. A method for processing images, comprising: obtaining, using at least one image sensor of an electronic device, a first image frame and a plurality of second image frames of a scene, each of the plurality of second image frames having a different exposure time than an exposure time of the first image frame; generating, using at least one processor, corresponding feature maps by encoding, using a convolutional neural network, each of the first image frame and the second image frames; generating, using the at least one processor, at least one optical flow map representing a pixel-by-pixel difference between pairs of at least one of the first image frame and the second image frames, by using at least one optical flow network; and aligning, using the at least one processor, encoded features of the feature maps corresponding to the first image frame and encoded features of the feature maps corresponding to the second image frames, by performing a warping operation on the encoded features of the feature maps corresponding to the second image frames using the at least one optical flow map.

2. The method of claim 1, further comprising: generating, using the at least one processor, a target image frame of the scene by decoding, using the convolutional neural network, the feature maps having the aligned encoded features.

3. The method of claim 1, wherein, the exposure time of each second image frame is shorter than the exposure time of the first image frame.

4. The method of claim 1, wherein: the exposure time of one of the second image frames is longer than the exposure time of the first image frame; and the exposure time of another of the second image frames is shorter than the exposure time of the first image frame.

5. The method of claim 1, further comprising: concatenating the feature maps having the aligned encoded features prior to decoding the feature maps having the aligned encoded features.

6. The method of claim 1, wherein, the first image frame is used as a reference frame and the second image frames are used as non-reference frames.

7. The method of claim 1, wherein, the convolutional neural network comprises a generative adversarial network.

8. An electronic device, comprising: at least one image sensor; and at least one processor configured to perform the method for processing images of claim 1.

9. A non-transitory machine-readable medium containing instructions that, when executed, cause at least one processor of an electronic device to perform the method for processing images of claim 1. ​

Images