Electronic device and method for image alignment in multi-frame fusion of tetra or other image data

Abstract

Obtain a non-Bayer color filter array (CFA) input image, where the non-Bayer CFA input image includes a reference image and a non-reference image, each having a non-Bayer CFA pattern. Generate a reference luminance image and a non-reference luminance image corresponding to the reference Bayer-like pattern based on the reference image and the non-reference Bayer-like pattern based on the non-reference image, respectively. The resolution of each luminance image is approximately half the resolution of each input image. Perform a smoothing operation on the luminance images to remove artifacts and generate a filtered reference luminance image and a filtered non-reference luminance image. Identify motion vectors based on the filtered luminance images and enlarge the motion vectors and the filtered luminance images to generate enlarged motion vectors and enlarged luminance images. Perform high-resolution thinning on the enlarged motion vectors based on the enlarged luminance images to generate the final motion vectors, and align the input images to each other based on the final motion vectors.

Description

Technical Field

[0001] This disclosure generally relates to image processing. More specifically, this disclosure relates to an electronic apparatus and method for image alignment in multi-frame fusion of Tetra or other image data. Background Technology

[0002] Image registration, or image alignment, refers to aligning multiple images / frames of a color filter array (CFA) camera sensor data without introducing significant image distortion, even in the presence of camera motion or small object movement. Image registration is often a necessary or desired component of image processing workflows, such as in multi-frame blending algorithms to support high dynamic range (HDR) imaging or motion blur reduction. These and other algorithms can be used to fuse multiple images, such as those captured using different exposure / ISO sensitivity settings. Summary of the Invention

[0003] Technical solution This disclosure relates to image alignment for multi-frame fusion of Tetra or other image data.

[0004] In an embodiment, a method may include: obtaining a non-Bayer color filter array (CFA) input image, wherein the non-Bayer CFA input image includes a reference image and a non-reference image, each having a non-Bayer CFA pattern. In an embodiment, the method may include: generating a reference luminance image corresponding to a reference Bayer-like pattern based on the reference image, and generating a non-reference luminance image corresponding to a non-reference Bayer-like pattern based on the non-reference image. The resolution of each of the reference luminance image and the non-reference luminance image is approximately half the resolution of each of the non-Bayer CFA input images. In an embodiment, the method may include: performing a smoothing operation on the reference luminance image and the non-reference luminance image to remove artifacts caused by the non-Bayer CFA pattern and generating a filtered reference luminance image and a filtered non-reference luminance image. In an embodiment, the method may include: identifying motion vectors based on the filtered luminance image. In an embodiment, the method may include: magnifying the motion vectors and the filtered luminance image to generate magnified motion vectors, a magnified reference luminance image, and a magnified non-reference luminance image. In an embodiment, the method may include: performing high-resolution thinning of the magnified motion vector based on the magnified brightness image to generate a final motion vector, and aligning the non-Bayer CFA input images to each other based on the final motion vector.

[0005] In an embodiment, an electronic device may include a memory storing instructions. In an embodiment, an electronic device may include at least one processing unit configured to: obtain a non-Bayer CFA input image, wherein the non-Bayer CFA input image includes a reference image and a non-reference image, each having a non-Bayer CFA pattern. In an embodiment, the at least one processing unit may be configured to: generate a reference brightness image corresponding to a reference Bayer-like pattern based on the reference image, and generate a non-reference brightness image corresponding to a non-reference Bayer-like pattern based on the non-reference image. The resolution of each of the reference brightness image and the non-reference brightness image is approximately half the resolution of each of the non-Bayer CFA input images. In an embodiment, the at least one processing unit may be configured to: perform a smoothing operation on the reference brightness image and the non-reference brightness image to remove artifacts caused by the non-Bayer CFA pattern and generate a filtered reference brightness image and a filtered non-reference brightness image. In an embodiment, the at least one processing unit may be configured to: identify a motion vector based on the filtered brightness image. In an embodiment, the at least one processing unit may be configured to: magnify the motion vector and the filtered brightness image to generate a magnified motion vector, a magnified reference brightness image, and a magnified non-reference brightness image. In an embodiment, at least one processing device may be configured to: perform high-resolution thinning of the magnified motion vector based on the magnified brightness image to generate a final motion vector, and align the non-Bayer CFA input images to each other based on the final motion vector.

[0006] In an embodiment, a machine-readable medium includes instructions that, when executed, cause at least one processor of an electronic device to perform the following operations: obtain a non-Bayer CFA input image, wherein the non-Bayer CFA input image includes a reference image and a non-reference image, each having a non-Bayer CFA pattern. The instructions, when executed, also cause at least one processor to perform the following operations: generate a reference brightness image corresponding to a reference Bayer-like pattern based on the reference image, and generate a non-reference brightness image corresponding to a non-reference Bayer-like pattern based on the non-reference image. The resolution of each of the reference brightness image and the non-reference brightness image is approximately half the resolution of each of the non-Bayer CFA input images. The instructions, when executed, also cause at least one processor to perform the following operations: perform a smoothing operation on the reference brightness image and the non-reference brightness image to remove artifacts caused by the non-Bayer CFA pattern and generate a filtered reference brightness image and a filtered non-reference brightness image. The instructions, when executed, also cause at least one processor to perform the following operations: identify motion vectors based on the filtered brightness image. When executed, the instructions also cause at least one processor to perform the following operations: upscale the motion vector and the filtered luminance image to generate an upscaled motion vector, an upscaled reference luminance image, and an upscaled non-reference luminance image. Furthermore, when executed, the instructions also cause at least one processor to perform the following operations: perform high-resolution thinning of the upscaled motion vector based on the upscaled luminance image to generate a final motion vector, and align the non-Bayer CFA input images to each other based on the final motion vector.

[0007] Any single feature or any combination of the following features may be used with the first, second, or third embodiment. The smoothing operation may include a bicubic filtering operation. Motion vectors can be identified by: comparing a filtered reference brightness image and a filtered non-reference brightness image to generate an initial motion vector with coarse-to-fine alignment; regularizing the initial motion vector using a median filter to generate a regularized motion vector; and performing local alignment of the regularized motion vector based on structure-guided mesh deformation (SGMW) to generate the motion vector. High-resolution thinning can be performed by: deforming a magnified non-reference brightness image based on the magnified motion vector to generate a deformed magnified non-reference brightness image; performing a block search for each pixel of the deformed magnified non-reference brightness image based on a comparison of the deformed magnified non-reference brightness image and the magnified reference brightness image; and thinning the magnified motion vector based on the block search. Each of the non-Bayer CFA input images can be re-stitched to generate a reference Bayer-like pattern and a non-reference Bayer-like pattern, and the resolution of each of the reference Bayer-like pattern and the non-reference Bayer-like pattern can be equal to the resolution of each of the reference non-Bayer CFA input images and the non-reference non-Bayer CFA input images. The non-Bayer CFA pattern may include a Tetra CFA pattern. Each of the reference non-Bayer CFA input image and the non-reference non-Bayer CFA input image can be re-stitched by swapping pixels within the central region of the Tetra CFA pattern for each of the reference non-Bayer CFA input image and the non-reference non-Bayer CFA input image to form multiple Bayer CFA patterns. A reference brightness image and a non-reference brightness image can be generated by identifying brightness based on pixels read from the central region of the non-Bayer CFA pattern within the Bayer-like CFA pattern.

[0008] Other technical features will be obvious to those skilled in the art based on the following figures, description and claims.

[0009] Before proceeding with the detailed embodiments below, it may be advantageous to define the specific words and phrases used throughout this patent document. The terms “transmit,” “receive,” and “communicate,” and their derivatives, include both direct and indirect communication. The terms “comprising” and “including,” and their derivatives, indicate, but are not limited to, including. The term “or” is inclusive, indicating and / or. The phrase “associated with,” and its derivatives, indicate including, being included within, interconnected with, containing, contained within, connected to or connected with, combined with or combined with, communicable with, cooperating with, interleaved, juxtaposed, proximate, bound to or bound with, having, possessing the nature of, having a relationship to or related to, etc.

[0010] Furthermore, the various functions described below may be implemented or supported by one or more computer programs, each of which is formed by computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, associated data, or portions thereof suitable for implementation in suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer-readable medium" includes any type of medium accessible by a computer, such as read-only memory (ROM), random access memory (RAM), hard disk drive, optical disc (CD), digital video disc (DVD), or any other type of storage. "Non-transitory" computer-readable medium excludes wired, wireless, optical, or other communication links that transmit transient electrical or other signals. Non-transitory computer-readable medium includes media that permanently store data and media that store data and can be rewritten later, such as rewritable optical discs or erasable memory devices.

[0011] As used herein, terms and phrases such as “having,” “may have,” “comprising,” or “may include” indicate the presence of a feature (such as a number, function, operation, or component, like a part) and do not exclude the presence of other features. Furthermore, as used herein, the phrases “A or B,” “at least one of A and / or B,” or “one or more of A and / or B” can include all possible combinations of A and B. For example, “A or B,” “at least one of A and B,” and “at least one of A or B” can indicate all of the following: (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B. Furthermore, as used herein, the terms “first” and “second” can modify various components regardless of importance and do not limit the components. These terms are used only to distinguish one component from another. For example, a first user device and a second user device can indicate user devices that are different from each other, regardless of the order or importance of the devices. A first component can be referred to as a second component without departing from the scope of this disclosure, and vice versa.

[0012] It will be understood that when an element (such as a first element) is referred to as being (operably or communicatively) "combined" with / "attached to" another element (such as a second element) or "connected" with / "connected to" another element (such as a second element), it may be directly combined or connected to / attached to or connected to that other element, or combined or connected to / attached to / connected to that other element via a third element. Conversely, it will be understood that when an element (such as a first element) is referred to as being "directly combined" with / "directly attached to" another element (such as a second element) or "directly connected" with / "directly connected to" another element (such as a second element), no other element (such as a third element) is between that element and that other element.

[0013] As used herein, the phrase “configured (or set) to” may be used interchangeably with the phrases “suitable for,” “capable of,” “designed to,” “adapted to,” “manufactured as,” or “capable”, depending on the context. The phrase “configured (or set) to” does not inherently imply “specifically designed to” in hardware. Rather, the phrase “configured to” may indicate that a device can perform operations in conjunction with another device or component. For example, the phrase “processor configured (or set) to perform A, B, and C” may refer to a general-purpose processor (such as a CPU or application processor) that can perform operations by running one or more software programs stored in a memory device, or a dedicated processor (such as an embedded processor) for performing operations.

[0014] The terms and phrases used herein are provided only to describe some embodiments of this disclosure and are not intended to limit the scope of other embodiments of this disclosure. It should be understood that, unless the context clearly indicates otherwise, the singular form includes plural references. All terms and phrases used herein (including technical and scientific terms and phrases) have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this disclosure pertain. It will also be understood that terms and phrases (such as those defined in common dictionaries) should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly defined herein. In some cases, the terms and phrases defined herein may be interpreted as excluding embodiments of this disclosure.

[0015] Examples of "electronic devices" according to embodiments of this disclosure may include at least one of the following: smartphones, tablet PCs, mobile phones, video phones, e-book readers, desktop PCs, laptop computers, netbooks, workstations, personal digital assistants (PDAs), portable multimedia players (PMPs), MP3 players, mobile medical devices, cameras, or wearable devices (such as smart glasses, head-mounted displays (HMDs), electronic clothing, electronic bracelets, electronic necklaces, electronic accessories, electronic tattoos, smart mirrors, or smartwatches). Other examples of electronic devices include smart home appliances. Examples of smart home appliances may include at least one of the following: television, digital video disc (DVD) player, audio player, refrigerator, air conditioner, vacuum cleaner, oven, microwave oven, washing machine, dryer, air purifier, set-top box, home automation control panel, security control panel, TV box (such as SAMSUNG HOMESYNC, APPLE TV, or GOOGLE TV), smart speaker or speaker with integrated digital assistant (such as SAMSUNG GALAXY HOME, APPLE HOMEPOD, or AMAZON ECHO), game console (such as XBOX, PLAYSTATION, or NINTENDO), electronic dictionary, electronic key, camera, or electronic photo frame. Other examples of electronic devices include at least one of the following: various medical devices (such as various portable medical measurement devices (e.g., blood glucose measuring devices, heart rate measuring devices, or body temperature measuring devices), magnetic resonance angiography (MRA) devices, magnetic resonance imaging (MRI) devices, computed tomography (CT) devices, imaging devices, or ultrasound devices), navigation devices, global positioning system (GPS) receivers, event data loggers (EDR), flight data loggers (FDR), automotive infotainment devices, navigation electronics (such as navigation devices or gyrocompasses), avionics devices, safety devices, vehicle head units, industrial or household robots, automated teller machines (ATMs), point-of-sale (POS) devices, or Internet of Things (IoT) devices (such as light bulbs, various sensors, electricity or gas meters, sprinklers, fire alarms, thermostats, streetlights, toasters, fitness equipment, hot water tanks, heaters, or boilers). Other examples of electronic devices include at least a portion of a piece of furniture or building / structure, electronic boards, electronic signature receivers, projectors, or various measuring devices (such as devices for measuring water, electricity, gas, or electromagnetic waves). It should be noted that, according to various embodiments of this disclosure, the electronic device may be one or a combination of the devices listed above. According to some embodiments of this disclosure, the electronic device may be a flexible electronic device. The electronic devices disclosed herein are not limited to the devices listed above and may include new electronic devices as the technology develops.

[0016] In the following description, electronic devices are described with reference to the accompanying drawings according to various embodiments of the present disclosure. As used herein, the term "user" may refer to a person using the electronic device or another device (such as an artificial intelligence electronic device).

[0017] Definitions for other specific words and phrases are provided throughout this patent document. It will be understood by those skilled in the art that, in many cases (and even most cases), these definitions apply to the prior and future use of the words and phrases defined in this way.

[0018] The descriptions in this application should not be construed as implying that any particular element, step, or function is an essential element that must be included within the scope of the claims. The scope of the patent subject matter is defined solely by the claims. Furthermore, unless the exact phrase "method for..." is followed by a participle, none of the claims are intended to invoke 35 USC. 112(f). Any other term used in the claims (including, but not limited to, “mechanism,” “module,” “device,” “unit,” “component,” “element,” “building,” “device,” “machine,” “system,” “processor,” or “controller”) is understood by the applicant to refer to a structure known to a person skilled in the art and is not intended to invoke 35 USC. 112 (f). Attached Figure Description

[0019] To gain a more complete understanding of this disclosure and its advantages, reference will now be made to the following description taken in conjunction with the accompanying drawings, wherein similar reference numerals denote similar parts: Figure 1 An example network configuration for alignment of multi-frame non-Bayer data according to this disclosure is shown; Figure 2 An example process for aligning multiple frames of non-Bayer data according to this disclosure is shown; Figure 3 An example flow for aligning multi-frame Tetra data according to this disclosure is shown; Figure 4 The present disclosure shows the method for use Figure 3 Example image alignment of multiple frames of Tetra data within an example process; Figures 5A to 5E An example of re-stitching during image alignment of multi-frame Tetra data is shown according to this disclosure; Figure 6 The following is illustrated in the combination according to the present disclosure. Figures 5A to 5E An example array of brightness values ​​derived after re-stitching; Figure 7A and Figure 7BAn example brightness image with smoothing operation applied according to this disclosure and an identical brightness image without smoothing operation are shown; Figures 8A to 8C An example of motion vectors with coarse-to-fine alignment according to this disclosure is shown; Figure 9A and Figure 9B An example effect of median filter regularization according to this disclosure is shown; Figure 10A and Figure 10B Example variables are shown for the process of structure-guided mesh deformation according to this disclosure; Figure 11A and Figure 11B as well as Figure 12A and Figure 12B An example effect of structural preservation refinement during deformation is shown according to this disclosure; Figure 13 and Figure 14 An example of re-stitching and brightness calculation applied to a Hexa pattern CFA according to this disclosure is shown; Figure 15A and Figure 15B An example effect on HDR contrast achieved through improved alignment according to this disclosure is shown; Figure 16A and Figure 16B An example effect on edge sharpness achieved through improved alignment according to this disclosure is shown; and Figure 17A and Figure 17B An example of blurring effect achieved through improved alignment according to this disclosure is shown. Detailed Implementation

[0020] The following discussion is described with reference to the accompanying drawings. Figures 1 to 17B Various embodiments of this disclosure are also described. However, it should be understood that this disclosure is not limited to these embodiments, and all changes and / or equivalents or substitutions thereof are also within the scope of this disclosure. Throughout the specification and drawings, the same or similar reference numerals may be used to refer to the same or similar elements.

[0021] As mentioned above, image registration or image alignment refers to aligning multiple images / frames of a color filter array (CFA) camera sensor data without introducing significant image distortion in the presence of camera motion or small object movement. Image registration is often a necessary or desired component of image processing workflows, such as in multi-frame blending algorithms to support high dynamic range (HDR) imaging or motion blur reduction. These and other algorithms can be used to fuse multiple images, such as images captured using different exposure / ISO sensitivity settings.

[0022] In multi-frame fusion of Tetra camera data, aligning the Tetra CFA can be a useful or important step. If this alignment is of low quality, the result can directly impact subsequent image blending operations, potentially leading to insufficient blending levels or even ghosting artifacts. The Tetra CFA has a repeating pattern of four pixels for each color, which differs from the pixel pattern in the associated Bayer CFA. Algorithms designed for Bayer CFA registration, based on first converting the Tetra CFA to a Bayer pattern by merging every four pixels, can be directly applied to the Tetra CFA, but this process is not optimal and results in a reduction in image data resolution.

[0023] As the smartphone camera industry moves towards high-megapixel cameras (such as 200-megapixel (MP) or higher), image sensors are now employing more novel color filter array patterns (such as the Tetra pattern, Nona pattern, and Hexa Deca (Tetra2) pattern). The Tetra sensor (also referred to herein as the Tetra sensor) is gaining popularity. The Tetra sensor offers the flexibility of extremely high-resolution capture (compared to Bayer sensors or other types of sensors), while also allowing for the flexible merging of adjacent pixels consisting of the same color channels to better image the signal in low-light or short-exposure scenes, which can improve the signal-to-noise ratio at the expense of resolution. Given the very high resolution in a Tetra sensor, the captured data can be noisier compared to a typical 12MP Bayer sensor due to the small pixel size.

[0024] Tetra core patterns are typically presented as 4×4 pixel arrays. In a Tetra sensor, each 4×4 pixel array comprises multiple 2×2 arrays. Each 2×2 array is aggregated using the same color filter. For example, each 4×4 Tetra core pattern includes one 2×2 array with a blue color filter, another 2×2 array with a red color filter, and two more 2×2 arrays. Aligning the Tetra CFA can be a useful or important operation in multi-frame fusion of Tetra camera data. If this operation is of low quality, various problems such as those described above can occur.

[0025] A Tetra-to-Bayer conversion can be used for Tetra color filter arrays, which have a repeating pattern of four pixels for each color, unlike the pixel pattern in the associated Bayer color filter array. Based on first converting Tetra pixel data into a Bayer pattern by merging every four pixels, current methods designed for alignment of associated Bayer color filter arrays can be directly applied to Tetra color filter arrays. In an example Tetra-to-Bayer pattern conversion, the four color pixels are simply averaged, producing a 2×2 array with one pixel corresponding to the blue filter, one pixel corresponding to the red filter, and two pixels corresponding to the green filter. This method of processing Tetra images is not optimal and leads to a reduction in data resolution, such as from... (in, It is the height of the sensor array, measured in pixels. (This refers to the width of the sensor array, measured in pixels) to .

[0026] This disclosure employs the re-stitching of Tetra patterns or other patterns to convert an image into a Bayer color filter array pattern while minimizing or avoiding resolution loss. In embodiments, a bicubic filter or other smoothing operation may be applied to remove color filter array artifacts. A registration algorithm may be applied to the luminance data, and optionally, a regularized median filter may be used for the motion vector. The final motion vector may be obtained, for example, by magnification and thinning at full resolution, to improve accuracy.

[0027] Figure 1 An example network configuration 100 for alignment of multi-frame non-Bayer data according to this disclosure is shown. Figure 1 The embodiment of network configuration 100 shown is for illustrative purposes only. Other embodiments of network configuration 100 may be used without departing from the scope of this disclosure.

[0028] According to embodiments of this disclosure, electronic device 101 is included in network configuration 100. Electronic device 101 may include at least one of bus 110, processor 120, memory 130, input / output (I / O) interface 150, display 160, communication interface 170, or sensor 180. In embodiments, electronic device 101 may not include at least one of these components, or at least one other component may be added. Bus 110 includes circuitry for connecting components 120-180 to each other and for transmitting communication (such as control messages and / or data) between components.

[0029] Processor 120 includes one or more processing devices, such as one or more microprocessors, microcontrollers, digital signal processors (DSPs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). In embodiments, processor 120 includes one or more of a central processing unit (CPU), application processor (AP), communication processor (CP), or graphics processing unit (GPU). Processor 120 is capable of performing control and / or performing operations or data processing related to communication or other functions on at least one of the other components of electronic device 101.

[0030] Memory 130 may include volatile memory and / or non-volatile memory. For example, memory 130 may store commands or data associated with at least one other component of electronic device 101. According to embodiments of this disclosure, memory 130 may store software and / or program 140. Program 140 includes, for example, kernel 141, middleware 143, application programming interface (API) 145, and / or application programs (or “applications” 147). At least a portion of kernel 141, middleware 143, or API 145 may be represented as an operating system (OS).

[0031] Kernel 141 can control or manage system resources (such as bus 110, processor 120, or memory 130) used to perform operations or functions implemented in other programs (such as middleware 143, API 145, or application 147). Kernel 141 provides an interface that allows middleware 143, API 145, or application 147 to access various components of electronic device 101 to control or manage system resources. Application 147 can support various functions related to image alignment for multi-frame fusion of Tetra or other image data. These functions can be performed by a single application or by multiple applications, where each application performs one or more of these functions. For example, middleware 143 can act as a repeater to allow API 145 or application 147 to communicate data with kernel 141. Multiple applications 147 can be provided. Middleware 143 is able to control work requests received from application 147, such as by prioritizing the use of system resources (such as bus 110, processor 120, or memory 130) of electronic device 101 for at least one of the multiple applications 147. API 145 is an interface that allows application 147 to control functionality provided from kernel 141 or middleware 143. For example, API 145 includes at least one interface or function (such as commands) for file control, window control, image processing, or text control.

[0032] I / O interface 150 serves as an interface for transmitting, for example, commands or data input from a user or other external device to other components of electronic device 101. I / O interface 150 can also output commands or data received from other components of electronic device 101 to the user or other external device.

[0033] Display 160 includes, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, a quantum dot light-emitting diode (QLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display. Display 160 can also be a depth-sensing display (such as a multi-focus display). Display 160 is capable of displaying various content to a user, such as text, images, videos, icons, or symbols. Display 160 may include a touchscreen and can receive input such as touch, gestures, proximity, or hover input using an electronic pen or a user's body part.

[0034] Communication interface 170, for example, enables communication between electronic device 101 and external electronic devices (such as first electronic device 102, second electronic device 104, or server 106). For instance, communication interface 170 can be connected to network 162 or 164 via wireless or wired communication to communicate with external electronic devices. Communication interface 170 can be a wired transceiver or a wireless transceiver or any other component for transmitting and receiving signals.

[0035] Wireless communication can use at least one of the following as a communication protocol: WiFi, LTE, LTE-A, 5G, millimeter wave or 60 GHz wireless communication, wireless USB, CDMA, WCDMA, UMTS, Wi-Fi, or GSM. Wired connections may include at least one of the following: USB, HDMI, RS-232, or POTS. Network 162 or 164 includes at least one communication network, such as a computer network (e.g., a local area network (LAN) or wide area network (WAN)), the Internet, or a telephone network.

[0036] Electronic device 101 also includes one or more sensors 180 that can measure physical quantities or detect the activation state of electronic device 101 and convert the measured or detected information into electrical signals. For example, one or more sensors 180 may include one or more cameras or other imaging sensors for capturing images of a scene. Sensor 180 may also include one or more buttons for touch input, one or more microphones, gesture sensors, gyroscopes or gyroscope sensors, atmospheric pressure sensors, magnetic sensors or magnetometers, accelerometers or accelerometers, grip sensors, proximity sensors, color sensors (such as RGB sensors), biophysical sensors, temperature sensors, humidity sensors, illuminance sensors, ultraviolet (UV) sensors, electromyography (EMG) sensors, electroencephalography (EEG) sensors, electrocardiography (ECG) sensors, infrared (IR) sensors, ultrasonic sensors, iris sensors, or fingerprint sensors. Sensor 180 may also include an inertial measurement unit, which may include one or more accelerometers, gyroscopes, and other components. Furthermore, sensor 180 may include control circuitry for controlling at least one of the sensors included herein. Any of these sensors 180 may be located within the electronic device 101. In some embodiments, the sensor 180 includes at least one camera or other imaging sensor that captures multiple frames of images, and the electronic device 101 may perform image alignment of two or more images within the captured multiple frames, as described in further detail below.

[0037] In embodiments, the first external electronic device 102 or the second external electronic device 104 may be a wearable device or an electronically mountable wearable device (such as a head-mounted display (or "HMD"). When electronic device 101 is mounted in electronic device 102 (such as an HMD), electronic device 101 can communicate with electronic device 102 via communication interface 170. Electronic device 101 can be directly connected to electronic device 102 to communicate with electronic device 102 without involving a separate network. Electronic device 101 may also be an augmented reality wearable device (such as glasses) or a VR or XR headset that includes one or more imaging sensors.

[0038] The first external electronic device 102, the second external electronic device 104, and the server 106 may be devices of the same or different types as electronic device 101. According to embodiments of this disclosure, server 106 includes a combination of one or more servers. Furthermore, according to embodiments of this disclosure, all or some of the operations performed on electronic device 101 can be performed on another electronic device or a plurality of other electronic devices (such as electronic devices 102 and 104 or server 106). Furthermore, according to embodiments of this disclosure, when electronic device 101 is required to perform some functions or services automatically or on request, electronic device 101 may request other devices (such as electronic devices 102 and 104 or server 106) to perform at least some of the functions associated with it, rather than performing the function or service alone, or electronic device 101 may request other devices (such as electronic devices 102 and 104 or server 106) to perform at least some of the functions associated with it in addition to running the function or service. Other electronic devices (such as electronic devices 102 and 104 or server 106) are capable of performing the requested function or additional function and transmitting the result of the performance to electronic device 101. Electronic device 101 can provide the requested function or service by processing the received result as is or additionally. For this purpose, cloud computing, distributed computing, or client-server computing technologies can be used, for example. Although Figure 1 The electronic device 101 is shown to include a communication interface 170 for communicating with an external electronic device 104 or a server 106 via a network 162 or 164, but according to embodiments of this disclosure, the electronic device 101 can operate independently without a separate communication function.

[0039] Server 106 may include components 110-180 (or suitable subsets thereof) that are the same as or similar to those in electronic device 101. Server 106 may support electronic device 101 by performing at least one of the operations (or functions) implemented on electronic device 101. For example, server 106 may include a processing module or processor that can support processor 120 implemented in electronic device 101. As described in more detail below, server 106 may perform various operations related to image alignment for multi-frame fusion of Tetra or other non-Bayer image data.

[0040] although Figure 1 An example of a network configuration 100 is shown, including an electronic device 101 for aligning multiple frames of non-Bayer data, but it is possible to modify the network configuration 100. Figure 1 Various changes can be made. For example, network configuration 100 can include any number of each component in any suitable arrangement. Typically, computing and communication systems have a wide variety of configurations, and Figure 1 This disclosure is not intended to limit the scope to any particular configuration. Furthermore, although… Figure 1An operating environment is shown in which the various features disclosed in this patent document can be used, but these features can be used in any other suitable system.

[0041] Figure 2 An example procedure 200 for aligning multiple frames of non-Bayer data according to this disclosure is shown. For ease of explanation, Figure 2 Process 200 is described as using Figure 1 The process 200 is executed by electronic device 101 in network configuration 100. However, any other suitable device can be used and the process 200 can be executed in any other suitable system.

[0042] like Figure 2 As shown, process 200 obtains a non-Bayer CFA input image including a reference image and a non-reference image (step 201). In an embodiment, the non-Bayer CFA input image may be a Tetra image or a Hexa image. A reference brightness image corresponding to a reference Bayer-like pattern is generated based on the reference image, and a non-reference brightness image corresponding to a non-reference Bayer-like pattern is generated based on the non-reference image (step 202). For example, in an embodiment, the operation of re-stitching the reference image and the non-reference image into a Bayer-like pattern may be performed. In an embodiment, the Bayer-like pattern may be formed by swapping pixels in the central region of the non-Bayer CFA. In an embodiment, pixel data may be read from the reference image and the non-reference image in a manner corresponding to the Bayer-like pattern. The resolution of each of the reference brightness image and the non-reference brightness image may be approximately half the resolution of each of the non-Bayer CFA input images.

[0043] Smoothing operations are performed on the reference and non-reference brightness images, such as to remove artifacts caused by non-Bayer CFA patterns, and a filtered reference brightness image and a filtered non-reference brightness image are generated (step 203). Motion vectors are identified based on the filtered brightness images (step 204). For example, in an embodiment, motion vectors can be identified by: comparing the filtered reference brightness image and the filtered non-reference brightness image to generate initial motion vectors with coarse-to-fine alignment, regularizing the initial motion vectors using a median filter, and performing local alignment of the regularized motion vectors based on structure-guided mesh deformation.

[0044] The motion vector and filtered brightness image are magnified to generate magnified motion vectors, a magnified reference brightness image, and a magnified non-reference brightness image (step 205). High-resolution thinning of the magnified motion vectors is performed based on the magnified brightness image to generate the final motion vector (step 206). Example thinning may involve: warping the magnified non-reference brightness image based on the magnified motion vectors; performing a block search for each pixel in the resulting warped magnified non-reference brightness image based on a comparison with the magnified reference brightness image; and thinning the magnified motion vectors based on the block search. The non-Bayer CFA input images are aligned with each other based on the final motion vector (step 207).

[0045] although Figure 2 An example of a process 200 for aligning multiple frames of non-Bayer data is shown, but it is possible to modify... Figure 2 Make various changes. For example, although shown as a series of steps, Figure 2 The steps in the process can overlap, occur in parallel, occur in different orders, or occur any number of times (including zero times).

[0046] Figure 3 An example flow 300 for aligning multi-frame Tetra data according to this disclosure is shown. For ease of explanation, Figure 3 Process 300 is described as being by Figure 1 The electronic device 101 in the network configuration 100 is implemented or used. However, the example process 300 can be implemented or used by any other suitable device and in any other suitable system.

[0047] like Figure 3 As shown, multiple frames of images 301 (such as those acquired during multi-frame capture) are processed through a series of operations to produce a single image frame 306. Essentially, the single image frame 306 is obtained by fusing the multiple frames of images 301 using a series of operations including preprocessing 302, image alignment 303, image blending 304, and post-processing 305. Examples of preprocessing 302 may include white balance adjustment, black level adjustment, and lens shading correction. Both the input and output of preprocessing 302 may include image data from multiple frames.

[0048] Image alignment 303 involves selecting a frame from a multi-frame image 301 as a reference for alignment purposes. The selected frame can be the first frame, an intermediate frame, or any other frame in the multi-frame image set (e.g., in the case of a Reference Frame Selection (RFS) algorithm). Both the input and output of image alignment 303 can include image data from multiple frames. Image alignment 303 can align multiple images / frames of data in Tetra color filter array format or other non-Bayer color filter array format in the presence of camera motion or object motion without introducing significant image distortion in the output single image frame 306. Therefore, image alignment 303 can be used as a component of flow 300 in any multi-frame blending process, such as high dynamic range (HDR) imaging, motion blur reduction, or other processes such as fusing multiple images (multiple images captured with different exposure / ISO sensitivity settings).

[0049] Image alignment according to this disclosure can re-stitch Tetra patterns or other non-Bayer patterns to convert them into Bayer-like CFAs without loss of resolution. Afterward, smoothing filters (such as bicubic filters) can be applied to remove CFA artifacts. A registration process can be applied to the luminance data, where a regularized median filter may be used on the motion vectors. The resulting final motion vectors can be magnified and thinned at full resolution to improve accuracy. Therefore, this disclosure proposes a technique for alignment of Tetra CFA or other CFA data that combines robustness provided by median filtering with higher accuracy provided by increased resolution. The details of image alignment 303 are further described below.

[0050] During image blending 304, frames from multiple images 301 (such as frames aligned with a selected reference image) are averaged or otherwise blended in areas with little or no motion, while areas with motion are selected from the reference frame to avoid ghosting artifacts. The input to image blending 304 may include image data from multiple frames, and the output of image blending 304 may include a single frame. Post-processing 305 following image blending 304 may include, for example, image enhancement, tone mapping, color correction, etc. Both the input and output of post-processing 305 may include a single image frame 306.

[0051] Figure 4 The present disclosure shows the method for use Figure 3 Example image alignment of multiple frames of Tetra data within example flow 300, example image alignment 303. (Example flow 300) Figure 4 As shown, image alignment 303 receives a plurality of preprocessed image frames 401a-401n from preprocessing 302, wherein each of the image frames 401a-401n corresponds to an image 301 after preprocessing of the type described above. Figure 4In the example, each of image frames 401a-401n has a Tetra CFA pattern with the same resolution (such as a 2H×2W array, where H is the height of the array in pixels and W is the width of the array in pixels). Process 300 re-stitches the Tetra CFA patterns to convert them into Bayer-like CFA patterns without loss of resolution. In blocks 402a-402n, each of image frames 401a-401n can be actually or effectively re-stitched and subjected to luminance conversion and smoothing operations.

[0052] although Figure 3 An example flow 300 for aligning multi-frame non-Bayer data is shown, and Figure 4 An example of an architecture for image alignment 303 for multi-frame non-Bayer images is shown, but it is applicable to other methods. Figure 3 and Figure 4 Various changes were made. For example, although described in the context of the Tetra image CFA pattern... Figure 3 and Figure 4 This is an example, but the same process 300 and architecture for image alignment 303 can be used with other CFA patterns. For example, process 300 and architecture for image alignment 303 can be implemented for Hexa-Deca CFA patterns or Nona CFA patterns.

[0053] Figures 5A to 5E An example re-stitching is shown during image alignment 303 of multi-frame Tetra data according to this disclosure, and Figure 6 The following is illustrated in the combination according to the present disclosure. Figures 5A to 5E An example array of brightness values ​​derived after re-stitching is explained. In some previous methods, a regular Tetra-to-Bayer conversion is performed by merging every four pixels. According to this disclosure, Tetra data or other non-Bayer image data can be re-stitched.

[0054] Figure 5A A marker pixel arrangement 500 for a 16×16 array of Tetra CFA pattern image data is shown, wherein each pixel can be uniquely labeled. During re-stitching, using... Figure 5B The central region 510 of the array is shown. The central region 510 is a collection of pixels not located in the first column, last column, first row, or last row of the Tetra CFA pattern image data array. This can be achieved via, for example... Figure 5B The arrows in the diagram indicate the pixel swapping used to reassemble the pixels within the central region 510, thereby producing... Figure 5C The marked pixel arrangement 520 shown in the figure (where pixels are reserved) Figure 5A(The original pixel markers). As can be seen here, the resulting arrangement comprises a Bayer-like CFA pattern set in the central region 510. For the purpose of determining luminance values, it can be as follows: Figure 5D The pixels in the central region 510 of the remarked array are shown to produce a marked pixel arrangement 540.

[0055] like Figure 6 The brightness array 600 shown can be based on the brightness values. Figure 5D The arrangement of marker pixels 520 in the formula is calculated using the following equation:

[0056] here, Corresponding to Figure 5D The marker pixels are arranged within a 540-pixel Bayer-like CFA pattern index. Furthermore, It is a linear function, which can have the following forms:

[0057] The result of the re-stitching is an intermediate Bayer-like pattern used to calculate the luma array for subsequent registration of different image frames 401a-401n. As can be seen here, the luma array 600 has approximately half the resolution (e.g., H×W) of the input Tetra CFA pattern image data. Furthermore, it should be noted that this can be achieved without actually generating... Figure 5C and Figure 5D The luminance array 600 is derived in the case of an intermediate Bayer-like array of the type shown. In other methods, for example, it can be based on... Figure 5E The marked pixel arrangement 540 shown marks the pixels in the central region 510, reading the Tetra image data pixels in a modified order. By reading the pixel data in this manner, the brightness array 600 can be obtained using the same calculations given above without producing an intermediate Bayer-like array.

[0058] although Figures 5A to 5E as well as Figure 6 Together, they illustrate an example of re-stitching during image alignment and the corresponding brightness array generated, but this is subject to change. Figures 5A to 5E as well as Figure 6 Various changes were made. For example, although described in the context of the Tetra image CFA pattern... Figures 5A to 5E as well as Figure 6 This is an example, but similar re-stitching and brightness array generation can be achieved for Hexa-Deca CFA patterns, as shown below. Figure 13 and Figure 14 Described.

[0059] Return to reference Figure 4After calculating the luminance arrays for each image frame 401a-401n, a smoothing filter (such as a bicubic filter) can be applied to each luminance array (e.g., to remove CFA artifacts), thereby producing smoothed luminance arrays 403a-403n corresponding to the input Tetra image data 401a-401n respectively. The smoothing filter (such as a bicubic filter or a Gaussian filter) can be applied to both the Tetra luminance image and the non-Tetra luminance image, for example, to smooth artifacts caused by spatial offset. In an embodiment, a general equation for such a smoothing filter can be expressed as follows.

[0060]

[0061] here, Pixel position , The brightness value at that location, and It is the output of the smoothing filter, and These are the coefficients of the smoothing filter, and , yes and The kernel width in the direction. A canonical example of such a value can be represented as follows.

[0062]

[0063] Figure 7A and Figure 7B An example brightness image with smoothing operation applied according to this disclosure and an identical brightness image without smoothing operation are shown. As can be seen in the comparison, the smoothing operation reduces noise artifacts around the region.

[0064] As described above, one of the images 301 is selected as a reference. Therefore, the corresponding image frame (such as reference image frame 401a) can be used to derive the reference luminance array 403a, while the remaining parts (image frames other than reference image frame 401a, including non-reference image frame 401n) can each be used as non-reference luminance array 403n.

[0065] Luminance arrays 403a-403n can be used by motion vector computation 404. For example, in block 405, motion vectors with coarse-to-fine alignment can be generated. As an example, a reference luminance array 403a and a non-reference luminance array 403n from blocks 402a-402n can be compared to generate motion vectors with coarse-to-fine alignment (or, in other words, in a coarse-to-fine search scheme, to find where each pixel in the reference luminance array 403a has moved to in the non-reference luminance array 403n). In block 406, the motion vectors output from block 405 can be regularized, for example, using a median filter. In block 407, structure-guided mesh deformation (SGMW) can be performed on the output of block 406 to perform local alignment on regions where features are found and global alignment on regions where features are sparse. The output from motion vector computation 404 (in Figure 4 The output of block 407 in the example may include a motion vector of an M×N array.

[0066] The luminance arrays 403a-403n can also be magnified by blocks 409a-409n (e.g., magnified to 2H×2W), and the motion vectors from motion vector calculation 404 can similarly be magnified by block 408 (e.g., magnified to 2M×2N). In an embodiment, the magnification operation can be matched to the resolution of a non-Bayer CFA input. In block 410, high-resolution thinning of the motion vectors output by motion vector calculation 404 can be performed, for example, by using the magnified motion vectors from block 408 and the magnified luminance arrays from blocks 409a-409n. In an embodiment, block search can be used to perform high-resolution thinning, such as to remove small errors. The thinned (and magnified) motion vectors output by block 410 can be used by image blending 304 in process 300.

[0067] Figures 8A to 8C An example of motion vector generation with coarse-to-fine alignment according to this disclosure is shown. This can be used, for example, as... Figure 4 It is a portion of block 405 that is executed. More specifically, Figure 8A Reference image frame 801a is shown (such as that which can be used as) Figure 4 (a specific image in reference image frame 401a), while Figure 8B A non-reference image frame 801n is shown (such as one that can be used as a reference). Figure 4 The corresponding image of non-reference image frame 401n and reference image frame 801a in the image.

[0068] In an embodiment, Figures 8A to 8C The method shown is similar to the method described in U.S. Patent No. 11,151,731 (the entire contents of which are incorporated herein by reference). Figures 8A to 8CIn the example, coarse-to-fine alignment can be performed on the four Gaussian pyramids of the input frame regions 810a-810d. At each level, a search can be performed on the corresponding patch within the non-reference image frame 801n in the neighborhood of each patch in the reference image frame 801a, for example, by using a motion vector estimated from a coarser scale as an initial guess. The patch size and search radius can vary with different levels. When upsampling the motion vector at coarser levels, multiple hypotheses can be evaluated to avoid boundary issues. In an embodiment, for images with the same exposure, the search for the nearest matching patch can minimize the L2 norm distance. In an embodiment, for images with different exposures, the search can maximize the normalized cross-correlation. The search can generate pixel-level alignment, and quadratic functions or other functions can be used to fit the nearest pixel minimum and directly compute the subpixel minimum to generate subpixel-accurate motion vectors.

[0069] although Figures 8A to 8C An example of motion vector generation during image alignment is shown, but it is applicable to other methods. Figures 8A to 8C Various changes can be made. For example, during the search, it is not necessary to constrain the motion vectors of large moving objects.

[0070] Return to reference Figure 4 In block 405, motion vectors can be generated in a coarse-to-fine scheme based on the reference brightness array 403a and the non-reference brightness array 403n, and in block 406, the motion vectors can be regularized using a median filter. This can be achieved, for example, by replacing the motion vector at each tile with the median of the motion vectors in the neighborhood of that tile. This represents a regularized form of motion estimation that helps remove noise. The estimated motion vectors can be... , An example of performing median filtering can be defined as follows, where, At pixel position place The values ​​of motion in the direction (estimated from coarse to fine alignment), and At the same location Corresponding motion in the direction. Therefore, in this embodiment, the refined motion vector... , It can be represented as follows.

[0071]

[0072]

[0073] Here, the median is defined as the value that separates the upper and lower halves of the set, and , It calculates the median of the motion vectors. direction and The range in direction. In an embodiment, .

[0074] Figure 9A and Figure 9B An example effect of median filter regularization according to this disclosure is shown. More specifically, Figure 9A It is a non-reference image deformed using motion vectors computed with coarse-to-fine alignment without median filter regularization (based on the output of block 405 without using block 406). Figure 9B This is the same non-reference image deformed after median filtering of the motion vectors used (using the output of block 406). The comparison shows that median filtering preserves the continuity of edges in the deformed image.

[0075] Return to reference Figure 4 The regularized motion vectors from block 406 can be further refined by combining global and local information. For example, block 407 can perform structure-guided mesh deformation (SGMW) on the regularized motion vectors to perform local alignment on regions where features are found and global alignment on regions where features are sparse.

[0076] Figure 10A and Figure 10B Example variables of the process for structure-guided mesh deformation according to this disclosure are shown. More specifically, Figure 10A The mesh before deformation is depicted, where feature points in the mesh cells (or "tiles") are... Having vertices , , and This can be achieved by using the coordinates of the grid vertices. , and The size of the formed triangle To define each grid cell. Figure 10B The same mesh, after deformation, is depicted. Image structure is preserved using structure-preserving refinement (such as by imposing secondary constraints on mesh vertices), as shown below:

[0077] here, and It's a scaling factor. Local alignment item. This indicates how feature points in a non-reference frame (which can be represented by a bilinear combination of vertices) can be deformed to align with their corresponding feature points in the reference frame. Feature points in the reference frame can include the centers of tiles at the finest scale. Corresponding feature points in the non-reference frame can include feature points that are identical to those in the reference frame but have had their calculated motion vectors shifted. Similarity Term The indicator is used to represent the coordinates of a triangle (formed by three vertices) that remains fixed after deformation. Global constraint terms. Facilitates global affine transformations in "flat regions" and "large motion regions". In an embodiment, the structure-guided mesh deformation may be similar to the structure-guided mesh deformation disclosed in U.S. Patent No. 11,151,731.

[0078] The result of the structure-guided refinement in block 407 may include motion vectors with a resolution of M×N, where M and N do indeed exceed the resolution of the input luminance image (such as H×W in this case). Figure 11A and Figure 11B as well as Figure 12A and Figure 12B An example effect of structural refinement during deformation according to this disclosure is shown. More specifically, Figure 11A and Figure 12A It is an image that is deformed without the structural refinement described, and Figure 11B and Figure 12B It is the corresponding image that is deformed while retaining the structure and refining it.

[0079] Refer again Figure 4 Amplification can be performed on each of the following items to match the resolution of non-Bayer CFAs: (a) The mesh obtained from the structure-guided mesh deformation of block 407 (in block 408) produces a 2M×2N motion vector; (b) The reference brightness image obtained from block 402a (in block 409a) is used to generate a 2H×2W magnified reference brightness image; and (c) The non-reference brightness image obtained from block 402n (in block 409n) is used to generate a 2H×2W magnified non-reference brightness image.

[0080] In an embodiment, blocks 408, 409a, and 409n can perform bilinear magnification, wherein linear interpolation is first applied in one direction to obtain interpolation between adjacent pixel / motion vector positions, and then linear interpolation is applied in an orthogonal direction between adjacent positions to obtain a brightness image and a motion vector with twice the resolution of the original resolution.

[0081] In block 410, high-resolution thinning can be performed, for example via block search, to remove small errors, using the magnified reference brightness array from block 409a, the magnified motion vector from block 408, and the magnified non-reference brightness array from block 409n. Here, the magnified motion vector from block 408 can be used to warp the magnified non-reference brightness image from block 409n. In an embodiment, block search can be performed for each pixel of the warped and magnified non-reference images to thin the motion vector. The resolution of the resulting thinned motion vector is twice the resolution of the original motion vector generated by motion vector calculation 404.

[0082] In this embodiment, the block search can be performed in the same manner as coarse-to-fine alignment, but at a single scale. For example, a search can be performed on the corresponding tile in the neighborhood of each tile in the magnified reference frame from block 409a, using the magnified motion vector estimated from block 408 as an initial guess. The search can be performed by finding tiles in the non-reference frame that minimize the L2 norm relative to the reference tile. The location of the tile minimizing the L2 norm is relative to the reference tile location. direction, The difference in direction can be considered as a motion vector at that tile. The resulting motion vector (2M×2N) can be used to align a 2H×2W non-reference image frame 401n in a non-Bayer CFA image with a reference image frame 401a in a non-Bayer CFA image. In an embodiment, the alignment algorithm that can be used involves the following operations: for a position in the reference image The tile at the location arrive Find the tile that minimizes the L2 norm between the non-reference tile and the reference tile in the non-reference image. In a particular embodiment, .

[0083] Figures 5A to 5E as well as Figure 6 The re-stitching and brightness calculation for Tetra image input are shown, where, Figure 5E The associated description explains how Tetra image pixels can be read in a modified order to efficiently re-stitch the input for luminance calculation purposes. Re-stitching and luminance calculation can be similarly applied to any general-purpose non-Bayer RGB CFA to calculate luminance without loss of resolution and without merging. Figure 13 and Figure 14 An example of re-stitching and brightness calculation applied to a Hexa pattern CFA according to this disclosure is shown. This is done to obtain [the desired result] without creating an intermediate pixel pattern. Figure 14 brightness array, Figure 13The pixels can be read in a modified order and used to calculate brightness, such as in the following manner.

[0084]

[0085] Other embodiments may use the following generalized equation.

[0086]

[0087]

[0088]

[0089] or

[0090] In the context of HDR applications, the techniques described above can use multiple input frames with different exposure times but the same ISO (meaning some of the input frames may be overexposed or underexposed) to restore the high dynamic range of a scene. For example, Figure 15A and Figure 15B An example effect on HDR contrast achieved through improved alignment according to this disclosure is shown. Here, alignment without improvement is shown ( Figure 15A ) and with improved alignment ( Figure 15B The difference in HDR contrast, where the contrast in the sky area allows for a clearer view of cloud boundaries.

[0091] In the context of noise reduction applications, improved alignment allows for precise blending and merging of frames to reduce noise and improve detail. Figure 16A and Figure 16B An example effect on edge sharpness achieved through improved alignment according to this disclosure is shown. Here, an example of edge sharpness without improved alignment is shown. Figure 16A ) and with improved alignment ( Figure 16B The edge obtained under the condition of ).

[0092] In the context of motion blur reduction across multiple input frames involving moving objects, precise registration across different frames aligns moving objects and reduces blur caused by motion. Figure 17A and Figure 17B An example of blurring achieved through improved alignment according to this disclosure is shown. Here, it can be seen that without improved alignment ( Figure 17A ) and with improved alignment ( Figure 17B In the same image frame processed under the same conditions, the lines become less blurry and the text becomes clearer.

[0093] In summary, within the context of multi-frame fusion, the aforementioned techniques can improve HDR applications (where input frames are exposed differently), motion blur reduction applications (where input frames have different noise levels), burst noise reduction (where input frames are exposed equally and have similar noise levels), panoramic views (where input frames are captured from different angles), and multi-camera fusion (where input frames are captured from different lenses). For multi-camera fusion, these techniques can be used to align input frames from Tetra data or other data from different lenses / sensors, provided there is a sufficiently large overlap in the content.

[0094] It should be noted that the functions shown in the figures or described above can be implemented in any suitable manner in electronic devices 101, 102, 104, server 106, or other devices. For example, in an embodiment, at least some of the functions shown in the figures or described above can be implemented or supported using one or more software applications or other software instructions executed by the processor 120 of electronic devices 101, 102, 104, server 106, or other devices. In an embodiment, at least some of the functions shown in the figures or described above can be implemented or supported using dedicated hardware components. Generally, any suitable hardware or any suitable combination of hardware and software / firmware instructions can be used to perform the functions shown in the figures or described above. Furthermore, the functions shown in the figures or described above can be performed by a single device or multiple devices.

[0095] Although this disclosure has been described with reference to various exemplary embodiments, various changes and modifications may be suggested to those skilled in the art. This disclosure is intended to cover such changes and modifications that fall within the scope of the appended claims.

[0096] The specific examples provided to explain embodiments according to this disclosure are merely combinations of each standard, method, detailed method, and operation, and the various embodiments described herein can be performed by combinations of at least two or more of the various techniques described. Furthermore, in this case, the method can be performed according to a method determined by one or at least two or more of the foregoing techniques. For example, a combination of a portion of the operation of one embodiment and a portion of the operation of another embodiment can be performed.

Claims

1. A method comprising: At least one processing device using an electronic device obtains a non-Bayer color filter array (CFA) input image, wherein the non-Bayer CFA input image includes a reference non-Bayer CFA input image and a non-reference non-Bayer CFA input image, each of the reference non-Bayer CFA input image and the non-reference non-Bayer CFA input image having a non-Bayer CFA pattern; The at least one processing device generates a reference brightness image corresponding to a reference Bayer-like pattern based on the reference non-Bayer CFA input image, and generates a non-reference brightness image corresponding to a non-reference Bayer-like pattern based on the non-reference non-Bayer CFA input image, wherein the resolution of each of the reference brightness image and the non-reference brightness image is approximately half the resolution of each of the reference non-Bayer CFA input image and the non-reference non-Bayer CFA input image; The at least one processing device is used to perform a smoothing operation on the reference brightness image and the non-reference brightness image to remove artifacts caused by the non-Bayer CFA pattern and generate a filtered reference brightness image and a filtered non-reference brightness image. The at least one processing device is used to identify motion vectors based on a filtered reference brightness image and a filtered non-reference brightness image; The motion vector, the filtered reference brightness image, and the filtered non-reference brightness image are magnified using the at least one processing device to generate magnified motion vector, magnified reference brightness image, and magnified non-reference brightness image; The at least one processing device is used to perform high-resolution thinning of the magnified motion vector based on the magnified filtered reference brightness image and the magnified filtered non-reference brightness image to generate the final motion vector; and The reference non-Bayer CFA input image and the non-reference non-Bayer CFA input image are aligned with each other based on the final motion vectors.

2. The method as described in claim 1, wherein, The smoothing operation includes a bicubic filtering operation.

3. The method according to any one of claims 1 to 2, wherein, The operation of identifying the motion vector includes: The filtered reference brightness image and the filtered non-reference brightness image are compared to generate an initial motion vector with coarse-to-fine alignment; The initial motion vector is regularized using a median filter to generate a regularized motion vector; and Structure-guided mesh deformation (SGMW) performs local alignment of regularized motion vectors to generate the motion vectors.

4. The method according to any one of claims 1 to 3, wherein, The operation of performing the high-resolution refinement includes: The magnified non-reference brightness image is deformed based on the magnified motion vector to generate a deformed magnified non-reference brightness image; Based on the comparison between a deformed, magnified non-reference brightness image and a magnified reference brightness image, a block search is performed for each pixel of the deformed, magnified non-reference brightness image; and The amplified motion vector is refined based on the block search.

5. The method according to any one of claims 1 to 4, further comprising: Each of the non-Bayer CFA input images is re-stitched to generate the reference Bayer-like pattern and the non-reference Bayer-like pattern; The resolution of each of the reference Bayer-like pattern and the non-reference Bayer-like pattern is equal to the resolution of each of the reference non-Bayer CFA input image and the non-reference non-Bayer CFA input image.

6. The method of claim 5, wherein: The non-Bayer CFA pattern includes the Tetra CFA pattern; and The operation of re-stitching each of the reference non-Bayer CFA input image and the non-reference non-Bayer CFA input image includes: for each of the reference non-Bayer CFA input image and the non-reference non-Bayer CFA input image, swapping the pixels in the central region of the Tetra CFA pattern to form multiple Bayer CFA patterns.

7. The method according to any one of claims 1 to 6, wherein, The operations for generating the reference brightness image and the non-reference brightness image include: Brightness is identified based on pixels read from the central region of the non-Bayer CFA pattern within a Bayer-like CFA pattern.

8. An electronic device comprising: Memory, storing instructions; At least one processing device is configured to: Obtain a non-Bayer color filter array CFA input image, wherein the non-Bayer CFA input image includes a reference non-Bayer CFA input image and a non-reference non-Bayer CFA input image, and each of the reference non-Bayer CFA input image and the non-reference non-Bayer CFA input image has a non-Bayer CFA pattern; A reference brightness image corresponding to a reference Bayer-like pattern is generated based on the reference non-Bayer CFA input image, and a non-reference brightness image corresponding to a non-reference Bayer-like pattern is generated based on the non-reference non-Bayer CFA input image, wherein the resolution of each of the reference brightness image and the non-reference brightness image is approximately half the resolution of each of the reference non-Bayer CFA input image and the non-reference non-Bayer CFA input image. A smoothing operation is performed on the reference brightness image and the non-reference brightness image to remove artifacts caused by the non-Bayer CFA pattern and generate a filtered reference brightness image and a filtered non-reference brightness image. Motion vectors are identified based on filtered reference brightness images and filtered non-reference brightness images; The motion vector, the filtered reference brightness image, and the filtered non-reference brightness image are magnified to generate magnified motion vector, magnified reference brightness image, and magnified non-reference brightness image; High-resolution thinning of the magnified motion vector is performed using both a magnified filtered reference brightness image and a magnified filtered non-reference brightness image to generate the final motion vector; and The reference non-Bayer CFA input image and the non-reference non-Bayer CFA input image are aligned with each other based on the final motion vectors.

9. The electronic device as claimed in claim 8, wherein, The smoothing operation includes a bicubic filtering operation.

10. The electronic device according to any one of claims 8 to 9, wherein, In order to identify the motion vector, the at least one processing device is configured to: The filtered reference brightness image and the filtered non-reference brightness image are compared to generate an initial motion vector with coarse-to-fine alignment; The initial motion vector is regularized using a median filter to generate a regularized motion vector; as well as Structure-guided mesh deformation (SGMW) performs local alignment of regularized motion vectors to generate the motion vectors.

11. The electronic device according to any one of claims 8 to 10, wherein, In order to perform the high-resolution refinement, the at least one processing device is configured to: The magnified non-reference brightness image is deformed based on the magnified motion vector to generate a deformed magnified non-reference brightness image; Based on the comparison between a deformed, magnified non-reference brightness image and a magnified reference brightness image, a block search is performed for each pixel of the deformed, magnified non-reference brightness image; and The amplified motion vector is refined based on the block search.

12. The electronic device according to any one of claims 8 to 11, wherein: The at least one processing device is further configured to: re-stitch each of the reference non-Bayer CFA input image and the non-reference non-Bayer CFA input image to generate the reference Bayer similar pattern and the non-reference Bayer similar pattern; and The resolution of each of the reference Bayer similar pattern and the non-reference Bayer similar pattern is equal to the resolution of each of the reference non-Bayer CFA input image and the non-reference non-Bayer CFA input image.

13. The electronic device of claim 12, wherein: The non-Bayer CFA pattern includes the Tetra CFA pattern; and In order to reassemble each of the reference non-Bayer CFA input image and the non-reference non-Bayer CFA input image, the at least one processing device is configured to: for each of the reference non-Bayer CFA input image and the non-reference non-Bayer CFA input image, swap the pixels in the central region of the Tetra CFA pattern to form a plurality of Bayer CFA patterns.

14. The electronic device according to any one of claims 8 to 13, wherein, In order to generate the reference brightness image and the non-reference brightness image, the at least one processing device is configured to: Brightness is identified based on pixels read from the central region of the non-Bayer CFA pattern within a Bayer-like CFA pattern.

15. A machine-readable medium containing instructions, wherein, When the instruction is executed, it causes at least one processor of the electronic device to perform the following operations: Obtain a non-Bayer color filter array CFA input image, wherein the non-Bayer CFA input image includes a reference non-Bayer CFA input image and a non-reference non-Bayer CFA input image, and each of the reference non-Bayer CFA input image and the non-reference non-Bayer CFA input image has a non-Bayer CFA pattern; A reference brightness image corresponding to a reference Bayer-like pattern is generated based on the reference non-Bayer CFA input image, and a non-reference brightness image corresponding to a non-reference Bayer-like pattern is generated based on the non-reference non-Bayer CFA input image, wherein the resolution of each of the reference brightness image and the non-reference brightness image is approximately half the resolution of each of the reference non-Bayer CFA input image and the non-reference non-Bayer CFA input image. A smoothing operation is performed on the reference brightness image and the non-reference brightness image to remove artifacts caused by the non-Bayer CFA pattern and generate a filtered reference brightness image and a filtered non-reference brightness image. Motion vectors are identified based on filtered reference brightness images and filtered non-reference brightness images; The motion vector, the filtered reference brightness image, and the filtered non-reference brightness image are magnified to generate magnified motion vector, magnified reference brightness image, and magnified non-reference brightness image; High-resolution thinning of the magnified motion vector is performed using both a magnified filtered reference brightness image and a magnified filtered non-reference brightness image to generate the final motion vector; and The reference non-Bayer CFA input image and the non-reference non-Bayer CFA input image are aligned with each other based on the final motion vectors.

Images