Dual remosaic-based image processing method, and apparatus using same

Abstract

A dual remosaic-based image processing method, and an apparatus using same are provided. The apparatus comprises: a hardware remosaic block, which performs hardware-based remosaic on non-Bayer raw image frames so as to generate Bayer raw image frames; a memory, which uses a ring buffer so as to store the non-Bayer raw image frames; a software remosaic block, which performs software-based remosaic on non-Bayer-based raw image frames from among the non-Bayer raw image frames in the ring buffer according to an image capture command, so as to generate a Bayer-based raw image frame; and an image signal processor which performs image signal processing on the Bayer raw image frames so as to generate preview color image frames, and which synthesizes at least one Bayer raw image frame of the Bayer raw image frames and a Bayer-based raw image frame according to an image capture command so as to generate a synthesized color image frame.

Description

Image processing method based on dual remosaic and device using the same

[0001] The following embodiments relate to a remosaic-based image processing method and a device using the same.

[0002] Pixel information based on new patterns, such as the Tetra pattern, Nona pattern, and 4PD (photo diode) hexadeca pattern, can be converted into pixel information based on the Bayer pattern through re-mosaic. As the resolution of image sensors increases, structures in which multiple PDs are arranged under a single microlens have begun to be used, and various new patterns of color filter arrays, such as the Tetra pattern, Nona pattern, and 4PD hexadeca pattern, have been introduced. Since existing image signal processing is mainly based on the Bayer pattern, pixel information acquired through these new pattern color filter arrays can be converted to fit the Bayer pattern through re-mosaic.

[0003]

[0004] According to one embodiment, an electronic device may include an image sensor that generates a series of non-Bayer raw image frames using a color filter array of a non-Bayer pattern. The electronic device may include a hardware re-mosaic block that performs hardware-based re-mosaic on the non-Bayer raw image frames to generate Bayer raw image frames. The electronic device may include a memory that stores the non-Bayer raw image frames using a ring buffer. The electronic device may include a software re-mosaic block that performs software-based re-mosaic on non-Bayer base raw image frames among the non-Bayer raw image frames in the ring buffer in response to an image capture command to generate Bayer base raw image frames. The electronic device may include an image signal processor that performs image signal processing on the Bayer raw image frames to generate preview color image frames, and synthesizes at least one Bayer raw image frame among the Bayer raw image frames and the Bayer base raw image frame in response to the image capture command to generate a synthesized color image frame. The electronic device may include a display module that displays preview color image frames.

[0005] According to one embodiment, an electronic device may include an image sensor that generates a series of non-Bayer raw image frames using a color filter array of a non-Bayer pattern. The electronic device may include a hardware re-mosaic block that performs hardware-based re-mosaic on the non-Bayer raw image frames to generate Bayer raw image frames. The electronic device may include an image signal processor that performs image signal processing on the Bayer raw image frames to generate color image frames. The electronic device may include a temporal noise reduction (TNR) block that enhances the color image frames using image information of a previous color image frame of the color image frames to generate preview color image frames. The electronic device may include a memory that stores the non-Bayer raw image frames and the preview color image frames using a ring buffer. The electronic device may include a software re-mosaic block that performs software-based re-mosaic on non-Bayer base raw image frames among the non-Bayer raw image frames in the ring buffer in response to a video snapshot command during video recording to generate Bayer base raw image frames. The electronic device may include an image synthesis block configured to generate a synthesized color image frame by synthesizing a base color image frame generated by an image signal processor based on a preview color image frame of preview color image frames in a ring buffer and a Bayer base raw image frame according to a video snapshot command. The electronic device may include a display module configured to display the preview color image frames.

[0006] According to one embodiment, an image processing method may include generating a series of non-Bayer raw image frames using a color filter array of a non-Bayer pattern. The image processing method may include performing hardware-based re-mosaic on the non-Bayer raw image frames to generate Bayer raw image frames. The image processing method may include storing the non-Bayer raw image frames using a ring buffer. The image processing method may include performing image signal processing on the Bayer raw image frames to generate preview color image frames. The image processing method may include performing software-based re-mosaic on a non-Bayer base raw image frame among the non-Bayer raw image frames in the ring buffer in response to an image capture command to generate a Bayer base raw image frame. The image processing method may include compositing at least one Bayer raw image frame among the Bayer raw image frames with the Bayer base raw image frame in response to the image capture command to generate a composite color image frame.

[0007] FIG. 1 illustrates a block diagram of an electronic device within a network environment, according to one embodiment.

[0008] FIG. 2 is a block diagram illustrating a camera module according to one embodiment.

[0009] FIG. 3 is a block diagram exemplarily showing the configuration of an electronic device according to one embodiment.

[0010] FIG. 4 is a drawing showing examples of various patterns of a color filter array according to one embodiment.

[0011] FIG. 5A is a diagram illustrating an image preview operation based on hardware remosaic according to one embodiment.

[0012] FIG. 5b is a diagram exemplarily illustrating an image capture operation based on hardware remosaic and software remosaic according to one embodiment.

[0013] FIG. 6A is a diagram exemplifying an image preview operation using dual streams according to one embodiment.

[0014] FIG. 6b is a diagram illustrating an image capture operation using dual streams according to one embodiment.

[0015] FIG. 7 is a block diagram illustrating an example of an electronic device according to one embodiment.

[0016] FIG. 8A is a diagram exemplifying an image preview operation using a TNR block according to one embodiment.

[0017] FIG. 8b is a diagram illustrating an image capture operation using a TNR block according to one embodiment.

[0018] FIG. 9 is a flow chart exemplarily illustrating an image processing method according to one embodiment.

[0019] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0020] FIG. 1 illustrates a block diagram of an electronic device within a network environment according to one embodiment. Referring to FIG. 1 , in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0021] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0022] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0023] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0024] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0025] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0026] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0027] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0028] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0029] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a fingerprint sensor, a temperature sensor, a humidity sensor, an illuminance sensor, or an interference sensor.

[0030] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0031] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0032] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0033] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0034] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0035] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0036] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0037] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0038] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0039] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0040] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0041] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0042] FIG. 2 is a block diagram illustrating a camera module according to one embodiment. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. The lens assembly (210) may include one or more filters.

[0043] According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. The lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0044] The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0045] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting the read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). In one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130) or as a separate memory that operates independently therefrom.

[0046] The image signal processor (260) can perform one or more image processing operations on an image acquired through the image sensor (230) or an image stored in the memory (250). The one or more image processing operations can include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) can perform control (e.g., exposure time control, readout timing control, etc.) on at least one of the components included in the camera module (180) (e.g., image sensor (230)). An image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may be configured as at least a part of the processor (120) or may be configured as a separate processor that operates independently of the processor (120). When the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) by the processor (120) as is or after undergoing additional image processing.

[0047] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

[0048] FIG. 3 is a block diagram exemplarily showing the configuration of an electronic device according to one embodiment. Since conventional image signal processing is mainly based on the Bayer pattern, pixel information obtained through a color filter array having a pattern different from the Bayer pattern (e.g., the color filter array (400) of FIG. 4) can be converted to fit the Bayer pattern through re-mosaic. The Bayer pattern can be restored from a non-Bayer pattern through re-mosaic. The non-Bayer pattern may mean a color filter array pattern other than the Bayer pattern.

[0049] Remosaic methods may include hardware-based remosaic, which performs remosaic using hardware (e.g., circuits), and software-based remosaic, which performs remosaic using software (e.g., artificial intelligence (AI)). In the case of hardware-based remosaic, conversion processing on signals from an image sensor may be performed by hardware circuits designed in advance based on remosaic operations. In the case of software-based remosaic, conversion processing on signals from an image sensor may be performed by a conversion model based on machine learning (e.g., deep learning).

[0050] In the case of a pixel arrangement characteristic in which the same color is arranged continuously, such as the color filter array (400) of FIG. 4, it may be difficult to precisely restore the Bayer pattern through hardware-based re-mosaic using a re-mosaic operation on the pixel arrangement. In the case of a pixel arrangement such as the color filter array (400) of FIG. 4, software-based re-mosaic can exhibit higher restoration performance than hardware-based re-mosaic. Since hardware-based re-mosaic utilizes pre-designed hardware circuits, it can exhibit faster processing speeds than software-based re-mosaic. For example, real-time processing such as a preview can be provided through hardware-based re-mosaic. Hardware-based re-mosaic can operate with less power.

[0051] In one embodiment, dual re-mosaic can be performed by combining hardware-based and software-based re-mosaic. In one embodiment, zero shutter lag can be achieved through dual re-mosaic using a ring buffer.

[0052] Referring to FIG. 3, an electronic device (300) (e.g., the electronic device (101) of FIG. 1, the electronic device (700) of FIG. 7) may include an image sensor (310) (e.g., the image sensor (230) of FIG. 2, the image sensor (510) of FIGS. 5A and 5B, the image sensor (610) of FIGS. 6A and 6B, the image sensor (710) of FIG. 7, the image sensor (810) of FIGS. 8A and 8B)) that generates a series of non-Bayer image frames using a color filter array of a non-Bayer pattern (e.g., the color filter array (400) of FIG. 4). The electronic device (300) may include a hardware re-mosaic block (320) (e.g., hardware re-mosaic block (520) of FIGS. 5A and 5B , hardware re-mosaic block (620) of FIGS. 6A and 6B , hardware re-mosaic block (720) of FIG. 7 , hardware re-mosaic block (820) of FIGS. 8A and 8B ) that performs hardware-based re-mosaic on non-Bayer image frames to generate Bayer image frames. The electronic device (300) may include a memory (350) (e.g., memory (130) of FIG. 1, memory (250) of FIG. 2, memory (550) of FIGS. 5A and 5B, memory (650) of FIGS. 6A and 6B, memory (760) of FIG. 7, memory (860) of FIGS. 8A and 8B) that stores non-Bayer image frames using a ring buffer (e.g., ring buffer (551) of FIGS. 5A and 5B, ring buffer (651) of FIGS. 6A and 6B, ring buffer (861) of FIGS. 8A and 8B).The electronic device (300) may include a software re-mosaic block (360) (e.g., the processor (120) of FIG. 1, the software re-mosaic block (560) of FIG. 5b, the software re-mosaic block (660) of FIG. 6b, the software re-mosaic block (770) of FIG. 7, and the software re-mosaic block (870) of FIG. 8b) that performs software-based re-mosaic on non-Bayer raw image frames among non-Bayer raw image frames in the ring buffer according to an image capture command to generate a Bayer raw image frame. The electronic device (300) may include an image signal processor (330) (e.g., an image signal processor (260) of FIG. 2, an image signal processor (530) of FIGS. 5A and 5B, an image signal processor (630) of FIGS. 6A and 6B, an image signal processor (730) of FIG. 7, an image signal processor (830) of FIGS. 8A and 8B) that performs image signal processing on Bayer raw image frames to generate preview color image frames, and synthesizes at least one Bayer raw image frame and a Bayer base raw image frame of the Bayer raw image frames according to an image capture command to generate a synthesized color image frame. The electronic device (300) may include a display module (340) that displays preview color image frames (e.g., display module (160) of FIG. 1, display module (540) of FIGS. 5A and 5B, display module (640) of FIGS. 6A and 6B, display module (750) of FIG. 7, display module (850) of FIGS. 8A and 8B).

[0053] When an image capture command is issued, a non-Bayer raw image frame of the non-Bayer raw image frames in the ring buffer corresponding to the preview color image frame of the preview color image frames displayed at the time point when the image capture command is issued can be selected as the non-Bayer raw image frame.

[0054] The image signal processor (330) can generate a synthesized color image frame by synthesizing the entire image area of ​​at least one raw image frame and the entire image area of ​​a Bayer-based raw image frame.

[0055] The image sensor (310) can generate non-Bayer raw image frames having the size of a portion of the image sensor in a crop zoom mode. The hardware remosaic block (320) can perform hardware-based remosaic on the non-Bayer raw image frames having the size of a portion of the image sensor in a crop zoom mode.

[0056] In crop zoom mode, a data stream corresponding to non-Bayer raw image frames generated by the image sensor and a data stream corresponding to Bayer raw image frames generated by the hardware remosaic block can be combined to form a dual stream.

[0057] The ring buffer can store non-Bayer raw image frames and Bayer raw image frames based on dual streams, with the non-Bayer raw image frames and Bayer raw image frames mapped to each other by frame number. Bayer raw image frames of Bayer raw image frames mapped to non-Bayer raw image frames based on dual streams can be extracted from the ring buffer and used to generate a composite color image frame.

[0058] The image signal processor (330) can generate a composite color image frame by synthesizing a Bayer image frame and a Bayer image frame having the same frame number as the Bayer image frame in Bayer basis.

[0059] The image signal processor (330) can generate a composite color image frame by synthesizing a plurality of raw image frames corresponding to at least a portion of the Bayer raw image frames and a Bayer base raw image frame.

[0060] The number of multiple raw image frames can be adjusted based on the processing time of software-based remosaic.

[0061] The hardware-based re-mosaic of the hardware re-mosaic block (320) and the software-based re-mosaic of the software re-mosaic block (360) can be performed in parallel.

[0062] The image signal processor (330) can generate a snapshot image frame by synthesizing at least one Bayer raw image frame and a Bayer base raw image frame of Bayer raw image frames according to a video snapshot command during video recording.

[0063] FIG. 4 is a diagram showing examples of various patterns of a color filter array according to one embodiment. Referring to FIG. 4, the color filter array (400) may include various patterns in which sensing pixels of the same color are arranged adjacently in a form such as 2*2, 3*3, or 4*4, such as a Tetra pattern (410), a 4PD pattern (420), a 2PD Tetra pattern (430), a Nona pattern (430), or a 4*4 Qcell pattern (440). A circle (411) may represent a microlens. The Tetra pattern (410), the 4PD pattern (420), the 2PD Tetra pattern (430), the Nona pattern (430), and the 4*4 Qcell pattern (440) are examples of various patterns in which sensing pixels of the same color are arranged adjacently, and the color filter array (400) is not limited thereto. The tetra pattern (410) can be called a quad pattern, the 4PD pattern (420) can be called a 2*2 OCL pattern (on chip lens pattern), and the 2PD tetra pattern (430) can be called an octa PD.

[0064] The color filter array (400) may be arranged in front of a sensing pixel array of an image sensor (e.g., the image sensor (230) of FIG. 2, the image sensor (310) of FIG. 3, the image sensor (510) of FIGS. 5A and 5B, the image sensor (610) of FIGS. 6A and 6B, the image sensor (710) of FIG. 7, and the image sensor (810) of FIGS. 8A and 8B). The sensing pixels may include photodiodes (PDs). The color filter array (400) may be based on color channels of a Bayer pattern. In this case, the pixels of the color filter array (400) may have red (R), green (G), and blue (B) colors. However, the color filter array (400) is not limited thereto. For example, it may be based on color channels of other patterns, such as a CYYM pattern or an RCCB pattern. In this case, the color filter array (400) may have different colors such as C (cyan) color, Y (yellow) color, M (magenta) color, and c (clear) color.

[0065] FIG. 5A is a diagram exemplarily illustrating an image preview operation based on hardware re-mosaic according to one embodiment. Referring to FIG. 5A, an image sensor (510) (e.g., the image sensor (230) of FIG. 2 , the image sensor (310) of FIG. 3 , the image sensor (610) of FIGS. 6A and 6B , the image sensor (710) of FIG. 7 , and the image sensor (810) of FIGS. 8A and 8B ) may generate non-Bayer raw image frames (e.g., the non-Bayer raw image frame (5011) of #n+9) using a color filter array of a non-Bayer pattern (e.g., the color filter array (400) of FIG. 4 ). "Raw" may represent a state output from the image sensor (510) to which no separate image signal processing is applied. #n+k may represent a frame number. n may correspond to a reference number. The image sensor (510) can sequentially generate a series of non-Bayer raw image frames (5011, 5021 to 5027) from a point corresponding to #n+1 to a point corresponding to #n+9.

[0066] Fig. 5a may represent a situation at a point in time corresponding to #n+9. The non-Bayer raw image frame (5011) of #n+9 may be being exposed by the image sensor (510). The non-Bayer raw image frames (5021 to 5027) of #n+1 to #n+7 may be stored in a memory (550) (e.g., the memory (130) of Fig. 1, the memory (250) of Fig. 2, the memory (350) of Fig. 3, the memory (650) of Figs. 6a and 6b, the memory (760) of Fig. 7, and the memory (860) of Figs. 8a and 8b). The memory (550) can store non-Bayer raw image frames (e.g., non-Bayer raw image frames (5021 to 5027) of #n+1 to #n+7) using a ring buffer (551) (e.g., ring buffer (651) of FIGS. 6A and 6B, ring buffer (861) of FIGS. 8A and 8B). The ring buffer (551) can store a limited number of image frames in a FIFO (first input first output) manner. The ring buffer (551) may correspond to a volatile memory (e.g., volatile memory (132)).

[0067] A hardware re-mosaic block (520) (e.g., hardware re-mosaic block (320) of FIG. 3, hardware re-mosaic block (620) of FIGS. 6A and 6B, hardware re-mosaic block (720) of FIG. 7, hardware re-mosaic block (820) of FIGS. 8A and 8B) can perform hardware-based re-mosaic on non-Bayer raw image frames (e.g., non-Bayer raw image frames #n+8) to generate Bayer raw image frames (e.g., Bayer raw image frames #n+8 (5012)).

[0068] An image signal processor (530) (e.g., the image signal processor (260) of FIG. 2, the image signal processor (330) of FIG. 3, the image signal processor (630) of FIGS. 6A and 6B, the image signal processor (730) of FIG. 7, and the image signal processor (830) of FIGS. 8A and 8B) can perform image signal processing on Bayer raw image frames (e.g., the Bayer raw image frame (5013) of #n+6, the Bayer raw image frame (5014) of #n+7) to generate preview color image frames (e.g., the preview color image frame (5013) of #n+6, the preview color image frame (5014) of #n+7). The color images may correspond to YUV images or RGB images.

[0069] In FIG. 5, the image signal processor (530) is illustrated to process multiple image frames of the Bayer raw image frame (5013) of #n+6 and the Bayer raw image frame (5014) of #n+7 while the image sensor (510), the hardware re-mosaic block (520), and the display module (540) process one image frame of the non-Bayer raw image frame (5011) of #n+9, the non-Bayer raw image frame (5012) of #n+8, and the non-Bayer raw image frame (5015) of #n+5, respectively. This illustrates the image processing delay and pipeline technique of the image signal processor (530), but this is only one example, and the image signal processor (530) may process one image frame or three or more image frames while the image sensor (510), the hardware re-mosaic block (520), and the display module (540) each process one image frame.

[0070] A display module (540) (e.g., display module (160) of FIG. 1, display module (340) of FIG. 3, display module (640) of FIGS. 6A and 6B, display module (750) of FIG. 7, display module (850) of FIGS. 8A and 8B) can display preview color image frames (e.g., preview color image frame (5015) of #n+5). Due to the fast processing speed of the hardware remosaic block (520), preview color image frames can be displayed in near real time.

[0071] FIG. 5B is a diagram illustrating an image capture operation based on hardware remosaic and software remosaic according to one embodiment. A user's image capture command may be generated in a situation where an image preview operation (e.g., the image preview operation of FIG. 5A) is performed.

[0072] Referring to FIG. 5b, a software remosaic block (560) (e.g., the processor (120) of FIG. 1, the software remosaic block (360) of FIG. 3, the software remosaic block (660) of FIG. 6b, the software remosaic block (770) of FIG. 7, and the software remosaic block (870) of FIG. 8b) may perform software-based remosaic on a non-Bayer raw image frame (e.g., the non-Bayer raw image frame (5025) of #n+5) in a ring buffer (551) according to an image capture command to generate a Bayer raw image frame (e.g., the Bayer raw image frame (504) of #n+5). Among the non-Bayer raw image frames, a non-Bayer raw image frame used as a base frame for image synthesis can be called a non-Bayer raw base image frame. A Bayer raw base image frame can be generated by software-based remosaicing of a non-Bayer raw base image frame.

[0073] In FIG. 5b, the non-Bayer raw image frame (5025) of #n+5 may correspond to a non-Bayer base raw image frame, and the Bayer raw image frame (504) of #n+5 may correspond to a Bayer base raw image frame. A non-Bayer raw image frame in which zero shutter lag can be achieved may be selected as the non-Bayer base raw image frame. FIG. 5b shows an example in which the Bayer raw image frame (504) of #n+5 corresponds to a Bayer base raw image frame, but is not limited thereto. Zero shutter lag can be achieved by using a ring buffer (551). Zero shutter lag may refer to a technology that makes the frame difference zero between an image that a user is viewing through a preview and an image that is stored according to a capture command. When an image capture command is issued, a non-Bayer raw image frame (e.g., a non-Bayer raw image frame (5025)) in the ring buffer (551) corresponding to a preview color image frame (e.g., a preview color image frame (5015) of #n+5 in FIG. 5A) displayed at the time point (e.g., a time point corresponding to #n+5) at which the image capture command is issued can be selected as a non-Bayer raw image frame. When shaking or eye closure is detected from a non-Bayer raw image frame, another non-Bayer raw image frame can be set as a new non-Bayer raw image frame.

[0074] The hardware re-mosaic block (520) can perform hardware-based re-mosaic on non-Bayer raw image frames (e.g., non-Bayer raw image frame (5025) of #n+5) according to an image capture command to generate Bayer raw image frames (e.g., Bayer raw image frame (503) of #n+5). Performing hardware re-mosaic and software re-mosaic on the same non-Bayer raw image frame (e.g., non-Bayer raw image frame (5025) of #n+5) can be advantageous for improving the quality of the composite result (e.g., the composited color image frame (505)). However, this is not limited to this, and it is also possible for hardware remosaic and software remosaic to be performed on different non-Bayer raw image frames (e.g., non-Bayer raw image frame (5024) of #n+4 and non-Bayer raw image frame (5025) of #n+5).

[0075] The image signal processor (530) can generate a composite color image frame (505) by synthesizing at least one Bayer raw image frame (e.g., at least one Bayer raw image frame including n+5 Bayer raw image frames (503)) and a Bayer base raw image frame (e.g., n+5 Bayer raw image frames (504)) according to an image capture command. The composite color image frame (505) can be stored in a non-volatile memory (e.g., the non-volatile memory (134)). Even when an image capture operation (e.g., the image capture operation of FIG. 5B) is performed according to an image capture command, an image preview operation (e.g., the image preview operation of FIG. 5A) can be continuously performed.

[0076] The hardware-based re-mosaic of the hardware re-mosaic block (520) and the software-based re-mosaic of the software re-mosaic block (560) can be performed in parallel. By parallel processing of the hardware-based re-mosaic and the software-based re-mosaic, the increase in processing time resulting from performing the two types of re-mosaic can be minimized.

[0077] The image signal processor (530) can generate a composite color image frame (505) by synthesizing a plurality of Bayer raw image frames (e.g., a plurality of Bayer raw image frames including #n+5 Bayer raw image frames (503)) generated by the hardware remosaic block (520) and a Bayer base raw image frame (e.g., a Bayer raw image frame (504)) generated by the software remosaic block (560). As more image frames are used for image synthesis, the quality of the composite color image frame (505) can be improved. For example, when a plurality of raw image frames with different exposure times are used, the dynamic range (DR) can be improved. As another example, when a plurality of raw image frames are used in a low-light environment, the signal-to-noise ratio (SNR) can be improved.

[0078] The number of multiple Bayer raw image frames provided by the hardware remosaic block (520) can be adjusted based on the processing time of the software-based remosaic. For example, the number of multiple Bayer raw image frames provided by the hardware remosaic block (520) can be adjusted such that the processing time required to generate the multiple Bayer raw image frames (e.g., the multiple Bayer raw image frames including #n+5 Bayer raw image frames (503)) by the hardware-based remosaic does not exceed the processing time required to generate the Bayer base raw image frame (e.g., the Bayer raw image frame (504) by the software-based remosaic).

[0079] The image signal processor (530) can generate a composite color image frame (505) by synthesizing a Bayer raw image frame according to hardware-based re-mosaic (e.g., a Bayer raw image frame of #n+5) with a Bayer base raw image frame according to software-based re-mosaic (e.g., a Bayer raw image frame (504) of #n+5). When a plurality of Bayer raw image frames according to hardware-based re-mosaic are used, the plurality of Bayer raw image frames can be configured to include a Bayer raw image frame having a frame number that is the same as a frame number of a Bayer base raw image frame according to software-based re-mosaic (e.g., a Bayer raw image frame (504) of #n+5).

[0080] The image signal processor (530) can generate a composite color image frame (505) by synthesizing the entire image area of ​​a Bayer raw image frame (e.g., a Bayer raw image frame (503) of n+5) and the entire image area of ​​a Bayer base raw image frame (e.g., a Bayer raw image frame (504) of #n+5). The Bayer raw image frame, the Bayer base raw image frame, and the composite color image frame (505) can have corresponding sizes (e.g., the same size) to each other.

[0081] FIG. 6A is a diagram exemplarily showing an image preview operation using a dual stream according to one embodiment. Referring to FIG. 6A, a sensing area of ​​an image sensor (610) (e.g., the image sensor (230) of FIG. 2 , the image sensor (310) of FIG. 3 , the image sensor (510) of FIGS. 5A and 5B , the image sensor (710) of FIG. 7 , and the image sensor (810) of FIGS. 8A and 8B ) may include a first area (611) that supports hardware re-mosaic (e.g., the first area (811) of FIG. 8 ) and a second area (612) that does not support hardware re-mosaic (e.g., the second area (812) of FIG. 8 ). A processing area of ​​a hardware re-mosaic block (620) may be designed to be smaller than the processing area of ​​the image sensor (610). In this case, the size of the image frame on which hardware re-mosaic is performed by the hardware re-mosaic block (620) may be smaller than the size of the image frame output by the image sensor (610). The second area (612) may correspond to a processing area of ​​the image sensor (610) that exceeds the processing area of ​​the hardware re-mosaic block (620). Alternatively, the processing area size of the image sensor (610) may be the same as the processing area size of the hardware re-mosaic block (620). In this case, the image sensor (610) may not include the second area (612).

[0082] The sensing signal output from the first region (611) may proceed to the next operation with or without passing through the hardware re-mosaic block (620) (e.g., the hardware re-mosaic block (320) of FIG. 3, the hardware re-mosaic block (520) of FIGS. 5A and 5B, the hardware re-mosaic block (720) of FIG. 7, and the hardware re-mosaic block (820) of FIGS. 8A and 8B). The sensing signal that has passed through the hardware re-mosaic block (620) may be combined with the sensing signal that has not passed through the hardware re-mosaic block (620) to form a dual stream (601). When the sizes of the processing region of the image sensor (610) and the processing region of the hardware re-mosaic block (620) are different, a dual stream (601) according to the first region (611) corresponding to a part of the image sensor (610) may be generated. The signal of the second region (612) may not be output or may be discarded. Conversely, if the size of the processing region of the image sensor (610) and the size of the processing region of the hardware remosaic block (620) are the same, a dual stream (601) according to the first region (611) corresponding to the entire region of the image sensor (610) may be generated.

[0083] The image sensor (610) can generate non-Bayer raw image frames having a size of a first region (611) corresponding to a portion of the image sensor (610) in a crop zoom mode. The crop zoom mode may refer to a mode that provides a zoom function by using the first region (611), which is a crop region of the image sensor (610). Unlike optical zoom, the zoom function of the crop zoom mode may be provided through an increase in resolution due to re-mosaic. The hardware re-mosaic block (620) can perform hardware-based re-mosaic on non-Bayer raw image frames having a size of the first region (611) of the image sensor (610) in the crop zoom mode.

[0084] In crop zoom mode, non-Bayer raw image frames generated by the image sensor (610) using the first region (611) can form a first data stream. In crop zoom mode, a hardware re-mosaic block (620) can perform re-mosaic on the non-Bayer raw image frames of the first region (611) to generate Bayer raw image frames. The Bayer raw image frames can form a second data stream. The first data stream and the second data stream can be combined to form a dual stream (601). For example, the dual stream (601) can alternatively include the first data stream and the second data stream according to a time division of a certain time period.

[0085] The memory (650) (e.g., the memory (130) of FIG. 1, the memory (250) of FIG. 2, the memory (350) of FIG. 3, the memory (550) of FIGS. 5A and 5B, the memory (760) of FIG. 7, the memory (860) of FIGS. 8A and 8B) may include a ring buffer (651) (e.g., the ring buffer (551) of FIGS. 5A and 5B, the ring buffer (861) of FIGS. 8A and 8B). The ring buffer (651) can store non-Bayer raw image frames (e.g., non-Bayer raw image frames (6021 to 6027) of #n+1 to #n+7) and Bayer raw image frames (e.g., Bayer raw image frames (6031 to 6037) of #n+1 to #n+7) based on the dual stream (601) in a state where the non-Bayer raw image frames and Bayer raw image frames are mapped to each other by frame number. FIG. 6A shows an example in which a mapping pair is stored in each memory space of one ring buffer (651). Unlike FIG. 6A, two ring buffers (not shown) may be provided, and non-Bayer raw image frames may be stored in one ring buffer, and Bayer raw image frames may be stored in the other ring buffer.

[0086] An image signal processor (630) (e.g., an image signal processor (260) of FIG. 2, an image signal processor (330) of FIG. 3, an image signal processor (530) of FIGS. 5A and 5B, an image signal processor (730) of FIG. 7, and an image signal processor (830) of FIGS. 8A and 8B) can perform image signal processing on Bayer raw image frames of the second data stream of the dual stream (601) to generate preview color image frames. A display module (640) (e.g., a display module (160) of FIG. 1, a display module (340) of FIG. 3, a display module (540) of FIGS. 5A and 5B, a display module (750) of FIG. 7, and a display module (850) of FIGS. 8A and 8B) can display preview color image frames (e.g., a preview color image frame (604) of #n+5).

[0087] FIG. 6B is a diagram illustrating an image capture operation using a dual stream according to one embodiment. A user's image capture command may be generated in a situation where an image preview operation (e.g., the image preview operation of FIG. 6A) is performed.

[0088] Referring to FIG. 6b, a software remosaic block (660) (e.g., the processor (120) of FIG. 1, the software remosaic block (360) of FIG. 3, the software remosaic block (560) of FIG. 5b, the software remosaic block (770) of FIG. 7, and the software remosaic block (870) of FIG. 8b) may perform software-based remosaic on a non-Bayer raw image frame (e.g., the non-Bayer raw image frame (6025) of #n+5) in the ring buffer (651) according to an image capture command to generate a Bayer raw image frame (e.g., the Bayer raw image frame (605) of #n+5).

[0089] The image signal processor (630) can generate a composite color image frame (606) by synthesizing at least one Bayer raw image frame (e.g., n+5 Bayer raw image frames (6035)) and a Bayer base raw image frame (e.g., n+5 Bayer raw image frames (605)) according to an image capture command. The image signal processor (630) can extract the Bayer raw image frames from a ring buffer (651). When a Bayer raw image frame and a Bayer base raw image frame of the same frame number are composited, a mapping pair of a Bayer raw image frame (e.g., a Bayer raw image frame (6035) of #n+5) and a non-Bayer base raw image frame (e.g., a non-Bayer raw image frame (6025) of #n+5) can be extracted from the ring buffer (651), and the non-Bayer base raw image frame can be used for remosaicing in the software remosaic block (660), and the Bayer raw image frame can be used for image composite. For example, a Bayer raw image frame (e.g., a Bayer raw image frame (6030) of #n+5) mapped to a non-Bayer raw image frame (e.g., a non-Bayer raw image frame (6025) of #n+5) based on a dual stream (601) can be extracted from a ring buffer (650) and used to generate a composited color image frame (606).

[0090] In video recording mode, a user's video snapshot command may be issued. According to the video snapshot command, a snapshot image frame may be generated by compositing at least one Bayer raw image frame (e.g., at least one Bayer raw image frame including Bayer raw image frame #n+5 (6035)) and a Bayer base raw image frame (e.g., Bayer base raw image frame #n+5 (605)). Video may be recorded based on the preview color image frames.

[0091] FIG. 7 is a block diagram illustrating an example of an electronic device according to one embodiment. Referring to FIG. 7, an electronic device (700) (e.g., the electronic device (101) of FIG. 1, the electronic device (300) of FIG. 3) may include an image sensor (710) (e.g., the image sensor (230) of FIG. 2, the image sensor (310) of FIG. 3, the image sensor (510) of FIGS. 5A and 5B, the image sensor (610) of FIGS. 6A and 6B, the image sensor (810) of FIGS. 8A and 8B) that generates a series of non-Bayer raw image frames using a color filter array of a non-Bayer pattern (e.g., the color filter array (400) of FIG. 4). The electronic device (700) may include a hardware re-mosaic block (720) (e.g., hardware re-mosaic block (320) of FIG. 3 , hardware re-mosaic block (520) of FIGS. 5A and 5B , hardware re-mosaic block (620) of FIGS. 6A and 6B , hardware re-mosaic block (820) of FIGS. 8A and 8B ) that performs hardware-based re-mosaic on non-Bayer image frames to generate Bayer image frames. The electronic device (700) may include an image signal processor (730) (e.g., the image signal processor (260) of FIG. 2, the image signal processor (330) of FIG. 3, the image signal processor (530) of FIGS. 5A and 5B, the image signal processor (630) of FIGS. 6A and 6B, and the image signal processor (830) of FIGS. 8A and 8B) that performs image signal processing on image frames using Bayer R code to generate color image frames. The electronic device (700) may include a temporal noise reduction (TNR) block (740) (e.g., the TNR block (840) of FIGS. 8A and 8B) that enhances the color image frames using image information of a previous color image frame of the color image frames to generate preview color image frames.The electronic device (700) may include a memory (760) (e.g., memory (130) of FIG. 1, memory (250) of FIG. 2, memory (350) of FIG. 3, memory (550) of FIGS. 5A and 5B, memory (650) of FIGS. 6A and 6B, memory (860) of FIGS. 8A and 8B) for storing non-Bayer raw image frames and preview color image frames using a ring buffer (e.g., ring buffer (551) of FIGS. 5A and 5B, ring buffer (651) of FIGS. 6A and 6B, ring buffer (861) of FIGS. 8A and 8B). The electronic device (700) may include a software re-mosaic block (770) (e.g., processor (120) of FIG. 1, software re-mosaic block (360) of FIG. 3, software re-mosaic block (560) of FIG. 5B, software re-mosaic block (660) of FIG. 6B, software re-mosaic block (870) of FIG. 8B) that performs software-based re-mosaic on non-Bayer base raw image frames among non-Bayer raw image frames in the ring buffer in response to a video snapshot command during video recording to generate a Bayer base raw image frame. The electronic device (700) may include an image synthesis block (780) (e.g., (880) of FIG. 8B) that synthesizes a base color image frame generated by an image signal processor based on a preview color image frame of preview color image frames in the ring buffer and a Bayer base raw image frame in response to a video snapshot command to generate a synthesized color image frame. The electronic device (700) may include a display module (750) that displays preview color image frames (e.g., display module (160) of FIG. 1, display module (340) of FIG. 3, display module (540) of FIGS. 5A and 5B, display module (640) of FIGS. 6A and 6B, display module (850) of FIGS. 8A and 8B).

[0092] When a video snapshot command is issued, a non-Bayer raw image frame of the non-Bayer raw image frames in the ring buffer corresponding to the preview color image frame of the preview color image frames displayed at the time point when the video snapshot command is issued can be selected as the non-Bayer raw image frame.

[0093] The image sensor (710) generates non-Bayer raw image frames having a size of a portion of the image sensor (710) in a crop zoom mode, and the hardware remosaic block (720) can perform hardware-based remosaic on the non-Bayer raw image frames having a size of a portion of the image sensor (710) in a crop zoom mode.

[0094] In crop zoom mode, a data stream corresponding to non-Bayer raw image frames generated by the image sensor (710) and a data stream corresponding to Bayer raw image frames generated by the hardware remosaic block (720) may be combined to form a dual stream. The dual stream may include a data stream corresponding to non-Bayer raw image frames and a data stream corresponding to Bayer raw image frames alternately according to time divisions of a predetermined time period.

[0095] A ring buffer can store non-Bayer raw image frames and preview color image frames with different frame numbers mapped to each other.

[0096] The image synthesis block (780) can generate a synthesized color image frame by synthesizing a base color image frame and a preview color image frame having the same frame number as the image frame in Bayer basis.

[0097] FIG. 8A is a diagram exemplarily showing an image preview operation using a TNR block according to one embodiment. Referring to FIG. 8A, an image sensor (810) (e.g., the image sensor (230) of FIG. 2 , the image sensor (310) of FIG. 3 , the image sensor (510) of FIGS. 5A and 5B , the image sensor (610) of FIGS. 6A and 6B , and the image sensor (710) of FIG. 7 ) may generate image frames in a non-Bayer pattern using a color filter array (e.g., the color filter array (400) of FIG. 4 ). A sensing area of ​​the image sensor (810) may include a first area (811) that supports hardware re-mosaic (e.g., the first area (611) of FIG. 6 ) and a second area (812) that does not support hardware re-mosaic (e.g., the second area (612) of FIG. 6 ). The processing area of ​​the hardware re-mosaic block (820) may be designed to be smaller than the processing area of ​​the image sensor (810). In this case, the size of an image frame on which hardware re-mosaic is performed by the hardware re-mosaic block (820) may be smaller than the size of an image frame output by the image sensor (810). The second area (812) may correspond to a processing area of ​​the image sensor (810) that exceeds the processing area of ​​the hardware re-mosaic block (820). Alternatively, the processing area size of the image sensor (810) may be the same as the processing area size of the hardware re-mosaic block (820). In this case, the image sensor (810) may not include the second area (812).

[0098] The sensing signal output from the first region (811) may proceed to the next operation with or without passing through the hardware re-mosaic block (820) (e.g., the hardware re-mosaic block (320) of FIG. 3 , the hardware re-mosaic block (520) of FIGS. 5A and 5B , the hardware re-mosaic block (620) of FIGS. 6A and 6B , and the hardware re-mosaic block (720) of FIG. 7 ). The sensing signal that has passed through the hardware re-mosaic block (820) may be combined with the sensing signal that has not passed through the hardware re-mosaic block (820) to form a dual stream. When the sizes of the processing region of the image sensor (810) and the processing region of the hardware re-mosaic block (820) are different, a dual stream may be generated according to the first region (811) corresponding to a part of the image sensor (810). The signal of the second region (812) may not be output or may be discarded. Alternatively, if the size of the processing region of the image sensor (810) and the size of the processing region of the hardware remosaic block (820) are the same, a dual stream may be generated according to the first region (811) corresponding to the entire region of the image sensor (810).

[0099] The image sensor (810) can generate non-Bayer raw image frames having a size of a first region (811) corresponding to a portion of the image sensor (810) in a crop zoom mode. The hardware re-mosaic block (820) can perform hardware-based re-mosaic on the non-Bayer raw image frames having a size of the first region (811) of the image sensor (810) in the crop zoom mode. The non-Bayer raw image frames generated by the image sensor (810) using the first region (811) in the crop zoom mode can form a first data stream. In the crop zoom mode, the hardware re-mosaic block (820) can perform re-mosaic on the non-Bayer raw image frames of the first region (811) to generate Bayer raw image frames. The Bayer raw image frames can form a second data stream. The first data stream and the second data stream can be combined to form a dual stream. For example, a dual stream may include a first data stream and a second data stream alternately according to time divisions of a certain time period.

[0100] An image signal processor (830) (e.g., image signal processor (260) of FIG. 2, image signal processor (330) of FIG. 3, image signal processor (530) of FIGS. 5A and 5B, image signal processor (630) of FIGS. 6A and 6B, image signal processor (730) of FIG. 7) can perform image signal processing on Bayer raw image frames to generate color image frames.

[0101] The TNR block (840) (e.g., the TNR block (740)) can enhance color image frames by using image information of previous color image frames of color image frames to generate preview color image frames (e.g., the preview color image frame (8025) of #n+5). The TNR block (840) may correspond to a hardware module and / or a software module that performs TRN-based image enhancement. The previous color image frame may refer to a color image frame generated previously compared to the color image frames. The image information of the previous color image frame may include information about the previous image enhancement result. The color image frames can be effectively enhanced by using this image information. The image information of the TNR block (840) may be continuously updated according to the generation of color image frames.

[0102] The memory (860) can store non-Bayer raw image frames (e.g., non-Bayer raw image frames (8011 to 8017) of #n+1 to #n+7) and preview color image frames (e.g., preview color image frames (8021 to 8025) of #n_1 to #n+5) using a ring buffer (861) (e.g., ring buffer (551) of FIGS. 5A and 5B, ring buffer (651) of FIGS. 6A and 6B). The ring buffer (861) can store non-Bayer raw image frames and preview color image frames in a state where the non-Bayer raw image frames and the preview color image frames having different frame numbers are mapped to each other. For example, FIG. 8A illustrates, but is not limited to, an example in which there is a difference of two frames between non-Bayer raw image frames (8011 to 8017) and preview color image frames (8021 to 8025) due to a time delay resulting from image processing of the hardware remosaic block (820), the image signal processor (830), and the TNR block (840).

[0103] A display module (850) (e.g., display module (160) of FIG. 1, display module (340) of FIG. 3, display module (540) of FIGS. 5A and 5B, display module (640) of FIGS. 6A and 6B, display module (750) of FIG. 7) can display preview color image frames (e.g., preview color image frame (8025) of #n+5).

[0104] FIG. 8B is a diagram illustrating an image capture operation using a TNR block according to one embodiment. The image preview operation using TNR (e.g., the image preview operation of FIG. 8A) can be performed in video recording mode. A user's video snapshot command can be issued while the image preview operation using TNR is being performed.

[0105] Referring to FIG. 8b, a software remosaic block (870) (e.g., the processor (120) of FIG. 1, the software remosaic block (360) of FIG. 3, the software remosaic block (560) of FIG. 5b, the software remosaic block (660) of FIG. 6b, and the software remosaic block (770) of FIG. 7) may perform software-based remosaic on a non-Bayer raw image frame (e.g., the non-Bayer raw image frame (8015) of #n+5) within a ring buffer (861) according to a video snapshot command during video recording to generate a Bayer raw image frame (e.g., the Bayer raw image frame (803) of #n+5).

[0106] When a video snapshot command is issued, a non-Bayer raw image frame for which zero shutter lag can be achieved may be selected as the non-Bayer base raw image frame. For example, a non-Bayer raw image frame (8015) in the ring buffer (861) corresponding to a preview color image frame (e.g., preview color image frame (8025)) displayed at the time the video snapshot command is issued may be selected as the non-Bayer base raw image frame.

[0107] The image signal processor (830) can generate a base color image frame (e.g., a base color image frame (804) of #n+5) based on a Bayer raw image frame (e.g., a Bayer raw image frame (803) of #n+5). The image synthesis block (880) (e.g., the image synthesis block (780) of FIG. 7) can generate a synthesized color image frame (805) by synthesizing a preview color image frame (e.g., a preview color image frame (8025) of #n+5) and a base color image frame (e.g., a base color image frame (804) of #n+5) within a ring buffer (861) according to a video snapshot command.

[0108] FIG. 9 is a flowchart exemplifying an image processing method according to one embodiment. Operations (910 to 960) of FIG. 9 may be performed sequentially or non-sequentially. For example, the order of operations (910 to 960) may be changed, and / or at least two of operations (910 to 960) may be performed in parallel. Operations (910 to 960) may be performed by at least one component of the electronic device (101).

[0109] Referring to FIG. 9, an image processing method (900) may include an operation (910) of generating a series of non-Bayer raw image frames using a color filter array of a non-Bayer pattern. The image processing method (900) may include an operation (920) of performing hardware-based re-mosaic on the non-Bayer raw image frames to generate Bayer raw image frames. The image processing method (900) may include an operation (930) of storing the non-Bayer raw image frames using a ring buffer. The image processing method (900) may include an operation (940) of performing image signal processing on the Bayer raw image frames to generate preview color image frames. The image processing method (900) may include an operation (950) of performing software-based re-mosaic on non-Bayer base raw image frames among the non-Bayer raw image frames in the ring buffer in response to an image capture command to generate Bayer base raw image frames. The image processing method (900) may include an operation (960) of generating a composite color image frame by synthesizing at least one Bayer raw image frame and a Bayer base raw image frame of Bayer raw image frames according to an image capture command.

[0110] When an image capture command is issued, a non-Bayer raw image frame of the non-Bayer raw image frames in the ring buffer corresponding to the preview color image frame of the preview color image frames displayed at the time point when the image capture command is issued can be selected as the non-Bayer raw image frame.

[0111] A data stream corresponding to non-Bayer raw image frames generated by an image sensor and a data stream corresponding to Bayer raw image frames generated by a hardware remosaic block can be combined to form a dual stream.

[0112] In addition, the descriptions of FIGS. 1 to 8 can be applied to the image processing method (900).

[0113] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0114] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0115] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0116] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0117] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0118] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. An image sensor (230;310;510;610;710;810) that generates a series of non-Bayer image frames using a color filter array of a non-Bayer pattern; A hardware re-mosaic block (320;520;620;720;820) that performs hardware-based re-mosaic on the above non-Bayer raw image frames to generate Bayer raw image frames; Memory for storing the non-Bayer raw image frames using a ring buffer (130;250;350;550;650;760;860); A software remosaic block (360;560;660;770;870) that performs software-based remosaic on non-Bayer base raw image frames among the non-Bayer raw image frames in the ring buffer according to an image capture command to generate Bayer base raw image frames; An image signal processor (260;330;530;630;730;830) for performing image signal processing on the above Bayer raw image frames to generate preview color image frames, and for generating a synthesized color image frame by synthesizing at least one Bayer raw image frame of the above Bayer raw image frames and the above Bayer base raw image frame according to the image capture command; and A display module (160;340;540;640;750;850) for displaying the above preview color image frames. An electronic device comprising:

2. In paragraph 1, When the image capture command occurs, a non-Bayer raw image frame of the non-Bayer raw image frames in the ring buffer corresponding to a preview color image frame of the preview color image frames displayed at the time point when the image capture command occurs is selected as the non-Bayer base raw image frame. Electronic devices.

3. In paragraph 1 or 2, The above image signal processor generating the synthesized color image frame by synthesizing the entire image area of ​​the at least one Bayer raw image frame and the entire image area of ​​the Bayer base raw image frame; Electronic devices.

4. In any one of the clauses 1 to 3, The above image sensor Generate non-Bayer image frames having a size of a portion of the image sensor in crop zoom mode, The above hardware remosaic block is Performing hardware-based re-mosaic on said non-Bayer raw image frames having said size of said portion of said image sensor in said crop zoom mode; Electronic devices.

5. In paragraph 4, In the crop zoom mode, the data stream corresponding to the non-Bayer raw image frames generated by the image sensor and the data stream corresponding to the Bayer raw image frames generated by the hardware remosaic block are combined to form a dual stream. Electronic devices.

6. In paragraph 5, The ring buffer stores the non-Bayer raw image frames and the Bayer raw image frames based on the dual stream, with the non-Bayer raw image frames and the Bayer raw image frames being mapped to each other by frame number, Based on the above dual streams, the Bayer raw image frames of the Bayer raw image frames mapped to the non-Bayer base raw image frames are extracted from the ring buffer and used to generate the synthesized color image frame. Electronic devices.

7. In any one of paragraphs 1 to 6, The above image signal processor Generating the synthesized color image frame by synthesizing the Bayer raw image frame and the Bayer raw image frame of the Bayer raw image frames having the same frame number as the image frame in the Bayer base. Electronic devices.

8. In any one of paragraphs 1 to 7, The above image signal processor Generating the synthesized color image frame by synthesizing a plurality of raw image frames corresponding to at least a portion of the above Bayer raw image frames and the Bayer base raw image frame. Electronic devices.

9. In paragraph 8, The number of the above multiple raw image frames is Based on the processing time of the above software-based remosaic, adjusted; Electronic devices.

10. In any one of the clauses 1 to 9, The hardware-based re-mosaic of the hardware re-mosaic block and the software-based re-mosaic of the software re-mosaic block are performed in parallel. Electronic devices.

11. In any one of clauses 1 to 10, The above image signal processor Generating a snapshot image frame by synthesizing at least one Bayer raw image frame of the Bayer raw image frames and the Bayer base raw image frame according to a video snapshot command during video recording, Electronic devices.

12. An image sensor (230;310;510;610;710;810) that generates a series of non-Bayer image frames using a color filter array of a non-Bayer pattern; A hardware re-mosaic block (320;520;620;720;820) that performs hardware-based re-mosaic on the above non-Bayer raw image frames to generate Bayer raw image frames; An image signal processor (260;330;530;630;730;830) for performing image signal processing on the above Bayer image frames to generate color image frames; A temporal noise reduction (TNR) block (740;840) for improving the color image frames by using image information of previous color image frames of the color image frames to generate preview color image frames; Memory (130;250;350;550;650;760;860) for storing the non-Bayer raw image frames and the preview color image frames using a ring buffer; A software re-mosaic block (360;560;660;770;870) for generating Bayer base raw image frames by performing software-based re-mosaic on non-Bayer base raw image frames among the non-Bayer raw image frames in the ring buffer according to a video snapshot command during video recording; An image synthesis block (780; 880) for generating a synthesized color image frame by synthesizing a base color image frame generated by the image signal processor based on the preview color image frame of the preview color image frames in the ring buffer and the Bayer base raw image frame according to the video snapshot command; and A display module (160;340;540;640;750;850) for displaying the above preview color image frames. An electronic device comprising:

13. In paragraph 12, When the above video snapshot command occurs, a non-Bayer raw image frame of the non-Bayer raw image frames in the ring buffer corresponding to a preview color image frame of the preview color image frames displayed at the time point when the above video snapshot command occurs is selected as the non-Bayer base raw image frame. Electronic devices.

14. In paragraph 12 or 13, The above image sensor Generate non-Bayer image frames having a size of a portion of the image sensor in crop zoom mode, The above hardware remosaic block is Performing hardware-based re-mosaic on said non-Bayer raw image frames having said size of said portion of said image sensor in said crop zoom mode; Electronic devices.

15. In any one of paragraphs 12 to 14, The above ring buffer Storing the non-Bayer raw image frames and the preview color image frames in a state where the non-Bayer raw image frames and the preview color image frames having different frame numbers are mapped to each other. Electronic devices.

Images