Image sensor and imaging apparatus including same

Abstract

In some embodiments, an imaging device may include an image sensor, a display device or a storage device, and an application processor. An image sensor may include: a pixel array including a plurality of pixels, the pixel array providing a pixel signal; a non-Bayer patterned color filter array disposed on the array of pixels; a readout circuit configured to output non-Bayer patterned first image data based on the pixel signal; and an image signal processor configured to rearrange at least a portion of the first image data and output Bayer-patterned second image data. The application processor may be configured to control the image sensor, generate Bayer-patterned third image data based on the Bayer-patterned second image data, and display the preview image on the display device or store the captured image in the storage device based on the third image data.

Description

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Korean Patent Application No. 10-2024-0130844, filed on September 26, 2024, in the Korean Intellectual Property Office, and Korean Patent Application No. 10-2024-0160580, filed on November 12, 2024, in the Korean Intellectual Property Office, the disclosures of which are incorporated herein by reference in their entireties. TECHNICAL FIELD

[0002] Embodiments disclosed relate to an image sensor and an imaging device including the same, and more particularly, to an image sensor and an imaging device for generating a captured image without experiencing shutter lag. BACKGROUND

[0003] An image sensor is a device for converting an optical signal into an electrical signal.

[0004] A pixel with a complementary metal-oxide semiconductor (CMOS) image sensor (CIS) can be used to obtain data related to a captured image based on electrical signals generated by CMOS transistors (also located on the pixel).

[0005] In-sensor zoom functionality has been used in image sensors and imaging devices. When performing in-sensor zoom functionality, an image sensor can crop a requested zoomed portion of sensed image data and output the cropped image data. However, when performing high resolution in-sensor zoom imaging using current technology, shutter lag occurs during the capturing of the cropped image data, resulting in a decrease in the quality, resolution, and timing accuracy of the captured image. This occurs at least in part because a pre-stored image cannot be used to generate the captured image due to required changes between image sensor modes. SUMMARY

[0006] Example embodiments include an image sensor and an imaging device for generating a captured image without experiencing shutter lag.

[0007] According to an example embodiment, an imaging device includes an image sensor, a display device or a storage device, and an application processor. The image sensor can include a pixel array including a plurality of pixels, the pixel array providing pixel signals; a non-Bayer patterned color filter array disposed on the pixel array; a readout circuit configured to output first image data that is non-Bayer patterned based on the pixel signals; and an image signal processor configured to rearrange at least a portion of the first image data and output second image data that is Bayer patterned. The application processor can be configured to control the image sensor, generate third image data that is Bayer patterned based on the second image data that is Bayer patterned, and display a preview image on the display device or store a captured image in the storage device based on the generated third image data that is Bayer patterned.

[0008] According to another example embodiment, an image sensor includes a pixel array including a plurality of pixels, the pixel array providing pixel signals; a non-Bayer patterned color filter array disposed on the pixel array; a readout circuit configured to output first image data that is non-Bayer patterned based on the pixel signals; and an image signal processor configured to receive a zoom command from an application processor, receive a preview command from the application processor, and in response to the zoom command and the preview command, rearrange at least a portion of the first image data that is non-Bayer patterned and output second image data that is Bayer patterned.

[0009] According to yet another example embodiment, an electronic device includes an image sensor, a display device or a storage device, and an application processor. The application processor can be configured to control the image sensor, receive first image data that is Bayer patterned from the image sensor, remove one or more artifacts from the first image data that is Bayer patterned to generate second image data that is Bayer patterned, and display a preview image based on the second image data that is Bayer patterned on the display device or store a captured image based on the second image data that is Bayer patterned in the storage device. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 FIG. 1 is a diagram illustrating an imaging device according to an example embodiment.

[0011] Figure 2 FIG. 2 is a diagram illustrating an image sensor according to an example embodiment.

[0012] Figure 3 FIG. 3 is a diagram illustrating a four-pattern color filter unit according to an example embodiment.

[0013] Figure 4A and Figure 4BThese are illustrations of intra-sensor scaling operations according to related technologies and intra-sensor scaling operations according to disclosed embodiments, respectively.

[0014] Figure 5 This is a block diagram illustrating an exemplary configuration of an image signal processor according to a disclosed embodiment.

[0015] Figure 6 This illustrates an embodiment according to the disclosed embodiments. Figure 5 A diagram illustrating an exemplary arrangement of color interpolation operations performed during the remosaic circuitry.

[0016] Figure 7 This illustrates an example embodiment. Figure 5 A diagram illustrating the operation of the rearrangement circuit.

[0017] Figure 8 This is a diagram illustrating an application processor according to an example embodiment.

[0018] Figure 9 This is a diagram illustrating a frame buffer of a memory device according to a disclosed embodiment.

[0019] Figure 10 This is a diagram illustrating the preview image generation operation of an imaging device according to an example embodiment.

[0020] Figure 11 This is a diagram illustrating the captured image generation operation of an imaging device according to an example embodiment.

[0021] Figure 12 This is a diagram illustrating the image capture generation operation according to related technologies.

[0022] Figure 13 This illustrates an example embodiment. Figure 1 A diagram illustrating the image capture and generation process of an imaging device.

[0023] Figure 14 This illustrates an example embodiment. Figure 1 A diagram illustrating the video frame generation operation of an imaging device.

[0024] Figure 15 This is a block diagram of an imaging device according to an example embodiment.

[0025] Figure 16 This is a flowchart illustrating the operation of an imaging device according to an example embodiment. Detailed Implementation

[0026] In the following description, exemplary embodiments will be illustrated with reference to the accompanying drawings.

[0027] Throughout the specification, the terms "circuit," "unit," and "block" are used to distinguish components, and each component can be implemented as hardware, software, or a combination of hardware and software. The terms "circuit," "unit," and "block" do not refer to any particular type, such as hardware and / or software.

[0028] Throughout the specification, the term "Bayer-patterned image data" refers to image data having an output form based on a Bayer-patterned pixel array, and "non-Bayer-patterned image data" refers to image data having an output form based on a non-Bayer-patterned pixel array. In particular, "Tetra-patterned image data" refers to image data based on a Tetra-patterned pixel array, and "Nona-patterned image data" refers to image data based on a Nona-patterned pixel array.

[0029] For example, Bayer-patterned image data in a matrix form corresponding to one frame can include a plurality of 2x2 matrix-form image data groups. In this case, two non-adjacent image data within each group can correspond to pixels including a green color filter, and the remaining two image data can correspond to pixels including a red color filter and a blue color filter, respectively.

[0030] Meanwhile, the term "non-Bayer-patterned image data" can refer to image data other than Bayer-patterned image data. For example, Tetra-patterned image data in a matrix form corresponding to one frame can include a plurality of image data groups each having a 2x2 matrix form following a Bayer pattern arrangement.

[0031] Figure 1 is a diagram illustrating an imaging device 10 according to an example embodiment.

[0032] As Figure 1 illustrated in , the imaging device 10 can receive the same image data ID2 from the image sensor 100 when generating a preview image and when generating a captured image. The imaging device 10 can generate a captured image having a zero shutter lag.

[0033] Referring to Figure 1 , the imaging device 10 can include the image sensor 100, an application processor (AP) 200, and a memory device 300.

[0034] In some embodiments, and as Figure 1 illustrated, the image sensor 100 can include a pixel array 110 and an image signal processor 130.

[0035] In some embodiments, as Figure 1As shown, the pixel array 110 can include a plurality of pixels. The plurality of pixels can generate a pixel signal based on a light signal received through a lens and a color filter. The pixel array 110 can output first image data ID1 based on the pixel signal.

[0036] In some embodiments, as Figure 1 As shown, the pixel array 110 can output the non-Bayer-patterned first image data ID1. In an example embodiment, a color filter array can be disposed on the pixel array 110 in a non-Bayer pattern rather than, for example, a Bayer pattern. For example, the color filter array can be a non-Bayer-patterned color filter array.

[0037] For example, in the case of a Bayer-patterned color filter array, color filters of different colors can be disposed in at least some adjacent pixels of the plurality of pixels. As another example, in the case of a non-Bayer-patterned color filter, color filters of the same color can be disposed in at least some adjacent pixels of the plurality of pixels.

[0038] In some embodiments, and as Figure 1 Further shown, the image signal processor 130 can output second image data ID2 obtained by image processing the non-Bayer-patterned first image data ID1 received from the pixel array 110. The second image data ID2 can be transmitted to the application processor 200.

[0039] In an example embodiment, the image signal processor 130 can generate the second image data ID2 by rearranging the non-Bayer-patterned first image data ID1 (e.g., reconstructing a full-resolution color image therefrom).

[0040] In some embodiments, the application processor 200 can control the image sensor 100 and the memory device 300. For example, the application processor 200 can support various applications such as a user application, a personal computer (PC) application, or a mobile application. In some embodiments, the application processor 200 can control the image sensor 100 and the memory device 300 according to a user request and / or an application request.

[0041] In some embodiments, the application processor 200 can temporarily store the second image data ID2 in the memory device 300. In some embodiments, the application processor 200 can generate third image data ID3 obtained by image processing the second image data ID2. In some embodiments, the application processor 200 can generate a preview image or a captured image using the third image data ID3.

[0042] In example embodiments, the application processor 200 can remove artifacts from the second image data ID2 that is patterned in a Bayer pattern and generate third image data ID3 that is patterned in a Bayer pattern. In some embodiments, the application processor 200 can demosaicing (e.g., reverse or de-rearrange) the second image data ID2 that is patterned in a Bayer pattern or the third image data ID3 that is patterned in a Bayer pattern and generate a preview image or a captured image.

[0043] In some embodiments, the preview image can be generated in response to a camera on request from a user and / or an application. In some embodiments, the application processor 200 can display the preview image on a display device.

[0044] In some embodiments, the captured image can be generated in response to a capture request from a user and / or an application. In some embodiments, the capture request can be a request to store an image in a storage device.

[0045] According to example embodiments, the application processor 200 can be configured not to request a change in image sensor operation (e.g., operation or mode of the image sensor 100) even when a captured image needs to be generated during generation of a preview image.

[0046] According to the above example embodiments, the operation of the image sensor 100 can be the same for both generation of a preview image and a captured image. In other words, the image sensor 100 does not need to change its operation(s) associated with generation of a captured image while performing operation(s) such as, for example, sending image data for generation of a preview image (and vice versa).

[0047] For example, the image signal processor 130 can rearrange and output the first image data ID1 that is not patterned in a Bayer pattern both when generating a preview image and when generating a captured image.

[0048] In example embodiments, the image sensor 100 can not change its operation(s) associated with generation of a preview image and / or its operation(s) associated with generation of a captured image, and the application processor 200 can receive the second image data ID2 that is patterned in a Bayer pattern both when generating a preview image and when generating a captured image from the image sensor 100.

[0049] In some embodiments, the application processor 200 can generate a capture image using third image data ID3 for the preview image stored in the memory device 300 in response to the capture request. The plurality of third image data ID3 used to generate the capture image can be the same as the plurality of image data stored in the memory device 300 (e.g., prior to receiving the capture request). As a result, the imaging device 10 can generate the capture image using the preview image data; thus, the capture image can have a minimal shutter lag or zero shutter lag because the operation (or mode) of the image sensor does not need to change (e.g., based on whether a preview image or a capture image is desired).

[0050] In example embodiments, the application processor 200 can generate the preview image and / or the capture image in an in-sensor zoom mode.

[0051] The in-sensor zoom mode can be a mode in which zoomed image data is output based on pixel signals generated from the pixel array 110 without physically moving the lens.

[0052] For example, in an optical zoom mode, the focal length of the lens and the field of view (FOV) of the camera can be changed by physically moving the lens, and image data based on the changed FOV can be output from the image sensor 100. In the in-sensor zoom mode, the image sensor 100 can crop a portion of the first image data ID1 to generate the second image data without adjusting the FOV of the camera.

[0053] As an example, in related art, the image signal processor can perform different image processing operations when generating a preview image than when generating a capture image. Thus, a mode change of the image sensor can occur based on whether the imaging device is generating a preview image or a capture image. Similarly, in related art, a sensor mode change can occur when performing an in-sensor zoom function (e.g., when viewing a preview image) to generate a capture image (upon receiving a capture request).

[0054] Thus, in related art, the occurrence of a mode change of the image sensor can take time. Thus, a shutter lag can occur when generating a capture image. Additionally, when the image sensor changes from a mode for generating a preview image to a mode for generating a capture image, the application processor can not be able to use pre-stored image data, for example, to generate a capture image in the same mode as the mode used to generate the preview image. Furthermore, in related art, it can be difficult to generate a high-quality capture image when using the same mode as the mode used to generate the preview image.

[0055] In some embodiments, and further with reference to Figure 1, the image sensor 100 can output the second image data ID2 patterned with a Bayer in the in-sensor zoom mode when generating a preview image and when generating a captured image. In the in-sensor zoom mode, the application processor 200 can generate a captured image using the image data for the preview image and stored in the frame buffer 310. As a result, the imaging device 10 can generate a captured image without shutter lag (e.g., with zero shutter lag) in the in-sensor zoom mode.

[0056] Figure 2 is a diagram illustrating an image sensor 100 according to an example embodiment.

[0057] In some embodiments, and with reference to Figure 2 , the image sensor 100 can include a pixel array 110, a readout circuit 120, an image signal processor 130, a row driver 140, and a timing controller 150.

[0058] In some embodiments, the pixel array 110 can include a plurality of pixels PX. The plurality of pixels PX can be configured, for example, in a matrix. The pixel array 110 can receive a plurality of pixel driving signals CSn, such as a selection signal, a reset signal, and a transfer control signal, from the row driver 140. The pixel array 110 can operate under the control of the received pixel driving signals CSn.

[0059] In some embodiments, each of the plurality of pixels PX can convert a light signal into an electrical signal using at least one photoelectric conversion element.

[0060] In some embodiments, the pixel array 110 can provide a pixel signal PS output from the plurality of pixels PX to the readout circuit 120 through a plurality of column lines CLm.

[0061] In some embodiments, the photoelectric conversion element can be a photodiode PD. The photodiode PD can refer to, for example, a photoelectric conversion element that generates a charge proportional to a light signal incident on each pixel and accumulates the generated charge. The photoelectric conversion element can be, for example, a photodiode PD, a photogalvanic cell, a photogate, a pinned photodiode PPD, a partial-pinned photodiode, an organic photodiode OPD, a quantum dot QD, or a combination thereof.

[0062] Although the example embodiments describe a photodiode PD as a photoelectric conversion element, other photoelectric conversion elements not limited to those described above can also be used. Thus, the photoelectric conversion element is not limited to a photodiode PD.

[0063] In some embodiments, the pixel array can include a color filter array. The color filter array can include a plurality of color filters. In an example embodiment, the color filter array can be a non-Bayer pattern color filter array. In some embodiments, each of the plurality of color filters can be disposed in the color filter array in a non-Bayer pattern.

[0064] For example, the same color filter can be disposed in some adjacent pixels of the plurality of pixels. In some embodiments, a color filter group can include color filters of the same color. In some embodiments, a color filter unit can include a plurality of color filter groups. In some embodiments, the color filters included in each of the plurality of color filter groups can be arranged in an M x N matrix, where M and N are positive integers.

[0065] In some embodiments, the readout circuit 120 can include an analog-to-digital converter. The analog-to-digital converter of the readout circuit 120 can convert the pixel signal PS into a digital signal and output the digital signal. For example, the analog-to-digital converter can sample the pixel signal using correlated double sampling and convert the sampled pixel signal into first image data ID1, e.g., a digital signal. To this end, in some embodiments, a correlated double sampler CDS can also be disposed before the analog-to-digital converter.

[0066] In some embodiments, the readout circuit 120 can convert the pixel signal PS of the pixel array 110 into a digital signal and output first image data ID1.

[0067] In some embodiments, the row driver 140 can select a single row of the pixel array 110 under the control of the timing controller 150. The row driver 140 can generate a selection signal CSn to select one row among the plurality of rows of the pixel array 110. In some embodiments, the row driver 140 can activate the pixels PX corresponding to the selected row. The pixel signal PS of the pixels PX of the selected row can be transmitted to the analog-to-digital converter of the readout circuit 120.

[0068] In some embodiments, the timing controller 150 can control the pixel array 110, the row driver 140, the readout circuit 120, and / or the image signal processor 130. In some embodiments, the timing controller 150 can provide a timing control signal TC to the row driver 140. In some embodiments, the timing controller 150 can provide a reference code RC to the readout circuit 120.

[0069] According to an example embodiment, the readout circuit 120 can output first image data ID1 that is non-Bayer patterned.

[0070] In some embodiments, image signal processor 130 can include a rearrangement circuit 133. Rearrangement circuit 133 can rearrange non-Bayer-patterned image data and output Bayer-patterned second image data ID2. In some embodiments, image signal processor 130 can rearrange non-Bayer-patterned image data and output Bayer-patterned second image data ID2 both when generating a preview image and when generating a captured image.

[0071] Figure 3 is a diagram illustrating an exemplary four-pattern color filter unit according to embodiments of the disclosure, one or more of which can include a non-Bayer-patterned color filter array.

[0072] In some embodiments, a color filter array can include a plurality of color filter groups and / or color filters.

[0073] In an example embodiment, and with reference to Figure 3 , color filter groups CFG1, CFG2, CFG3, and CFG4 can each contain four color filters CF having a single color. Color filter unit CFU can include a plurality of color filter groups CFG1, CFG2, CFG3, and CFG4. With reference to Figure 1 The described color filter array may, for example, include at least one color filter unit CFU.

[0074] In an example embodiment, a pixel array can include a plurality of four- pattern color filter units CFU repeatedly disposed in the pixel array (e.g., Figure 1 and Figure 2 pixel array 110) of Figure 3 In an example embodiment, a pixel array can include a plurality of four- pattern color filter units CFU repeatedly disposed in the pixel array (e.g.,

[0075] In some embodiments, and with further reference to Figure 3 Each of the four-pattern color filter units CFU can include four-pattern color filter groups CFG1, CFG2, CFG3, and CFG4, and each four-pattern color filter group CFG1, CFG2, CFG3, and CFG4 can include one color filter CF having the same color. In some embodiments, a color filter unit CFU can include two green color filter groups G, a single red color filter group R, and a single blue color filter group B, as shown in the example of Figure 3 In some embodiments, color filter groups having different colors can be disposed adjacent to each other. In some embodiments, color filter groups having the same color can be disposed adjacent to each other.

[0076] In addition to referring to Figure 3The four-pattern color filter described above or instead of the four-pattern color filter, the non-Bayer patterned color filter array according to some embodiments can be based on a nine (e.g., nine-cell) patterned color filter or a four-square (e.g., sixteen (Hexadeca) Bayer) patterned color filter. However, these are merely examples, and the disclosed embodiments are not limited thereto.

[0077] According to example embodiments, a pixel array (e.g., the pixel array 110 of Figure 1 and 2 may output first image data ID1 based on one of a four-pattern, a nine-pattern, and a four-square pattern. In some embodiments, an image signal processor (e.g., the image signal processor 130 of Figure 1 and 2 may rearrange the first image data ID1 based on one of a four-pattern, a nine-pattern, and a four-square pattern.

[0078] Figure 4A is a diagram illustrating an in-sensor zoom operation ISZ1 according to the related art, and Figure 4B is a diagram illustrating an exemplary in-sensor zoom operation ISZ2 according to the disclosed embodiments. Referring to Figure 4B The in-sensor zoom operation ISZ2 described above can be performed in an imaging device (e.g., the imaging device 10 of Figure 1 .

[0079] Figure 4A The image sensor shown in Figure 4A may include a low-resolution pixel array PA1. In Figure 4A the in-sensor zoom operation ISZ1 of the image sensor, during the upscaling operation, the cropped image has a significantly low resolution. Thus, the cropped image CI in the in-sensor zoom operation ISZ1 performed according to the related art is limited in its ability to improve the image quality, even after post-processing, for example. An application processor according to the related art can upscale the cropped image CI (e.g., perform upscaling on the cropped image CI), and generate an upscaled cropped image FI1 in an attempt to recover the image quality. However, the image quality recovered in the upscaled cropped image FI1 is still limited.

[0080] In contrast to the related art, and referring to Figure 4B , a pixel array according to the disclosed embodiments (e.g., the pixel array 110 of Figure 1 and Figure 2 may include one hundred million or more pixels. Thus, in Figure 4B , a cropped image FI2 captured during the in-sensor zoom operation ISZ2 of the pixel array PA2 has a higher resolution than the cropped image CI captured during the in-sensor zoom operation ISZ1 of the pixel array PA1 of Figure 4AThe cropped image CI generated in the intra-sensor scaling operation ISZ1 has a higher resolution. As a result, the cropped image FI2 in the intra-sensor scaling operation ISZ2 of pixel array PA2 can be used as a preview image and / or captured image after simple image processing, and further, no magnification is needed to improve or restore the cropped or captured image because the cropped image FI2 can have a higher resolution than... Figure 4A The enlarged, cropped image FI1 has a higher resolution.

[0081] Figure 5 This is a block diagram illustrating an exemplary configuration of an image signal processor 130 according to a disclosed embodiment. Figure 5 The image signal processor 130 can correspond to Figure 1 and Figure 2 Image signal processor 130. (Refer to...) Figure 1 , Figure 2 and Figure 5 Image signal processor 130 is described. (The following text omits references to the above.) Figure 1 and Figure 2 The provided description is redundant or a similarly detailed description.

[0082] According to such Figure 5 The image signal processor 130 of the illustrated example embodiment may include an image merging circuit 131, an image cropping circuit 132, and a rearrangement circuit 133. However, this is merely an example, and the image signal processor 130 is not limited thereto, and may also include circuitry for processing noise reduction, white balance, color correction, sharpening, or other features. In some embodiments, the image signal processor 130 may not include the image merging circuit 131.

[0083] In an example embodiment, the image signal processor 130 may process image data in different ways according to commands received from the application processor. For example, Figure 1 and Figure 2 The image sensor 100 can receive preview commands and / or zoom commands from the application processor 200.

[0084] Application processor 200 may, for example, send a preview command to image sensor 100 in response to a user's request to turn on the camera.

[0085] In an example embodiment, when no scaling operation is being performed, the image merging circuit 131 of the image signal processor 130 may merge first image data in response to a preview command. For example, the image merging circuit 131 may merge image data generated from multiple adjacent pixels. The image merging circuit 131 may merge the first image data ID1 to reduce noise or improve sensitivity, for example, in low-light (e.g., dimmed or low-brightness) environments.

[0086] In example embodiments, the image data generated from the pixels can be merged by the image merging circuit 131 in response to a preview command when a zoom operation is not being performed. In some embodiments, the image merging circuit 131 can not merge the image data.

[0087] In some embodiments, the application processor 200 can send a zoom command to the image sensor 100 in response to a zoom-in request by a user. The zoom command can be sent, for example, during a preview operation.

[0088] In example embodiments, and with reference to Figure 5 , the image cropping circuit 132 can crop a portion of the first image data ID1 in response to the zoom command. For example, the image cropping circuit 132 can only crop the first image data ID1 that corresponds to a region of the first image data ID1 after being zoomed-in by the user (e.g., upon receiving the zoom command).

[0089] Further with reference to Figure 5 example embodiments, the rearrangement circuit 133 can rearrange the first image data ID1 output from the image merging circuit 131 or from the image cropping circuit 132 and output the second image data ID2.

[0090] For example, the rearrangement circuit 133 can rearrange the non-Bayer patterned first image data ID1 based on a non-Bayer patterned color filter array, convert the rearranged version of the first image data ID1 to the Bayer patterned second image data ID2, and output the rearranged version.

[0091] In example embodiments, the rearrangement circuit 133 can rearrange the first image data ID1 based on interpolation. For example, the rearrangement circuit 133 can rearrange the first image data ID1 based on green interpolation and / or chroma (UV) interpolation.

[0092] Figure 6 is a diagram illustrating an example color arrangement used during example color interpolation operations performed during a rearrangement operation of the rearrangement circuit 133 of the image sensor 100 according to disclosed embodiments. Figure 5 illustrates an example color arrangement that can be used to generate image data for the green channel. Using a similar pattern while changing the respective colors, the rearrangement circuit 133 can generate image data, for example, for each of the green channel and the blue channel. Figure 6

[0093] Operations in which the rearrangement circuit 133 generates the green channel based on interpolation will be described with reference to Figure 6 The operations associated with Figure 5 may be performed by the rearrangement circuit 133, for example, during a rearrangement processing operation of the non-Bayer patterned first image data ID1 of Figure 6 . ​

[0094] The first image data ID1 may be based on the color arrangement of the color filter array. In an exemplary embodiment, the first image data ID1 may include information about only one color for each pixel. In such an embodiment, the rearrangement circuit 133 may interpolate the information about the individual colors to generate information about the other colors.

[0095] In the following text, in reference Figure 6 In the described embodiments, multiple first image data corresponding to a group of color filters for a specific color in the color filter array will be referred to as an image data group (e.g., a red image data group, a green image data group, or a blue image data group). For example, Figure 6 The diagram shows a red image data group R1 involving red image data at the center, and green image data groups G1, G2, G3, and G4 surrounding the red image data group R1, involving multiple green image data. Furthermore, Figure 6 This illustrates how specific numbers can be used, for example, to distinguish image data corresponding to each pixel in each group of image data. For example, such as... Figure 6 As shown, G41 can be image data corresponding to the first pixel of image data group G4. Image data corresponding to a specific pixel or group of pixels may be referred to as pixel data in this document.

[0096] Further reference Figure 6 Four 2×2 arrays of green image data groups G1, G2, G3, and G4 can be arranged adjacent to a 2×2 array of red image data group R1. By interpolating multiple pixel data from the green image data groups G1, G2, G3, and G4, information corresponding to the green components of multiple pixel data from the red image data group R1 can be generated.

[0097] For example, and refer to Figure 6 Information corresponding to the green component of pixel data R12 in the red image data group R1 can be generated by interpolating pixel data G13, G14, G41, and G43. Weights can be applied to the pixel data, and the applied weights can vary depending on, for example, the positions of pixel data G13, G14, G41, and G43 and the position of pixel data R12. Furthermore, depending on the interpolation method, multiple pixel data points other than pixel data G13, G14, G41, and G43 can be used or alternatively used. Moreover, the method is not limited to a specific or single interpolation method.

[0098] In some embodiments, when the same operation is performed on all pixel data of the red image data group R1, information corresponding to the green component of the red image data group R1 can be generated. The same operation can be performed on the remaining red image data groups, blue image data groups, or other color data groups. As a result, information corresponding to the green component of the entire image data (e.g., the entire pixel array) can be generated.

[0099] Figure 7 This illustrates an example embodiment. Figure 5 A diagram illustrating the operation of the rearrangement circuit 133.

[0100] In the example embodiment, and as Figure 7 As shown, the rearrangement circuit 133 can output Bayer-patterned second image data ID2. For example, the rearrangement circuit 133 can use non-Bayer-patterned first image data ID1 to generate full green data (full G data) through green interpolation, for example, as shown in the reference. Figure 6 As described, the rearrangement circuit 133 can generate chromaticity information through chromaticity interpolation of all-green data (all-G data), blue data (B data), and red data (R data). The rearrangement circuit 133 can generate all-red data (all-R) and all-blue data (all-B) based on, for example, all-green data (all-G data) and chromaticity information.

[0101] In an example embodiment, the rearrangement circuit (e.g., Figure 5 The rearrangement circuit 133 can generate luminance and chromaticity information based on information corresponding to the green component within the entire image data. For example, luminance and chromaticity information can be generated based on the YUV color space. In some embodiments, the rearrangement circuit 133 can generate luminance information Y based on all green data (all G data). In some embodiments, the rearrangement circuit 133 can generate chromaticity information (UV interpolation) based on all green data (all G data), blue data (B data), and red data (R data). In some embodiments, the rearrangement circuit 133 can generate all red data (all R) and / or all blue data (all B) based on all green data (all G data) and chromaticity information (e.g., the result of UV interpolation).

[0102] In some embodiments, such as Figure 7 As further shown, the rearrangement circuit 133 can perform Bayer sampling on the final RGB data (all R, all G, and all B) of the first image data ID1 and generate a Bayer patterned second image data ID2.

[0103] Figure 8 This is a diagram illustrating an application processor 200 according to an example embodiment. Figure 8 Application processor 200 may correspond to, for example Figure 1 Application processor 200. (Refer to...)Figure 1 and Figure 8 An application processor 200 is described. The following omits a redundant or similar detailed description provided above for Figure 1 the application processor 200.

[0104] As shown in Figure 8 FIG. 1, the application processor 200 according to an example embodiment can include an artifact removal circuit 210, a first image processing circuit 220, and a second image processing circuit 230.

[0105] The application processor 200 can receive second image data ID2 from a Bayer pattern of an image sensor (e.g., the image sensor 100). The application processor 200 can process the second image data ID2 and display the processed second image data ID2 on a display device or store the processed second image data ID2 in a storage device. The application processor 200 can perform image processing to improve image quality of the second image data ID2. The application processor 200 according to an example embodiment can perform processing such as artifact removal, noise reduction, white balance, color correction, or sharpening on the second image data ID2. Figure 1 In some embodiments, the artifact removal circuit 210 can include one or more of a neural processor, a graphics processor, a logic circuit, a field programmable gate array (FPGA), and an artificial intelligence (AI) accelerator.

[0106] In some embodiments, the artifact removal circuit 210 can remove artifacts according to a deep learning-based neural network.

[0107] For example, the artifact removal circuit 210 can input the Bayer-patterned second image data ID2 to the deep learning-based neural network and output third image data ID3 in which artifacts are removed from the Bayer-patterned image data. For example, the deep learning-based neural network can be trained with training data including labeled Bayer-patterned image data including artifacts and labeled Bayer-patterned image data from which artifacts are removed.

[0108] In an example embodiment, the artifact removal circuit 210 can remove artifacts of the Bayer-patterned second image data ID2 based on the deep learning-based neural network and output the third image data ID3 having the same resolution as the second image data ID2. Accordingly, the artifact removal circuit 210 can remove artifacts of the second image data ID2 at high speed, and the artifact removal circuit can operate in real time. As a result, the application processor 200 can generate captured images and video frames in real time using the Bayer-patterned second image data ID2.

[0109]

[0110] ​In some embodiments, the first image processing circuit 220 can include a gain adjustment circuit, a contrast adjustment circuit, or other circuit to correct the third image data ID3. For example, the first image processing circuit 220 can perform image gain adjustment, contrast adjustment, or other adjustment of the third image data ID3, and output a preview image. The preview image can be displayed on, for example, a display device (e.g., the display device 400).

[0111] In some embodiments, the second image processing circuit 230 can include a filter, a gain adjustment circuit, a noise reduction circuit, a contrast adjustment circuit, or other circuit to correct the third image data ID3. The second image processing circuit 230 can improve image quality, for example, based on each pixel data of the input third image data ID3. The second image processing circuit 230 can perform, for example, image gain adjustment, contrast adjustment, noise reduction, or other adjustment or reduction of the third image data ID3, and output a captured image. The captured image can be stored in, for example, a storage device (e.g., the memory device 500).

[0112] In an example embodiment, the second image processing circuit 230 can store a captured image obtained by converting the third image data ID3 into an image format such as Joint Photographic Experts Group (JPEG) in a storage device. However, this is merely an example, and the disclosed embodiments are not limited thereto. For example, the second image processing circuit 230 can convert the third image data ID3 into an image format such as Graphics Interchange Format (GIF), Bitmap (BMP), or Portable Network Graphics (PNG).

[0113] The storage device (or the memory device) can include a non-volatile memory such as a flash memory, a ferroelectric random access memory (FRAM), or a magnetoresistive random access memory (MRAM).

[0114] Figure 9 is a diagram showing an example frame buffer 310 of an example memory device 300 according to the disclosed embodiments. The frame buffer 310 can correspond to, for example, Figure 1 the frame buffer 310 of the

[0115] The memory device 300 can include, for example, a volatile memory device such as a dynamic random access memory (DRAM), a static random access memory (SRAM), or a video random access memory (VRAM).

[0116] Referring to Figure 9 , the frame buffer 310 can store and output at least one piece of image data. Figure 9 An example frame buffer 310 in which image data corresponding to five frames is stored is shown. It will be understood that example embodiments are not limited thereto.

[0117] The frame buffer 310 can store at least one frame of image data (e.g., first image data ID1, second image data ID2, or third image data ID3) received from the image sensor 100. Figure 1 The frame buffer 310 can store at least one frame of second image data ID2 received from the image sensor 100 within a predetermined time period from a current time. In some embodiments, the frame buffer 310 can store at least one frame of third image data ID3 output by the artifact removal circuit 210 within a predetermined time period from the current time. Figure 8 The frame buffer 310 can store at least one frame of second image data ID2 received from the image sensor 100 within a predetermined time period from a current time. In some embodiments, the frame buffer 310 can store at least one frame of third image data ID3 output by the artifact removal circuit 210 within a predetermined time period from the current time.

[0118] In example embodiments, the frame buffer 310 can store at least one frame of second image data ID2 received from the image sensor 100 within a predetermined time period from a current time. In some embodiments, the frame buffer 310 can store at least one frame of third image data ID3 output by the artifact removal circuit 210 within a predetermined time period from the current time.

[0119] Figure 9 The image data of the current time is shown as image data N, and image data N-1, image data N-2, image data N-3, and image data N-4 are shown in order of image data input with reference to the current time image data N, where N-4 is the oldest stored image data and N-1 is the most recent stored image data with reference to image data N. Figure 9 An example is shown in which image data N (image data input) of the current time is input and image data N-5 is deleted from the frame buffer 310.

[0120] The frame buffer 310 can operate, for example, in a first-in-first-out (FIFO) manner. The frame buffer 310 can store a predetermined number of image data, and when new image data is input, the image data that was first input to the frame buffer 310 can be deleted.

[0121] Figure 10 is a diagram showing a preview image generation operation of the imaging device 10 according to an example embodiment. Figure 10 The imaging device 10 of Figure 1 The imaging device 10 of

[0122] Referring to Figure 10 The pixel array 110 of the image sensor 100 can generate first image data ID1 that is not Bayer-patterned.

[0123] For example, when a user turns on a camera, the image sensor 100 can receive a preview command from the application processor 200. The image signal processor 130 can rearrange the first image data ID1 that is not Bayer-patterned in response to the preview command to generate second image data ID2 that is Bayer-patterned. In some embodiments, the image signal processor 130 can merge the first image data ID1 that is not Bayer-patterned and rearrange the merged first image data ID1.

[0124] Further referring to Figure 10In response to the zoom-in request, the image sensor 100 can output the first image data ID1 in a non-Bayer pattern. The image signal processor 130 can receive the first image data ID1 from the pixel array 110 of the image sensor 100. The image signal processor 130 can crop and rearrange the first image data ID1 in a non-Bayer pattern to generate second image data ID2 in a Bayer pattern.

[0125] In some embodiments, when the zoom-in command is received during the preview operation of the image sensor, the application processor 200 can receive the second image data ID2 in a Bayer pattern from the image sensor 100. The artifact removal circuit 210 of the application processor 200 can remove artifacts from the second image data ID2 and generate third image data ID3.

[0126] In some embodiments, the first image processing circuit 220 of the application processor 200 can receive the third image data ID3. The first image processing circuit 220 can process the input third image data ID3 to generate a preview image and display the preview image on a display device (e.g., the display device 400). Figure 10

[0127] Figure 11 is a diagram illustrating a captured image generation operation of the imaging device 10 according to an example embodiment. Figure 11 The imaging device 10 of Figure 1 may correspond to the imaging device 10 of Figure 10 The descriptions provided above with reference to may be redundant or similar.

[0128] Referring to Figure 11 , the pixel array 110 of the image sensor 100 can generate first image data ID1 in a non-Bayer pattern.

[0129] For example, when a user opens a camera, the image sensor 100 can receive a preview command from the application processor 200. The image sensor 100 can receive a zoom-in command from the application processor 200 during the preview operation. The pixel array 110 of the image sensor 100 can output first image data ID1 in a non-Bayer pattern, and the image signal processor 130 can transmit second image data ID2 obtained by rearranging the first image data ID1 in a non-Bayer pattern to the application processor 200. The artifact removal circuit 210 of the application processor 200 can store third image data ID3 generated by removing artifacts from the second image data ID2 in the memory device 300. For example, the artifact removal circuit 210 can store the third image data ID3 in the frame buffer 310 of the memory device 300. Figure 1

[0130] ​In some embodiments, the second image processing circuit 230 of the application processor 200 can process at least one piece of third image data ID3 stored in the storage device 300 in response to a user's capture request to generate a capture image.

[0131] For example, the second image processing circuit 230 can process at least one piece of third image data ID3 stored a predetermined amount of time before the time at which the capture request is received to generate a capture image, and can store the generated capture image in a storage device (e.g., the storage device 500 of the application processor 200). Figure 11

[0132] Figure 12 A capture image generation operation according to the related art is shown. According to the related art, as shown in Figure 12 , a capture image cannot be generated using image data pre-stored in a storage device. Thus, a capture image of the related art can be generated based on image data obtained a predetermined amount of time after the time at which a capture request is received from a user. As a result, shutter lag can occur.

[0133] In the case of the related art, when a capture request is made during a zoom-in state, the sensor mode of the image sensor can change. When the sensor mode of the image sensor changes, the format of the image data output by the image sensor can also change. Thus, when a capture request is made during a zoom-in state, the stored image data cannot be used until the capture request is received.

[0134] For example, in the related art, the image sensor outputs rearranged Bayer-patterned image data to generate a preview image at a high speed. In the related art, when a capture request is received during a sensor-in zoom operation, the image sensor does not perform rearrangement to improve image quality. For example, the image sensor according to the related art outputs non-Bayer-patterned image data, and the application processor performs a rearrangement process on the non-Bayer-patterned image data. Thus, the format of the image data output by the image sensor is changed, and the image data previously stored before the capture request is not available when generating a capture image. Thus, the application processor generates a capture image using image data after the capture request. As a result, shutter lag occurs based on the difference between the time at which the user presses a shutter (e.g., the capture request time) and the time at which image data used to generate a capture image is obtained.

[0135] As another example, Figure 12 An exemplary case in which a capture request is received from a user at time t2 is shown. In Figure 12 , t n and t n+1 ​The time difference between is 100 ms. In the related art, the above-described change of the sensor mode of the image sensor occurs after the capture request, and the image data (e.g., image data N-2 and image data N-1) pre-stored in the frame buffer becomes unusable. In addition, the capture image is generated using the image data after the time t2. For example, when the application processor uses the image data N+4 of the time t6, a shutter lag of 400 ms can occur. Figure 9

[0136] Figure 13 is a diagram illustrating a capture image generation operation of an imaging device (e.g., the imaging device 10) according to the disclosed embodiments. Figure 1

[0137] As illustrated in Figure 13 , the imaging device 10 according to the example embodiments can generate a capture image using image data pre-stored in a memory device. Thus, the capture image generated by the imaging device 10 can be generated based on image data of a predetermined amount of time before a capture request is received from a user. As a result, a shutter lag does not occur. The capture image generation operation of the imaging device 10 will be described with reference to Figure 1 and Figure 13 .

[0138] According to some embodiments, the imaging device 10 does not change the sensor mode of the image sensor even when a capture request is made in a zoom-in state. Figure 1

[0139] For example, the application processor 200 of the imaging device 10 can always receive the rearranged Bayer-patterned image data from the image sensor 100 to generate a preview image and a capture image.

[0140] Thus, the application processor 200 can generate a capture image using image data pre-stored before the capture request. As a result, the capture image can be generated using at least one piece of image data near the point in time at which the user presses a button or otherwise makes a capture request (e.g., a shutter capture request time). Unlike the related art, the second image processing circuit according to the example embodiments can use, at the time of making a capture request, at least one piece of third image data stored in the memory device before a predetermined time of making a capture request, among a plurality of pieces of third image data stored in the memory device. For example, Figure 1 The image sensor 100 of the imaging device 10 can always output rearranged image data and not change the sensor mode. As a result, the imaging device 10 can generate a capture image (as well as a preview image) without a shutter lag.

[0141] As an example, Figure 13 a case is illustrated in which a capture request is received from a user at the time t6. In Figure 13 ​​​In the example of FIG. 10, the time difference between t n and t n+1 is 100 ms. Figure 1 The application processor 200 of the imaging device 10 can generate a captured image using the image data N stored at time t2, which is a time 400 ms before the time t6 at which the capture request is received. Thus, shutter lag does not occur. However, this is merely an example, and example embodiments are not limited thereto. Further, the second image processing circuit can use image data stored in the memory device before a time greater than or less than 400 ms. Alternatively, the imaging device 10 can generate a captured image using the image data N stored at time t2, which is a time 400 ms before the time t6, and / or nearby image data (e.g., N-1 and N+1).

[0142] Figure 14 is a diagram illustrating a video frame generation operation of an imaging device (e.g., the imaging device 10 of Figure 1 according to an example embodiment. Redundant or similar detailed descriptions from the descriptions in the above-described embodiments are omitted below.

[0143] In some embodiments, the imaging device 10 can generate at least a portion of a video frame using image data pre-stored in the memory device. Thus, at least a portion of a video frame generated by the imaging device 10 can be generated based on image data at a predetermined amount of time before a video capture request is received from a user. As a result, shutter lag does not occur in the video frame.

[0144] In some embodiments, Figure 1 The application processor 200 of the imaging device 10 can generate at least a portion of a video frame using at least one piece of image data among a plurality of pieces of image data stored in the frame buffer at a predetermined amount of time before a time at which a video capture request can be received. For example, the application processor 200 can generate a first frame in a video frame using at least one piece of image data stored in the frame buffer at a predetermined amount of time before a time at which a video capture request is received. In addition, the application processor 200 can generate a plurality of frames in a video frame using image data stored in the frame buffer before a predetermined amount of time before a time at which a video capture request is received.

[0145] Referring to Figure 14 , an example is provided in which the application processor 200 receives a video capture request at time t0+α. The imaging device 10 can generate at least a portion of a video frame using image data stored in the frame buffer from time t0, which is a predetermined time α before the video capture request time t0+α, to time t0+α. Thus, shutter lag does not occur.

[0146] Figure 15This is a block diagram of an imaging device 1000 according to an example embodiment. Detailed descriptions that are redundant or similar to those described in the above embodiments are omitted below.

[0147] like Figure 15 As shown, the imaging device 1000 may include an imaging section 1100, an image sensor 1200, a processor 1300, a display device 1400, and a storage device 1500.

[0148] In some embodiments, the processor 1300 can control the overall operation of the imaging device 1000. In some embodiments, the processor 1300 can provide control signals to the actuator 1120 to control the position of the lens 1110. As a result, the focal length can be controlled.

[0149] The imaging portion 1100, which serves as a light receiving component, may include a lens 1110 and an actuator 1120. In some embodiments, the lens 1110 may include a plurality of lenses.

[0150] In some embodiments, actuator 1120 may move lens 1110 in a direction in which the distance to object S may increase or decrease, based on control signals from processor 1300, for example.

[0151] In some embodiments, the image sensor 1200 may generate image data and phase data based on incident light. In some embodiments, the image sensor 1200 may include a pixel array 1210, a timing controller 1220, a readout circuit 1230, and an image signal processor (ISP) 1240.

[0152] The pixels of the pixel array 1210 according to the example embodiment may include at least one photoelectric conversion element. In some embodiments, the pixel array 1210 may output non-Bayer patterned image data.

[0153] In some embodiments, the image signal processor 1240 may output Bayer patterned image data IMG obtained by rearranging non-Bayer patterned image data. The image sensor 1200 may output Bayer patterned image data IMG regardless of which command CMD (e.g., preview command or zoom command) the processor 1300 sends.

[0154] In some embodiments, the processor 1300 can process Bayer patterned image data (IMG) and generate a preview image or a captured image. In some embodiments, the processor 1300 can use image data stored in a frame buffer to generate a hysteresis-free captured image.

[0155] Figure 16 This is a flowchart illustrating the operation of an imaging device according to an example embodiment. Figure 16 The operation can be, for example, by Figure 1by the imaging device 10. For example, the operations of S100 can be performed by Figure 1 by the image sensor 100, and the operations of S200 can be performed by Figure 1 by the application processor 200. The operations of the imaging device 10 will be described with reference to Figure 16 redundant or similar detailed descriptions provided for Figures 1 to 15 are omitted below.

[0156] Referring to Figure 16 , in operation S110, the pixel array 110 can generate a pixel signal.

[0157] In operation S120, the readout circuit 120 can convert the pixel signal to a digital signal and output first image data ID1. The first image data ID1 can be non-Bayer patterned image data.

[0158] In operation S130, the image sensor 100 can determine whether a zoom command has been received from the application processor 200.

[0159] When the zoom operation is not being performed, the flow can proceed to operation S140, in which the image sensor 100 merges the first image data, rearranges the merged first image data, and generates second image data ID2. The second image data ID2 can be Bayer patterned image data.

[0160] When the zoom operation is being performed, the flow can proceed to operation S150, in which the image sensor 100 crops at least a portion of the first image data, rearranges the cropped first image data, and generates second image data ID2. The second image data ID2 can be Bayer patterned image data.

[0161] In operation S160, the image sensor 100 can output the second image data ID2.

[0162] In operation S210, the artifact removal circuit of the application processor 200 can receive the Bayer patterned second image data ID2 transmitted by the image sensor 100 and generate, for example, third image data ID3 that is Bayer patterned and artifact removed.

[0163] In operation S220, the application processor 200 can determine whether a capture request has been received from a user.

[0164] When the capture request has not been received, the flow can proceed to operation S230, in which the first image processing circuit can perform first image processing on the Bayer patterned third image data ID3 to generate a preview image.

[0165] In operation S240, the application processor 200 can be configured to display the preview image on the display device.

[0166] When the capture request has been received, the flow can proceed to operation S250, in which the application processor 200 can perform a second image processing on the third image data ID3 patterned in a Bayer pattern to generate a captured image. In operation S260, the application processor 200 can store the captured image in a storage device or a memory device.

[0167] As described and illustrated above, the image sensor and the imaging apparatus according to the example embodiments can generate a captured image without shutter lag.

[0168] Further, the image sensor and the imaging apparatus according to the disclosed embodiments can generate a captured image without shutter lag even when performing an in-sensor zoom function.

[0169] Further, the image sensor and the imaging apparatus according to the disclosed embodiments can generate a high-quality captured image without shutter lag even when performing an in-sensor zoom operation.

[0170] Although example embodiments have been shown and described above, it will be clear to those of ordinary skill in the art that modifications and changes can be made without departing from the scope of the inventive concept defined by the appended claims.

Claims

1. An imaging device, comprising: An image sensor, wherein the image sensor comprises: A pixel array comprising multiple pixels, the pixel array providing pixel signals. The readout circuit is configured to output non-Bayer patterned first image data based on the pixel signals, and An image signal processor configured to rearrange at least a portion of the first image data and output Bayer patterned second image data; Display devices or storage devices; and Application processor, wherein the application processor is configured to: Control the image sensor, Based on the second image data with Bayer patterning, a third image data with Bayer patterning is generated, and Based on the generated Bayer patterned third image data, a preview image is displayed on the display device or the captured image is stored in the storage device. The pixel array includes a non-Bayer patterned color filter array.

2. The imaging device as claimed in claim 1, wherein: The color filter array includes multiple color filter groups. Each of the plurality of color filter groups includes a plurality of color filters arranged adjacent to each other and configured to allow light of the same wavelength to pass through them, and Adjacent color filter groups among the plurality of color filter groups are configured to allow light of different wavelengths to pass through them.

3. The imaging device as described in claim 2, wherein, The application processor is also configured to: In response to a camera activation request, the preview image is generated by performing a first image processing on the third image data, and the preview image is displayed on the display device. as well as In response to a capture request, the captured image is generated by performing a second image processing on the third image data, and the captured image is stored in the storage device.

4. The imaging device as described in claim 3, wherein, In response to both the camera activation request and the capture request, the image signal processor rearranges at least a portion of the first image data and outputs the Bayer patterned second image data.

5. The imaging device as described in claim 3, wherein, The application processor is configured to remove one or more artifacts from the Bayer-patterned second image data to generate the Bayer-patterned third image data.

6. The imaging apparatus of claim 1, further comprising a memory device configured to store the third image data, wherein, The application processor is also configured to store the third image data in the memory device, the third image data comprising multiple frames.

7. The imaging device as claimed in claim 6, wherein, The application processor is also configured to generate the captured image in response to the capture request, based on at least one piece of third image data stored in the memory device before a predetermined time from among the plurality of frames of the third image data.

8. The imaging device as claimed in claim 1, wherein, The image signal processor includes: An image cropping circuit configured to crop a portion of the first image data in response to a scaling command received from the application processor; and A rearrangement circuit is configured to rearrange the cropped portion of the first image data and generate the second image data.

9. The imaging device as claimed in claim 1, wherein, The image signal processor includes: An image merging circuit configured to merge the first image data in response to a preview command received from the application processor; and A rearrangement circuit is configured to rearrange the merged portions of the first image data and generate the second image data.

10. The imaging device as claimed in claim 1, wherein, The plurality of pixels includes more than one hundred million pixels.

11. An image sensor, comprising: A pixel array comprising a plurality of pixels, the pixel array providing pixel signals; The readout circuit is configured to output non-Bayer patterned first image data based on the pixel signal; as well as The image signal processor is configured as follows: Receive scaling commands from the application processor. Receive preview command from the application processor, and In response to both the zoom command and the preview command, at least a portion of the non-Bayer patterned first image data is rearranged, and Bayer patterned second image data is output. The pixel array includes a non-Bayer patterned color filter array.

12. The image sensor of claim 11, wherein: The color filter array includes multiple color filter groups; Each of the plurality of color filter groups includes a plurality of color filters arranged adjacent to each other and configured to allow light of the same wavelength to pass through therethrough; and Adjacent color filter groups among the plurality of color filter groups are configured to allow light of different wavelengths to pass through them.

13. An electronic device comprising: Image sensor; Display device or storage device; as well as Application processor, the application processor being configured to: Control the image sensor; Receive Bayer patterned first image data from the image sensor; Remove one or more artifacts from the Bayer patterned first image data to generate Bayer patterned second image data; as well as A preview image based on the second image data with Bayer patterning is displayed on the display device, or a captured image based on the second image data with Bayer patterning is stored in the storage device.

14. The electronic device of claim 13, wherein, The resolution of the second image data patterned by Bayer is the same as the resolution of the first image data patterned by Bayer.

15. The electronic device of claim 14, wherein, The application processor is also configured to: Video frames are generated based on the second image data patterned by Bayer; and The video frames are stored in the storage device.

16. The electronic device of claim 13, wherein, The application processor is also configured to: Send a scaling command to the image sensor; and In response to the scaling command, the first image data, patterned by Bayer, is received from the image sensor.

17. The electronic device of claim 13, further comprising a memory device, wherein the application processor is further configured to: Multiple frames of the Bayer-patterned second image data are stored in the memory device; and In response to a capture request, the captured image is generated based on at least one Bayer patterned second image data of the plurality of frames of the Bayer patterned second image data that was stored in the memory device before a predetermined time.

18. The electronic device of claim 17, wherein, The application processor is also configured to: A video frame is generated in response to a video storage request, wherein at least a portion of the video frame is generated based on at least one Bayer patterned second image data stored in the memory device.

19. The electronic device of claim 18, wherein, The electronic device generates the first frame of the video frame using at least one Bayer patterned second image data stored in the memory device.

20. The electronic device of claim 13, wherein, The application processor performs different types of image signal processing on the Bayer patterned second image data to generate both the preview image and the captured image.

Images