Image sensing device and image merging method thereof

Abstract

An image sensing device is provided. The image sensing device includes an image sensor and a digital signal processor. The image sensor includes a pixel array and an analog end. The pixel array is configured to output a raw image with a Bayer pattern. The analog end is configured to perform an analog merging process on pixel groups of the same color in the raw image and output medians for different colors. The pixel groups of the same color include those in the same column of each of a plurality of sub-kernels corresponding to a first green pixel, a red pixel, a blue pixel, and a second green pixel. The digital signal processor is configured to perform a digital merging process on the medians for different colors included in different columns of each of the plurality of sub-kernels and output a merged image.

Description

Technical Field

[0001] Example embodiments of the present disclosure relate to an image sensing device and an image merging method thereof. Background Art

[0002] An image sensing device may be used in a mobile device or an electronic device (e.g., a smart phone, a tablet personal computer (PC), a digital camera, etc.). Generally, the image sensing device is configured to have two-dimensionally integrated pixels, and converts an electrical signal corresponding to the luminance of incident light into a digital signal and outputs the digital signal. Here, the image sensing device may include a Bayer pattern, and may provide Bayer image data corresponding to the luminance of light incident on the Bayer pattern.

[0003] With the development of technology, the number of pixels in each image sensing device (i.e., the resolution of the image sensing device) has increased, and the increase in pixel resolution may lead to an increase in the amount of data to be processed. Accordingly, the image sensing device performs merging. Merging includes not only creating an image using information of all pixels of the image sensing device, but also merging information about each adjacent pixel group into single integrated information, and creating a target image to be processed by using the integrated information. Summary of the Invention

[0004] One or more example embodiments provide an image merging method that can reduce or mitigate jagged noise and any false color defects, thereby outputting a merged image with improved quality.

[0005] One or more example embodiments also provide an image sensing device that can reduce or mitigate jagged noise and any false color defects, thereby outputting a merged image with improved quality.

[0006] However, the embodiments are not limited to those described herein. The above and other example embodiments will become more apparent to those of ordinary skill in the art to which the present disclosure pertains by referring to the following detailed description of the present disclosure.

[0007] According to one aspect of the exemplary embodiment, an image sensing device is provided, including an image sensor and a digital signal processor. The image sensor includes a pixel array and an analog end. The pixel array is configured to output a raw image having a Bayer pattern, and the analog end is configured to perform an analog merging process on pixel groups of the same color in the raw image and output medians for different colors. The pixel groups of the same color include those in the same column of each of a plurality of sub-kernels corresponding to a first green pixel, a red pixel, a blue pixel, and a second green pixel; and the digital signal processor is configured to perform a digital merging process on the medians for different colors included in different columns of each of the plurality of sub-kernels and output a merged image.

[0008] According to another aspect of the exemplary embodiment, an image merging method of an image sensing device is provided. The method includes: receiving a raw image having a Bayer pattern from a pixel array; outputting a median for each kernel based on performing an analog merging process on the raw image of unit kernels having an asymmetric arrangement with respect to different colors; and outputting a merged image based on performing a digital merging process on at least one of the medians.

[0009] According to still another aspect of the exemplary embodiment, an image sensing device is provided, including an image sensor and a digital signal processor. The image sensor includes a pixel array and an analog end. The pixel array is configured to output a raw image having a Bayer pattern. The analog end is configured to perform an analog merging process on a first pixel and a second pixel arranged in a diagonal direction for each color in each of the square unit kernels of the raw image and output a median for the corresponding color. The digital signal processor is configured to obtain a merged pixel value based on performing a digital merging process on the median and a third pixel of the corresponding color arranged at a corner in each of the unit kernels, output the merged pixel value, and generate a merged image including the merged pixel value. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and / or other aspects will become more apparent by describing the exemplary embodiment in detail with reference to the drawings, in which:

[0011] Figure 1 is a block diagram of an image sensing device according to the exemplary embodiment;

[0012] Figure 2 and Figure 3 show a raw Bayer image and a merged Bayer image for explaining the image merging method according to the exemplary embodiment, respectively;

[0013] Figure 4A and Figure 4B show the pixel array according to the exemplary embodiment as a Bayer pattern;

[0014] Figure 5 is Figure 4A the circuit diagram of the pixel array of;

[0015] Figure 6 is a flowchart showing an image merging method according to an exemplary embodiment;

[0016] Figure 7 and Figure 8 respectively show the original Bayer image and the merged Bayer image for explaining the image merging method according to the exemplary embodiment;

[0017] Figure 9A 、 Figure 9B and Figure 9C show the image merging method according to the exemplary embodiment;

[0018] Figure 10 is a flowchart showing Figure 9A 、 Figure 9B and Figure 9C the image merging method of;

[0019] Figure 11 shows an image sensor according to the exemplary embodiment;

[0020] Figure 12 is Figure 11 the circuit diagram of the pixel array of;

[0021] Figure 13 and Figure 14 show the Figure 11 image sensor for explaining the image merging method according to the exemplary embodiment;

[0022] Figure 15 and Figure 16 show the Bayer image for explaining the image merging method according to the exemplary embodiment;

[0023] Figure 17 is Figure 1 the perspective view of the image sensing device of;

[0024] Figure 18 is a block diagram of an electronic device including a multi-camera module according to the exemplary embodiment; and

[0025] Figure 19 is Figure 18 the detailed block diagram of the multi-camera module of; Detailed Description of the Invention

[0026] Exemplary embodiments will be described below with reference to the accompanying drawings.

[0027] The units and modules disclosed herein and the functional blocks shown in the drawings can be implemented as hardware, software, or a combination thereof configured to perform specific functions.

[0028] Figure 1 is a block diagram of an image sensing device according to an exemplary embodiment.

[0029] Referring to Figure 1 , the image sensing device 1 may include a display unit 300, a digital signal processor (DSP) 150, and an image sensor 200.

[0030] The image sensor 200 includes a pixel array 210, a row driver 220, a correlated double sampling (CDS) block 230, an analog-to-digital converter (ADC) 240, a ramp generator 260, a timing generator 270, a control register block 280, and a buffer 290.

[0031] The image sensor 200 may sense an object 400 captured via a lens 500 under the control of the DSP 150, and the DSP 150 may output the image sensed and output by the image sensor 200 to the display unit 300.

[0032] In an exemplary embodiment, the image sensor 200 may receive a raw image from the pixel array 210, may perform analog merging on the raw image via the ADC 240 and the buffer 290, and may output the resulting analog merged image to the DSP 150.

[0033] Examples of the display unit 300 may include various types of devices that can output or display an image. For example, the display unit 300 may be a computer, a mobile communication device, or another image output terminal.

[0034] The DSP 150 includes a camera controller 110, an image signal processor (ISP) 100, and an interface (I / F) 120.

[0035] The camera controller 110 may control the operation of the control register block 280. The camera controller 110 may use an internal integrated circuit (I2C) to control the operation of the image sensor 200, particularly the operation of the control register block 280, but the embodiment is not limited thereto.

[0036] The ISP 100 may receive image data from the buffer 290, may process the received image data to be user-friendly, and may output the processed image data to the display unit 300 via the I / F 120.

[0037] The ISP 100 can perform digital merging on the image output by the image sensor 200. The ISP 100 can output the resulting digitally merged image as the final merged image to the display unit 300. The image output by the image sensor 200 can be the original image from the pixel array 210, or can be the analog merged image.

[0038] Referring Figure 1 , the ISP 100 is shown as being located in the DSP 150. However, the embodiments are not limited thereto. For example, according to other exemplary embodiments, the ISP 100 can be included in the image sensor 200. Additionally, in other exemplary embodiments, the image sensor 200 and the ISP 100 can be implemented as a single package, for example, a multi-chip package (MCP).

[0039] The pixel array 210 can be implemented as an array of a plurality of photodetection devices (e.g., photodiodes or pinned photodiodes). The raw image data output from the pixel array 210 via the CDS block 230 and the ADC 240 can be Bayer image data configured in a Bayer format.

[0040] The Bayer image data can be processed by the ISP 100 into data in a red, green, blue (RGB) format, and then can be output to the display unit 300.

[0041] As used herein, the term "pixel" or "pixel value" can refer to the information or value output or obtained according to the optical signal via each pixel element forming the Bayer color filter. As used herein, the term "original image" can refer to an image based on the unit original pixels to be subjected to image signal processing (specifically, the pixel values sensed by the image sensor). As used herein, the term "merged image" can refer to a post-processed image obtained by processing the original image using a predetermined method.

[0042] In the following description, it is assumed that the original image has a Bayer pattern, as Figure 2 shown, in which the rows in which Gr pixels and R pixels are alternately arranged and the rows in which Gr pixels and B pixels are alternately arranged are alternately arranged. Herein, the R pixel refers to the red pixel, the B pixel refers to the blue pixel, and both the Gr pixel and the Gb pixel refer to the green pixel. Specifically, Gr refers to the green pixel included in the row having the red pixel, and Gb refers to the green pixel included in the row having the blue pixel.

[0043] The merged image is a reduced image, and each pixel of the merged image may be referred to by R', G' (e.g., Gr' or Gb'), or B'. The merged image may also have a Bayer pattern in which rows in which Gr' pixels and R' pixels are alternately arranged and rows in which Gr' pixels and B' pixels are alternately arranged are alternately arranged.

[0044] Figure 2 and Figure 3 respectively show the original Bayer image and the merged Bayer image for explaining the image merging method according to an exemplary embodiment.

[0045] The image merging method may be performed in units of a (4n + 4)×(4n + 4) kernel. Figure 2 and Figure 3 of the image merging method. Figure 2 and Figure 3 show an example where n = 1 (i.e., the kernel size is 8×8).

[0046] Referring to Figure 2 the original image may have a Bayer pattern in which rows in which Gr pixels and R pixels are alternately arranged and rows in which B pixels and Gb pixels are alternately arranged are alternately arranged.

[0047] The image merging may be performed in units of a kernel K, and each kernel K includes sub-kernels. Figure 2 and Figure 3 The image merging method of Figure 2 and Figure 3 will be described below as being performed on an original image with a kernel size of 8×8, but the embodiment is not limited thereto. That is,

[0048] The image merging method of

[0049] The image sensing device 1 may generate a merged image by merging the original image for different colors of different sub-kernels. That is, Gr pixels may be merged according to the sub-kernel K1, R pixels may be merged according to the sub-kernel K2, B pixels may be merged according to the sub-kernel K3, and Gb pixels may be merged according to the sub-kernel K4.

[0050] Specifically, referring toFigure 2 and Figure 3 , six Gr pixels, four R pixels, nine B pixels, and six Gb pixels can be combined according to the original image to generate a combined image with Gr', R', B', and Gb' pixels. The Gr', R', B', and Gb' pixels can be as shown in the following equation (1).

[0051] <Equation 1>

[0052] Gr' = (Gr1 + Gr2 + Gr3 + Gr4 + Gr5 + Gr6) / 6

[0053] R' = (R1 + R2 + R3 + R4) / 4

[0054] B' = (B1 + B2 + B3 + B4 + B5 + B6 + B7 + B8 + B9) / 9

[0055] Gb' = (Gb1 + Gb2 + Gb3 + Gb4 + Gb5 + Gb6) / 6

[0056] Each pixel of the combined image can be generated at a position corresponding to the centroid of each sub-kernel. For example, when the coordinates of the upper left corner of kernel K are (0, 0), the Gr' pixel is generated at the center (2, 1) of sub-kernel K1, the R' pixel is generated at the center (6, 1) of sub-kernel K2, the B' pixel is generated at the center (2, 5) of sub-kernel K3, and the Gb' pixel is generated at the center (6, 5) of sub-kernel K4.

[0057] Referring to Figure 3 , in the combined image, the distances between the Gr', R', B', and Gb' pixels (e.g., the distance between the Gr' and R' pixels, the distance between the B' and Gb' pixels, the distance between the Gr' and B' pixels, and the distance between the R' and Gb' pixels) can all be the same.

[0058] By making the number of input pixels used for the combination operation vary for different color channels with the change of the sub-kernel, the distance between the pixels of the combined image can be kept at a consistent ratio with the distance between the pixels of the original image. Therefore, the combined image can be made robust against any false color defect and aliasing.

[0059] It can be executed by ISP 100 Figure 2 and Figure 3 the image combination method.

[0060] As Figures 3 to 6As shown, each of the Gr', R', B', and Gb' pixels of the merged image can be obtained by performing analog merging on the original pixels belonging to the same column to obtain the median and adding and averaging the medians from different columns via digital merging. The analog merging can be performed by the output terminals of the pixel array 210 including the CDS 230, the ADC 240, and the buffer 290.

[0061] The output terminals (230, 240, and 290) of the pixel array 210 for performing analog merging will be referred to as the analog terminals hereinafter. The digital merging can be performed by the ISP 100, and the digital merging can include adding and averaging the medians stored in the buffer 290, thereby outputting the merged pixel values to the I / F 120.

[0062] Figure 4A and Figure 4B The pixel array according to the exemplary embodiment is shown as a Bayer pattern. Figure 5 is Figure 4A the circuit diagram of the pixel array. Figure 6 is a flowchart showing an image merging method according to some embodiments of the present disclosure.

[0063] Figure 2 The sub-core K1 of Figure 4A or Figure 4B can be implemented as shown. Referring to Figure 4A and Figure 4B , the ADC 240 and the current source can be connected to each column of the pixel array.

[0064] The sub-core K1 can correspond to a part of the pixel array. Figure 4A and Figure 4B show that the sub-core K1 includes five pixel columns and three pixel rows, that is, a total of 15 pixels, but the embodiment is not limited thereto. For example, the sub-core K1 can be implemented in various other ways.

[0065] Figure 4A and Figure 4B Each of the pixel arrays of Figure 5 includes a plurality of unit cores, and each of the unit cores can include four pixels, for example, Gr pixel, R pixel, B pixel, and Gb pixel. Figure 4A shows an example in which each pixel includes a photodiode, a transfer transistor, a reset transistor, a selection transistor, and a driving transistor. The row-direction selection signal Sel is applied to the gate of the selection transistor, and the column-direction selection signal Col is applied to the drain of the selection transistor. In the exemplary embodiment, as Figure 4BAs shown, all columns of the pixel array can be connected to the same common ADC and the same common current source via switches. For example, the common ADC and the common current source can be connected to the pixel array such that the common ADC and the common current source can be switched according to the Bayer pattern arrangement and can thus be shared by pixels of the same color. According to another exemplary embodiment, the common ADC and the common current source can be connected to the pixel array such that the common ADC and the common current source can be switched for image merging and can thus be shared unit by unit core.

[0066] Each pixel can include four transistors and at least one photodiode, as Figure 5 shown. However, the embodiment is not limited thereto. For example, each pixel can include three transistors and at least one photodiode, or each pixel can include five transistors and at least one photodiode. Figure 5 It is shown that each pixel of the pixel array includes four transistors and one photodiode, and different columns of the pixel array are connected to different ADCs and different current sources, but the embodiment is not limited thereto. For example, Figure 5 each pixel of the pixel array can include four transistors and one photodiode, and Figure 5 at least two columns of the pixel array can be connected to the same common ADC and the same common current source.

[0067] The image sensing device 1 can perform image merging unit by core. In an exemplary embodiment, referring to Figure 4A and Figure 6 , in response to an input raw image (S10), image merging is performed unit by sub-core. Then, the median is calculated by performing analog merging (S20) on a pixel group of the same column belonging to each sub-core, and the medians of different columns from the corresponding sub-cores can be added via digital merging (S30). The added result can be output as the pixel value of the merged image (S40).

[0068] Referring to Figure 4B and Figure 6 , in response to an input raw image (S10), image merging is performed unit by sub-core. Then, the median is calculated by performing analog merging (S20) on a pixel group of the same color belonging to the same row but different columns via an ADC and a current source commonly connected to at least two columns to which pixels of the same color belong. Thereafter, the pixel values of the same color belonging to at least one of the at least two columns but different rows can be added to the median via digital merging (S30), and the added result can be output as the pixel value of the merged image (S40).

[0069] The image sensing device 1 may use six Gr pixels of the sub-kernel K1 to generate the Gr' pixel of the merged image. The image sensing device 1 may select one color pixel from the same position of each unit kernel. For example, each unit kernel may be a 2×2 pixel array, and the pixel at (0, 0) (e.g., Gr pixel) is selected from the unit kernel including (0, 0), (1, 0), (0, 1), and (1, 1), and the Gr pixel at (2, 0) is selected from the unit kernel including (2, 0), (2, 1), (3, 0), and (3, 1).

[0070] The image sensing device 1 may select one pixel from each unit kernel, and may activate the Gr1 pixel, Gr2 pixel, Gr3 pixel, Gr4 pixel, Gr5 pixel, and Gr6 pixel at (0, 0), (0, 2), (2, 0), (2, 2), (4, 0), and (4, 2) respectively by turning on Sel[0] and Sel[2] and turning on Col[0], Col[2], and Col[4].

[0071] In order to find the value of the Gr' pixel according to Equation (1), the image sensing device 1 may output three Gr medians (e.g., Gr1+Gr4, Gr2+Gr5, and Gr3+Gr6) at the analog end including the ADC and the current source, and the three Gr medians are obtained by adding the values of the pixels at (0, 0) and (0, 2), the values of the pixels at (2, 0) and (2, 2), and the values of the pixels at (4, 0) and (4, 2) respectively. Then, the image sensing device 1 may add the three Gr medians at the digital end and divide the added result by the number of input pixels (i.e., 6), and may output the divided result as the value of the Gr' pixel.

[0072] The image sensing device 1 may find the values of the R', B', and Gb' pixels from the sub-kernels K2, K3, and K4 in the same way as finding the value of the Gr' pixel. For example, the image sensing device 1 may output two R medians (e.g., R1+R3 and R2+R4) at the analog end including the ADC and the current source, and the two R medians are obtained by adding the values of the pixels at (5, 0) and (5, 2), and the values of the pixels at (7, 0) and (7, 2) respectively. Then, the image sensing device 1 may add the two R medians at the digital end and divide the added result by the number of input pixels (i.e., 4), and may output the divided result as the value of the R' pixel.

[0073] To find the value of the B' pixel according to Equation (1), the image sensing device 1 may output three B medians (e.g., B1 + B4 + B7, B2 + B5 + B8, and B3 + B6 + B9) at the analog end, and the three B medians are obtained by adding the pixel values at (0, 3), (0, 5), and (0, 7), the pixel values at (2, 3), (2, 5), and (2, 7), and the pixel values at (4, 3), (4, 5), and (4, 7), respectively. Then, the image sensing device 1 may add the three B medians at the digital end and divide the added result by the number of input pixels (i.e., 9), and may output the divided result as the value of the B' pixel.

[0074] To find the value of the Gb' pixel according to Equation (1), the image sensing device 1 may output two Gb medians (i.e., Gb1 + Gb3 + Gb5 and Gb2 + Gb4 + Gb6) at the analog end including an ADC and a current source, and the two Gb medians are obtained by adding the pixel values at (5, 3), (5, 5), and (5, 7) and the pixel values at (7, 3), (7, 5), and (7, 7), respectively. Then, the image sensing device 1 may add the two Gb medians at the digital end and divide the added result by the number of input pixels (i.e., 6), and may output the divided result as the value of the Gb' pixel.

[0075] Figure 7 and Figure 8 respectively show the original Bayer image and the merged Bayer image for explaining the image merging method according to the exemplary embodiment.

[0076] The image merging method may be performed in units of (4n + 4)×(4n + 4) kernels Figure 7 and Figure 8 The image merging method. Figure 7 and Figure 8 show an example where n = 2 and the image merging is performed in units of 12×12 kernels.

[0077] Referring to Figure 7 and Figure 8 When the size of the kernel K is 12×12, the kernel K may be divided into a 7×5 sub-kernel K1, a 5×5 sub-kernel K2, a 7×7 sub-kernel K3, and a 5×7 sub-kernel K4.

[0078] That is, each pair of horizontally or vertically adjacent sub-kernels of the kernel K may satisfy the following Equation (2).

[0079] <Equation 2>

[0080] L = M + N (where M and N are the closest odd numbers)

[0081] Referring to Equation (2), L, M, and N are natural numbers, and respectively represent the length L of the kernel K in the first direction, the length M of the first sub-kernel of the kernel K in the first direction, and the length N of the second sub-kernel of the kernel K in the first direction.

[0082] If the size of the kernel K is 12×12, then L = 12, and M and N can be 5 and 7 (5 + 7 = 12). Thus, the kernel K can be divided into sub-kernels with lengths of five pixels or seven pixels. In Figure 7 and Figure 8 's example, the kernel K can be divided into sub-kernels with a length of seven pixels or five pixels in the horizontal direction and a length of five pixels and seven pixels in the vertical direction. Alternatively, the kernel K can be divided into sub-kernels with a length of five pixels or seven pixels in the horizontal direction and a length of seven pixels or five pixels in the vertical direction. However, the Gr pixels and Gb pixels can be merged according to sub-kernels with different lengths in the horizontal and vertical directions (e.g., sub-kernel K1 and sub-kernel K4).

[0083] Referring to Figure 8 's merged image, the Gr' pixels, R' pixels, B' pixels, and Gb' pixels can be respectively located in sub-kernel K1, sub-kernel K2, sub-kernel K3, and sub-kernel K4, specifically at (3, 2), (9, 2), (3, 8), and (9, 8) respectively.

[0084] The Gr', R', B', and Gb' pixels of the merged image can be output by the ISP 100 of the image sensing device 1. Each of the Gr', R', B', and Gb' pixels of the merged image can be obtained by performing analog merging on the original pixel groups belonging to the same column to obtain the median and adding and averaging the medians from different columns via digital merging. The analog merging can be performed at the output end of the pixel array 210, and the obtained median can be stored in the buffer 290. The digital merging can be performed by the ISP 100, and the digital merging can include adding and averaging the medians stored in the buffer 290 and outputting the obtained merged pixel values to the I / F 120.

[0085] Therefore, the Gr', R', B', and Gb' pixels of the merged image can be located on the same axis in the horizontal or vertical direction at the centroids of sub-kernel K1, sub-kernel K2, sub-kernel K3, and sub-kernel K4 respectively. Therefore, the Gr', R', B', and Gb' pixels of the merged image do not cause jagged artifacts and can be more robust against false color defects.

[0086] Figure 9A 、Figure 9B and Figure 9C illustrates an image merging method according to some embodiments of the present disclosure. Figure 10 is a flowchart showing Figure 9A , Figure 9B and Figure 9C of the image merging method.

[0087] Figure 9A , Figure 9B and Figure 9C illustrates an example of performing image merging on a kernel of size 4×4, but the embodiments are not limited thereto, and image merging can also be applied to various other kernel sizes.

[0088] Referring to Figure 9A the original 4×4 unit kernel image of, a first merging process is performed on the Gr pixels at (2, 0) and (0, 2) that are located in the first diagonal direction relative to each other. Therefore, the Gr median obtained through the first merging process is arranged at point A1, that is, at (1, 1). Referring to Figure 9B the intermediate 4×4 unit kernel image of, a second merging process is performed on the Gr median at (1, 1) and the Gr pixel at (0, 0) that is located in the second diagonal direction relative to the Gr median at (1, 1). Here, the second diagonal direction (i.e., the direction perpendicular to and intersecting the first diagonal direction) can be the direction from point A1 to point A2. Referring to Figure 9C the merged 4×4 unit kernel image of, the Gr' pixel obtained through the second merging process is arranged at the midpoint between point A1 and point A2.

[0089] Similarly, referring again to Figure 9A the original 4×4 unit kernel image of, a first merging process is performed on the R pixels at (1, 0) and (3, 2) that are located in the second diagonal direction relative to each other. Therefore, the R median obtained through the first merging process is arranged at point B1, that is, at (2, 1). Referring to Figure 9B the intermediate 4×4 unit kernel image of, a second merging process is performed on the R median at (2, 1) and the R pixel at (3, 0) that is located in the first diagonal direction relative to the R median at (2, 1). Here, the first diagonal direction (i.e., the direction perpendicular to and intersecting the second diagonal direction) can be the direction from point B1 to point B2. Referring to Figure 9C the merged 4×4 unit kernel image of, the R' pixel obtained through the second merging process is arranged at the midpoint between point B1 and point B2.

[0090] Similarly, referring again to Figure 9AFor the original 4×4 unit kernel image, a first merging process is performed on B pixels at (0, 1) and (2, 3) which are located in the first diagonal direction relative to each other. Thus, the B median value obtained through the first merging process is arranged at point C1, i.e., at (1, 2). Refer to Figure 9B For the intermediate 4×4 unit kernel image, a second merging process is performed on the B median value at (1, 2) and the B pixel at (0, 3) which is located in the second diagonal direction relative to the B median value at (1, 2). Here, the first diagonal direction (i.e., the direction perpendicular to the second diagonal direction) can be the direction from point C1 to point C2. Refer to Figure 9C For the merged 4×4 unit kernel image, the B' pixel obtained through the second merging process is arranged at the midpoint between point C1 and point C2 .

[0091] Similarly, referring again to Figure 9A For the original 4×4 unit kernel image, a first merging process is performed on Gb pixels at (1, 3) and (3, 1) which are located in the second diagonal direction relative to each other. Thus, the Gb median value obtained through the first merging process is arranged at point D1, i.e., at (2, 2). Refer to Figure 9B For the intermediate 4×4 unit kernel image, a second merging process is performed on the Gb median value at (2, 2) and the Gb pixel at (3, 3) which is located in the first diagonal direction relative to the Gb median value at (2, 2). Here, the first diagonal direction (i.e., the direction perpendicular to the second diagonal direction) can be the direction from point D1 to point D2. Refer to Figure 9C For the merged 4×4 unit kernel image, the centroid of the Gb' pixel obtained through the second merging process is arranged at the midpoint between point D1 and point D2 .

[0092] In the 2×2 merged image, four pixels (e.g., Gr pixel, R pixel, B pixel, and Gb pixel) whose centroids obtained through the second merging process overlap with each other can be located at (0, 0), (1, 0), (0, 1), and (1, 1), at (2, 0), (3, 0), (2, 1), and (3, 1), at (0, 2), (1, 2), (0, 3), and (1, 3), and at (2, 2), (3, 2), (2, 3), and (3, 3), respectively.

[0093] Figures 9A to 9C Shows the 2×2 merging. As a result of the 2×2 merging, each unit kernel of the original image can be converted from 4×4 to 2×2 or from 8×8 to 4×4. That is, the 2×2 merging can be merging a 2n×2n kernel into an n×n kernel, where n is an integer greater than 0.

[0094] Reference Figure 10 The image sensing device 1 can receive a raw image (S20), and can perform a first merging process on the raw image in a first direction or in a first diagonal direction (S21) to obtain a median value. Here, the first diagonal direction can be the direction between pixels of the same color from different rows.

[0095] The image sensing device 1 can perform a second merging process (S22) on the median value obtained through the first merging process and the raw pixels at the corners corresponding to the median value obtained through the first merging process. The second merging process can be performed in a second direction or a second diagonal direction. The second diagonal direction (i.e., the direction between the median value obtained through the first merging process and the raw pixels at the corners corresponding to the median value obtained through the first merging process) can be a direction intersecting the first diagonal direction. In an exemplary embodiment, the first merging process can also be a diagonal merging process, and the second merging process can also be a corner merging process. The result of the second merging process can be output as a merged image (S23).

[0096] Both the first merging process and the second merging process can be performed by Figure 1 the ISP 100. However, the embodiment is not limited thereto. For example, the first merging process can be performed by the image sensor 200, and the second merging process can be performed by the ISP 100. In this case, the first merging process can be performed by the ADC 240 or the buffer 290.

[0097] Figure 11 An image sensor according to an exemplary embodiment is shown. Figure 12 is Figure 11 a circuit diagram of the pixel array of Figure 13 and Figure 14 show an image sensor for explaining an image merging method according to an exemplary embodiment of Figure 11

[0098] In some embodiments, the analog end can perform the image merging method described above with reference to Figure 8 and Figures 9A to 9C . The analog end can include the ADC 240 and a current source.

[0099] The analog end can be commonly connected to at least two columns to which the pixels of the same column belong. For example, the analog end can be connected to at least two adjacent even or odd columns of the pixel array. Referring to Figure 11 , the ADC 240 can be connected between two adjacent even columns of the pixel array, and the analog end can perform the first merging process in the first diagonal direction. The ADC 240 can be connected between two adjacent odd columns of the pixel array, and the analog end can perform the first merging process in the first diagonal direction.

[0100] Referring to Figure 11 , the pixel array may include a plurality of unit cores each having 16 pixels, for example, a plurality of 4×4 unit cores. The pixel array is driven in units of unit cores. Figure 12 An example is shown in which each pixel includes a photodiode, a transfer transistor, a reset transistor, a selection transistor, and a driving transistor. A row-direction selection signal Sel is applied to the gate of the selection transistor, and a column-direction selection signal Col is applied to the drain of the selection transistor.

[0101] As Figure 12 shown, each pixel may include four transistors and at least one photodiode, each pixel may include three transistors and at least one photodiode, or each pixel may include five transistors and at least one photodiode.

[0102] Referring to Figure 11 , Figure 12 and Figure 13 , Gr pixel merging is performed on the sub-unit core KS1 of the unit core K1. Specifically, by turning on Sel[1], Sel[5], Col[0], and Col[2] in the unit core K1, two Gr pixels in the first diagonal direction marked by the symbol ☆ in Figure 13 are received, and the Gr pixel at Sel[0] and Col[0] is received by turning on Sel[0] and Col[0]. The ADC 240 is commonly connected to Col[0] and Col[2], and the analog end performs a first merging process on the two Gr pixels in the first diagonal direction marked by the symbol ☆ in Figure 13 , thereby outputting a Gr median value. The center of gravity of the Gr median value may be located at Sel[2] and Col[1]. The ISP 100 performs a second merging process on the value of the Gr pixel at Sel[0] and Col[0] and the Gr median value at Sel[2] and Col[1], thereby outputting a merged Gr' pixel.

[0103] Referring to Figure 11 , Figure 12 and Figure 14 , Gb pixel merging is performed on the sub-unit core KS2 of the unit core K1. Specifically, by turning on Sel[3], Sel[6], Col[1], and Col[3] in the sub-unit core KS2, two Gb pixels in the first diagonal direction marked by the symbol ☆ in Figure 14 are received, and the Gb pixel at Sel[7] and Col[3] is received by turning on Sel[7] and Col[3]. The ADC 240 is commonly connected to Col[1] and Col[3], and the analog end pairs the two Gb pixels in the first diagonal direction marked by the symbol ☆ in Figure 14The two Gb pixels in the first diagonal direction marked by the symbol ☆ are subjected to a first merging process to output the Gb median value. The centroid of the Gb median value can be located at Sel[5] and Col[2]. The ISP 100 performs a second merging process on the values of the Gb pixels at Sel[7] and Col[3] and the Gb median value at Sel[5] and Col[2], thereby outputting the merged Gr’ pixel. In this way, the merged R’ pixel and the merged B’ pixel can also be output according to the original image.

[0104] Refer to Figure 11 , B pixel merging is performed. Specifically, two B pixels in the first diagonal direction are received by turning on Sel[2], Sel[7], Col[0], and Col[2] in the pixel array, and the B pixel at Sel[6] and Col[0] is received by turning on Sel[6] and Col[0]. The ADC 240 is commonly connected to Col[0] and Col[2], and the analog end performs a first merging process on the two B pixels in the first diagonal direction, thereby outputting the B median value. The ISP 100 performs a second merging process on the value of the B pixel at Sel[6] and Col[0] and the B median value, thereby outputting the merged B’ pixel.

[0105] Refer to Figure 11 , R pixel merging is performed. Specifically, two R pixels in the first diagonal direction marked by the symbol ☆ in Figure 14 are received by turning on Sel[1], Sel[4], Col[1], and Col[3] in the pixel array, and the R pixel at Sel[0] and Col[3] is received by turning on Sel[0] and Col[3]. The ADC 240 is commonly connected to Col[0] and Col[2], and the analog end performs a first merging process on the two R pixels in the first diagonal direction, thereby outputting the R median value. The ISP 100 performs a second merging process on the value of the R pixel at Sel[0] and Col[3] and the R median value, thereby outputting the merged R’ pixel.

[0106] Figure 15 and Figure 16 show a Bayer image for explaining an image merging method according to some embodiments of the present disclosure.

[0107] The image sensing device 1 can perform 2N×2N merging on the original image. That is, image merging can be performed by repeating 2×2 merging N times. Figure 15 and Figure 16 show performing 4×4 merging by performing 2×2 merging twice.

[0108] Refer to Figure 15, the main 2×2 merging process can be performed in units of the core Ka. Specifically, the original image may include a plurality of cores Ka. The image sensing device 1 may perform the main 2×2 merging process on each of the plurality of cores Ka.

[0109] In the main 2×2 merging process, by performing diagonal merging, the median value is arranged at the position marked by the symbol ○ in the left core of Figure 15 , and by performing corner merging, the merged pixel is arranged at the position marked by the symbol ○ in the right core of Figure 15 . The merged pixel may be located at the centroid of the core Kb. Therefore, since the main 2×2 merging process is performed on the Gr pixel, R pixel, B pixel, and Gb pixel of each core Ka, the Gr' pixel, R' pixel, B' pixel, and Gb' pixel obtained through the main 2×2 merging process may be arranged in the corresponding entire core Kb in the corresponding core Ka, as shown in the left core of Figure 16 .

[0110] Referring to Figure 16 , the auxiliary 2×2 merging process may be performed on the merged image obtained through the main 2×2 merging process in units of the 2×2 array of the core Ka.

[0111] In the auxiliary 2×2 merging process, by performing diagonal merging, the median value is arranged at the position marked by the symbol ○ in the left core of Figure 16 , and by performing corner merging on the median value, the merged pixel is arranged at the position marked by the symbol ○ in the right core of Figure 16 . The merged pixel obtained through the auxiliary merging process may be arranged at the centroid of the core Ka. Therefore, since the auxiliary 2×2 merging process is performed on the Gr' pixel, R' pixel, B' pixel, and Gb' pixel, the Gr pixel, R pixel, B pixel, and Gb pixel obtained through the auxiliary 2×2 merging process are arranged in the corresponding entire core Ka.

[0112] Even if 4×4 merging is performed by executing two 2×2 mergings, the merged pixels can still be evenly arranged at the centroid of the core. Therefore, pseudo-color defects or jagged noises can be effectively reduced.

[0113] Figure 15 And Figure 16 The example embodiments of are for illustrating 4×4 merging, but the embodiments are not limited thereto. For example, according to the example embodiments, 2N×2N merging may be performed by executing N times of 2×2 merging, where N is a natural number.

[0114] Figure 17 Is Figure 1 A perspective view of the image sensing device of.

[0115] Reference Figure 17 , the image sensor 700 can be implemented as a stack of multiple layers. The pixel array 210 can be implemented on the first layer 1, and the rest of the image sensor 200 (e.g., the logic circuit LOGIC) can be implemented on the second layer 2. The logic circuit LOGIC can include all elements of the image sensor 200 except the pixel array 210. For example, the pixel array region and the logic circuit region can be stacked at the wafer level.

[0116] The first layer 1 can include a sensing region SA and a first pad region PA1. The sensing region SA includes a plurality of pixels PX, and the first pad region PA1 is disposed on the periphery of the sensing region SA. A plurality of upper pads PAD1 can be included in the first pad region PA1 and can be connected to the logic circuit LOGIC in the second pad region PA2 of the second layer 2 and the pads PAD21 and PAD22 through vias.

[0117] In an exemplary embodiment, the pads PAD21 and PAD22 can be input I / F pads, and the pad PAD23 can be an output I / F pad.

[0118] Figure 18 is a block diagram of an electronic device including a multi-camera module according to an exemplary embodiment. Figure 19 is Figure 18 a detailed block diagram of the multi-camera module of

[0119] Reference Figure 18 , the electronic device 1000 can include a camera module group 1100 (e.g., a camera component group), an application processor 1200, a power management integrated circuit (PMIC) 1300, and an external memory 1400.

[0120] The camera module group 1100 can include a plurality of camera modules 1100a, 1100b, and 1100c, e.g., a plurality of camera components. Figure 18 shows that the camera module group 1100 includes three camera modules, but the embodiment is not limited thereto. For example, the camera module group 1100 can include two camera modules, or the camera module group 1100 can include n camera modules, where n is a natural number of 4 or greater.

[0121] Hereinafter, reference will be made to Figure 19 to describe the structure of the camera module 1100b. The following description can be directly applied to other camera modules of the camera module group 1100, e.g., the camera modules 1100a and 1100c.

[0122] Reference Figure 19, the camera module 1100b may include a prism 1105, an optical path folding element (OPFE) 1110, an actuator 1130, an image sensing device 1140, and a storage unit 1150.

[0123] The prism 1105 may include a reflecting surface 1107 of a reflective material and may thus change the path of light L incident from the outside.

[0124] In an exemplary embodiment, the prism 1105 may change the path of the light L incident in the first direction X to a second direction Y perpendicular to the first direction X. Further, the prism 1105 may change the path of the light L incident in the first direction X to the second direction Y by rotating the light L from the central axis 1106 of the reflecting surface 1107 in the A direction or the B direction. In this case, the OPFE 1110 may move in a third direction Z perpendicular to the first direction X and the second direction Y.

[0125] In an exemplary embodiment, the maximum rotation angle of the prism 1105 may be 15 degrees or less in the positive A direction and 15 degrees or more in the negative A direction, but the embodiment is not limited thereto.

[0126] In an exemplary embodiment, the prism 1105 may move at an angle of about 20°, about 10° to 20°, or about 15° to about 20° in the positive B or negative B direction. The angles at which the prism 1105 moves in the positive B direction and the negative B direction may be the same or may be almost similar (differing by about 1°).

[0127] In an exemplary embodiment, the prism 1105 may move in a third direction Z that extends parallel to the direction in which the central axis 1106 of the reflecting surface 1107 extends.

[0128] The OPFE 1110 may include, for example, a set of m optical lenses, where m is a natural number. The m optical lenses may move in the second direction Y to change the optical zoom ratio of the camera module 1100b. For example, if the default optical zoom ratio of the camera module 1100b is Z, the optical zoom ratio of the camera module 1100b may be changed to 3Z or 5Z or greater by moving the m optical lenses of the OPFE 1110.

[0129] The actuator 1130 may move the OPFE 1110 or the optical lens to a specific position. For example, the actuator 1130 may adjust the position of the optical lens so that the image sensor 1142 may be located at the focal length of the optical lens for precise sensing.

[0130] The image sensing device 1140 may include an image sensor 1142, control logic 1144, and a memory 1146. The image sensor 1142 may sense an image of a target object using the light L provided thereto via an optical lens. The control logic 1144 may control the general operation of the camera module 1100b. For example, the control logic 1144 may control the operation of the camera module 1100b according to a control signal provided via a control signal line CSLb.

[0131] The memory 1146 may store information necessary for the operation of the camera module 1100b, such as calibration data 1147. The calibration data 1147 may include information necessary for generating image data using the light L. The calibration data 1147 may include, for example, rotation degree information, focal length information, and optical axis information. In the case where the camera module 1100b is implemented as a multi-state camera in which the focal length varies according to the position of the optical lens, the calibration data 1147 may include focal distances and autofocus information for different positions or different states of the optical lens.

[0132] The storage unit 1150 may store the image data sensed by the image sensor 1142. The storage unit 1150 may be provided outside the image sensing device 1140 and may be implemented as stacked on a sensor chip forming the image sensing device 140. In an exemplary embodiment, the storage unit 1150 may be implemented as an electrically erasable programmable read-only memory (EEPROM), but the embodiments are not limited thereto.

[0133] Refer to Figure 18 and Figure 19 , in an exemplary embodiment, the camera modules 1100a, 1100b, and 1100c may include an actuator 1130. According to the operation of the actuator 1130, the camera modules 1100a, 1100b, and 1100c may include the same calibration data 1147 or different calibration data 1147.

[0134] In an exemplary embodiment, one of the camera modules 1100a, 1100b, and 1100c (e.g., the camera module 1100b) may be a folded lens type camera module including a prism 1105 and an OPFE 1110, and the other camera modules (e.g., the camera modules 1100a and 1100c) may be vertical camera modules not including the prism 1105 and the OPFE 1110. However, the embodiments are not limited thereto.

[0135] In an exemplary embodiment, the camera module 1100c may be, for example, a depth camera capable of extracting depth information using infrared (IR) light. In this case, the application processor 1200 may generate a three-dimensional (3D) depth image by fusing the image data provided by the camera module 1100c and the image data provided by other camera modules (e.g., camera modules 1100a and 1100b).

[0136] In an exemplary embodiment, at least two of the camera modules 1100a, 1100b, and 1100c may have different fields of view. In this case, at least two of the camera modules 1100a, 1100b, and 1100c (e.g., camera modules 1100a and 1100b) may have different optical lenses, but the embodiment is not limited thereto.

[0137] In an exemplary embodiment, the camera modules 1100a, 1100b, and 1100c may have different fields of view. In this case, the camera modules 1100a, 1100b, and 1100c may have different optical lenses, but the embodiment is not limited thereto.

[0138] In an exemplary embodiment, the camera modules 1100a, 1100b, and 1100c may be set to be physically separated from each other. For example, the camera modules 1100a, 1100b, and 1100c may not share the sensing area of a single image sensor 1142 jointly, but independent image sensors 1142 may be provided in the camera modules 1100a, 1100b, and 1100c.

[0139] Referring again to Figure 18 , the application processor 1200 may include an image processor 1210, a storage controller 1220, and an internal memory 1230. The application processor 1200 may be implemented separately from the camera modules 1100a, 1100b, and 1100c. For example, the application processor 1200 and the camera modules 1100a, 1100b, and 1100c may be implemented as separate semiconductor chips.

[0140] The image processor 1210 may include a plurality of sub-image processors 1212a, 1212b, and 1212c, an image generator 1214, and a camera module controller 1216.

[0141] The image processor 1210 may include a plurality of sub-image processors corresponding to the plurality of camera modules, e.g., a plurality of sub-image processors 1212a, 1212b, and 1212c corresponding to the camera modules 1100a, 1100b, and 1100c, respectively.

[0142] Image data generated by camera modules 1100a, 1100b, and 1100c can be provided to sub-image processors 1212a, 1212b, and 1212c via separate image signal lines ISLa, ISLb, and ISLb. For example, image data generated by camera module 1100a can be provided to sub-image processor 1212a via image signal line ISLa, image data generated by camera module 1100b can be provided to sub-image processor 1212b via image signal line ISLb, and image data generated by camera module 1100c can be provided to sub-image processor 1212c via image signal line ISLc. Transmission of the image data can be performed via, for example, a camera serial interface (CIS) based on the Mobile Industry Processor Interface (MIPI), but the embodiments are not limited thereto.

[0143] In an exemplary embodiment, a single sub-image processor can be set to correspond to multiple camera modules. For example, sub-image processors 1212a and 1212c can be incorporated into a single integrated sub-image processor, and image data provided by camera module 1100a or image data provided by camera module 1100c can be selected by a selection element (e.g., a multiplexer), and then the image data can be provided to the integrated sub-image processor.

[0144] The image data provided to each of sub-image processors 1212a, 1212b, and 1212c can be provided to image generator 1214. Image generator 1214 can generate an output image using the image data provided to it from each of sub-image processors 1212a, 1212b, and 1212c according to image generation information or a mode signal.

[0145] For example, image generator 1214 can generate an output image by fusing at least some of the image data provided by camera modules 1100a, 1100b, and 1100c having different fields of view according to image generation information or a mode signal. Additionally, image generator 1214 can generate an output image by selecting one of the image data generated by camera modules 1100a, 1100b, and 1100c having different fields of view according to image generation information or a mode signal.

[0146] In an exemplary embodiment, the image generation information can include a zoom signal or a zoom factor. In an exemplary embodiment, the mode signal can be a signal based on a user-selected mode, for example.

[0147] In the case where the image generation information is a zoom signal or a zoom ratio and the camera modules 1100a, 1100b, and 1100c have different fields of view, the image generator 1214 may perform different operations according to the type of the zoom signal. For example, when the zoom signal is a first signal, the image data provided by the camera module 1100a and the image data provided by the camera module 1100c may be fused together, and the fused image data and the image data provided by the camera module 1100b may be used to generate an output image. When the zoom signal is a second signal different from the first signal, one of the image data provided by the camera module 1100a, the image data provided by the camera module 1100b, and the image data provided by the camera module 1100c may be selected, and the selected image data may be used to generate an output image. However, the embodiments are not limited to this example. The method of processing the image data may be changed as needed.

[0148] In an exemplary embodiment, the image generator 1214 may receive a plurality of image data having different exposure times from at least one of the sub-image processors 1212a, 1212b, and 1212c, and may perform high dynamic range processing on the plurality of image data to generate fused image data having an enhanced dynamic range.

[0149] The camera module group 1100 may include Figure 1 the image sensing device 1.

[0150] It may be implemented Figure 19 inside the camera module group 1100, in Figure 18 the sub-image processors 1212a, 1212b, and 1212c, or in Figure 18 the image generator 1214 of Figure 1 the ISP 100.

[0151] The camera module controller 1216 may provide control signals to the camera modules 1100a, 1100b, and 1100c. The control signals provided by the camera module controller 1216 may be provided to the camera modules 1100a, 1100b, and 1100c via separate control signal lines CSLa, CSLb, and CSLc.

[0152] One of the camera modules 1100a, 1100b, and 1100c (e.g., camera module 1100b) can be designated as the main camera according to a pattern signal or image generation information including a zoom signal, and the other camera modules (i.e., camera modules 1100a and 1100c) can be designated as slave cameras. This type of information can be included in a control signal, and then this type of information can be provided to the camera modules 1100a, 1100b, and 1100c via separate control signal lines CSLa, CSLb, and CSLc.

[0153] The main camera module and the slave camera module can be changed according to the zoom ratio or the pattern signal. For example, when camera module 1100a has a wider field of view but a smaller zoom ratio with a smaller zoom multiple compared to camera module 1100b, camera module 1100b can operate as the main module, and camera module 1100a can operate as the slave module. When camera module 1100a has a larger zoom ratio with a larger zoom multiple compared to camera module 1100b, camera module 1100a can operate as the main module, and camera module 1100b can operate as the slave module.

[0154] In an exemplary embodiment, the control signal provided from the slave camera module controller 1216 to the camera modules 1100a, 1100b, and 1100c can include a synchronization enable signal. For example, if camera module 1100b is the main camera and camera modules 1100a and 1100c are slave cameras, the camera module controller 1216 can send a synchronization enable signal to camera module 1100b. Camera module 1100b can receive the synchronization enable signal, can generate a synchronization signal based on the synchronization enable signal, and can provide the synchronization signal to camera modules 1100a and 1100c via the synchronization signal line SSL. The camera modules 1100a, 1100b, and 1100c can send image data to the application processor 1200 in synchronization with the synchronization signal.

[0155] In an exemplary embodiment, the control signal provided from the slave camera module controller 1216 to the camera modules 1100a, 1100b, and 1100c can include pattern information corresponding to the pattern signal provided to the camera modules 1100a, 1100b, and 1100c. The camera modules 1100a, 1100b, and 1100c can operate in one of a first operation mode and a second operation mode associated with the sensing speed according to the pattern information.

[0156] In the first operation mode, the camera modules 1100a, 1100b, and 1100c can generate image signals at a first speed (e.g., generate image signals at a first frame rate), can encode the image signals at a second speed higher than the first speed (e.g., encode the image signals at a second frame rate higher than the first frame rate), and can send the encoded image signals to the application processor 1200. Here, the second speed can be less than 30 times the first speed.

[0157] The application processor 1200 can store the received image signals (e.g., the encoded image signals) in the internal memory 1230 or the external memory 1400, can read the encoded image signals from the internal memory 1230 or the external memory 1400 and decode the encoded image signals, and can display the image data generated based on the decoded image signals. For example, the encoded image signals can be decoded by the sub-processors 1212a, 1212b, and 1212c of the image processing device 1210, and image processing can be performed on the decoded image signals.

[0158] In the second operation mode, the camera modules 1100a, 1100b, and 1100c can generate image signals at a third speed lower than the first speed (e.g., generate image signals at a third frame rate lower than the first frame rate), and can send the image signals to the application processor 1200. The image signals sent to the application processor 1200 can be unencoded signals. The application processor 1200 can perform image processing on the image signals received from the camera modules 1100a, 1100b, and 1100c, or can store the received image signals in the internal memory 1230 or the external memory 1400.

[0159] The PMIC 1300 can supply power, e.g., a power supply voltage, to the camera modules 1100a, 1100b, and 1100c. For example, the PMIC 1300 can supply a first power to the camera module 1100a via the power supply signal line PSLa, a second power to the camera module 1100b via the power supply signal line PSLb, and a third power to the camera module 1100c via the power supply signal line PSLc under the control of the application processor 1200.

[0160] The PMIC 1300 can generate power corresponding to each of the camera modules 1100a, 1100b, and 1100c in response to a power control signal PCON from the application processor 1200 and control the level of the power. The power control signal PCON can include power control signals for different operation modes of the camera modules 1100a, 1100b, and 1100c. For example, the operation modes of the camera modules 1100a, 1100b, and 1100c can include a low power mode, in which case the power control signal PCON can include information indicating which camera module will operate in the low power mode and information indicating the power level to be set. The provided power level can be the same for all camera modules 1100a, 1100b, and 1100c, or can be different for the camera modules. Additionally, the provided power level can vary dynamically.

[0161] Although the present disclosure has been shown and described with reference to exemplary embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope defined by the appended claims and their equivalents.

Claims

1. An image sensing device, comprising: An image sensor, comprising: A pixel array configured to output a raw image having a Bayer pattern; and An analog end configured to perform an analog merging process on a group of pixels of the same color in the raw image and output a median value for the corresponding color, the group of pixels of the same color including those in the same column of each of a plurality of sub-kernels corresponding to a first green pixel, a red pixel, a blue pixel, and a second green pixel; and A digital signal processor configured to perform a digital merging process on the median values for the corresponding color included in different columns of each of the plurality of sub-kernels and output a merged image, Wherein, the analog merging process and the digital merging process are performed on the raw image in units of unit kernels, each unit kernel including the plurality of sub-kernels, and the plurality of sub-kernels are arranged such that the sub-kernels included in the same column are arranged asymmetrically with respect to each other and do not overlap with each other in the vertical direction, or the sub-kernels included in the same row are arranged asymmetrically with respect to each other and do not overlap with each other in the horizontal direction.

2. The image sensing device according to claim 1, wherein, The sum of the lengths of the first sub-kernel in the first direction and the length of the second sub-kernel in the first direction is equal to the length of the unit kernel in the first direction, and Wherein, the lengths of the first sub-kernel and the second sub-kernel are the closest odd numbers.

3. The image sensing device according to claim 1, wherein The analog end includes: An analog-to-digital converter connected to a plurality of columns of the pixel array; and A buffer connected to the analog-to-digital converter and configured to store the median value, and Wherein, the analog-to-digital converter is configured to: digitize the sum of the values received from at least two pixels of the same color included in the same column and output the digitized result as one of the median values.

4. The image sensing device according to claim 1, wherein The digital merging process further includes: summing the median values included in different columns of the sub-kernels corresponding to a predetermined color, dividing the result of the summation by the number of pixels of the predetermined color included in the sub-kernels corresponding to the predetermined color, and outputting the result of the division as the pixel value for the predetermined color in the merged image.

5. The image sensing device according to claim 4, wherein, The center of gravity of the pixels for the predetermined color coincides with the center of gravity of the sub-kernels corresponding to the predetermined color.

6. An image merging method for an image sensing device, the image merging method comprising: Receiving a raw image with a Bayer pattern from a pixel array; Outputting a median value for each unit kernel based on performing an analog merging process on the raw image of unit kernels having an asymmetric arrangement with respect to different colors; And Outputting a merged image based on performing a digital merging process on at least one of the median values, Wherein, the raw image includes a plurality of unit kernels, each unit kernel including a plurality of sub-kernels, and the plurality of sub-kernels are arranged such that the sub-kernels included in the same column are arranged asymmetrically with respect to each other and do not overlap with each other in the vertical direction, or the sub-kernels included in the same row are arranged asymmetrically with respect to each other and do not overlap with each other in the horizontal direction.

7. The image merging method according to claim 6, wherein, The plurality of sub-kernels include: a first green sub-core and a red sub-core, the first green sub-core and the red sub-core being arranged adjacent to each other in a row direction and having different lengths in the row direction; and a blue sub-core and a second green sub-core, the blue sub-core and the second green sub-core being arranged adjacent to the first green sub-core and the red sub-core in a column direction and adjacent to each other in the row direction, the blue sub-core and the second green sub-core having different lengths in the row direction, wherein the first green sub-core and the blue sub-core have different lengths in the column direction, and wherein the red sub-core and the second green sub-core have different lengths in the column direction.

8. The image merging method according to claim 6, wherein, The length of a first sub-core among the plurality of sub-cores in a column direction or a row direction and the sum of the lengths of a second sub-core among the plurality of sub-cores in the column direction or the row direction are the same as the length of a unit core in the column direction or the row direction, the first sub-core and the second sub-core being adjacent to each other in the column direction or the row direction, and wherein the lengths of the first sub-core and the second sub-core are the closest odd numbers.

9. The image merging method according to claim 6, wherein, The analog merging process further includes: summing pixel values received from at least two first color pixels and outputting a first median, the at least two first color pixels being included in the same column of a first sub-core corresponding to a first color among the plurality of sub-cores, and The digital merging process further includes: summing the first medians included in different columns of the first sub-core, dividing the result of the summing by the number of first color pixels included in the first sub-core, and outputting the result of the division as the pixel value for the first color in the merged image.

10. An image merging method of an image sensing device, the image merging method including: receiving a raw image of a Bayer pattern from a pixel array; outputting a median for each unit core based on performing an analog merging process on the raw image of unit cores having an asymmetric arrangement with respect to different colors; and outputting a merged image based on performing a digital merging process on at least one of the medians, wherein the analog merging process further includes: summing the values of a first pixel and a second pixel of the same color arranged opposite to each other in a first diagonal direction in a unit core, and outputting the result of the summing as the median, and wherein the digital merging process further includes: summing the median and the value of a third pixel having the same color as the first pixel and the second pixel and outputting the result of the summing as the pixel value for the color of the first pixel, the second pixel, and the third pixel in the merged image.

11. The image merging method according to claim 10, wherein, The raw image includes a plurality of unit cores, and wherein each of the plurality of unit cores has a length of (4n + 4) pixels in a column direction and a row direction, where n is a natural number.

12. The image merging method according to claim 10, wherein, The third pixel and the median are arranged in a second diagonal direction perpendicular to the first diagonal direction.

13. The image merging method according to claim 10, wherein, The center of gravity of the merged pixel is set at the center of gravity among the first pixel, the second pixel, and the third pixel.

14. The image merging method according to claim 10, wherein, The merged image is an image obtained by performing the following operations at least twice: performing the analog merging process and the digital merging process on the original image.

15. An image sensing device, comprising: An image sensor, comprising: A pixel array configured to output a raw image in a Bayer pattern, and An analog end configured to perform an analog merging process on a first pixel and a second pixel arranged in a diagonal direction for each color in each square unit kernel of the raw image, and output a median value for the corresponding color; and A digital signal processor configured to obtain a merged pixel value based on performing a digital merging process on the median value and a third pixel of the corresponding color arranged at a corner in each of the unit kernels, output the merged pixel value, and generate a merged image including the merged pixel value.

16. The image sensing device according to claim 15, wherein, The analog end includes: A plurality of analog-to-digital converters shared by two adjacent odd-numbered columns and two adjacent even-numbered columns in the pixel array, and A buffer connected to the plurality of analog-to-digital converters and configured to store the median value.

17. The image sensing device according to claim 15, wherein, Each of the unit kernels has a length of (4n + 4) pixels in the column direction and the row direction, where n is a natural number.

18. The image sensing device according to claim 15, wherein, The center of gravity of the merged pixel is set at the center of gravity among the first pixel, the second pixel, and the third pixel.

19. The image sensing device according to claim 15, wherein, The merged image is an image obtained by performing both the analog merging process and the digital merging process at least once.

Images