Pixel array for reducing image information loss and image sensor including the same

Abstract

To provide a pixel array for reducing image information loss and an image sensor including the same.SOLUTION: In a pixel array of an image sensor according to the present invention, the pixel array includes a plurality of color filter array (CFA) cells each having a predetermined size, each of the CFA cells including a plurality of CFA blocks arranged in width and length directions. Each of the CFA blocks includes a sub block, which is located at a central region and includes m*n color pixels, and an outer region where color pixels other than the sub block are arranged. The m*n color pixels of the sub block include color pixels sensing first through third colors. The outer region includes first pixels sensing the first color and second pixels sensing at least one selected from the second color and the third color, the number of first pixels being greater than that of the second pixels.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an image sensor, and more particularly, to a pixel array for reducing video information loss and an image sensor including the same.

Background Art

[0002] An image sensor is a device that captures a two-dimensional or three-dimensional image of an object. The image sensor generates an image of the object using a photoelectric conversion element that reacts to the intensity of light reflected from the object. Recently, CMOS (Complementary Metal-Oxide Semiconductor) technology has developed, and CMOS image sensors (CIS) using CMOS are widely used.

[0003] An image sensor includes a pixel array. As the image sensor is implemented with a high number of pixels, the pixel array includes more color pixels. In order to convert the raw video output from the image sensor into a predetermined pattern such as an RGB video, a Remosaic process based on interpolation and / or extrapolation is performed. However, there is a problem that video information loss occurs when the distance to adjacent color pixels, which is referred to in such a process, increases.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The present invention has been made in view of the above-described conventional problems, and an object of the present invention is to prevent an increase in video loss during processing of a video captured by an image sensor.

Means for Solving the Problem

[0006] According to one aspect of the present invention made to achieve the above object, a pixel array of an image sensor in the pixel array of the image sensor, the pixel array includes a plurality of color filter array (CFA) cells each having a predetermined size, and each CFA cell includes a plurality of CFA blocks arranged in horizontal and vertical directions. Each of the plurality of CFA blocks includes a sub-block located in a center region and having m*n color pixels, and an outer region where color pixels other than the sub-block are arranged. The m*n color pixels of the sub-block include color pixels that sense first to third colors, and the outer region includes a relatively small number of second pixels that sense at least one of the second color or the third color together with a relatively large number of first pixels that sense the first color.

Advantages of the Invention

[0007] According to the pixel array and the image sensor including the same according to the technical idea of the present invention, by suitably designing the arrangement of color pixels that sense a plurality of colors in a CFA block of a predetermined size, the distance between color pixels referred to during interpolation processing is reduced, thereby reducing the loss of video information.

[0008] Also, according to the pixel array and the image sensor including the same according to the technical idea of the present invention, since signals of a plurality of identical color pixels can be added together in each CFA block during binning processing, there is an effect of reducing the loss of video information generated during the binning processing.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11A

Figure 11B

Figure 11C

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0011] FIG. 1 is a block diagram showing a pixel array and an image sensor including the same according to an embodiment of the present invention.

[0012] As shown in FIG. 1, the image sensor 100 includes a pixel array 110, a row driver 120, a readout circuit 130, and a controller 140. The image sensor 100 is a CMOS (Complementary Metal-Oxide Semiconductor) image sensor (CIS).

[0013] The controller 140 controls the loud driver 120 and the readout circuit 130. The pixel array 110 includes a plurality of pixels (e.g., color pixels), and each pixel includes one or more light sensing elements (not shown). The light sensing elements sense light at each pixel and generate an electrical signal according to the intensity of the sensed light. The light sensing elements include photodiodes, photogates, and photo-transistors, etc. Also, the pixel array 110 includes color pixels having various patterns according to an embodiment of the present invention, and at each color pixel, an electrical signal related to one or more colors is generated as a pixel signal. Also, although not shown in FIG. 1, processing operations such as remosaic are performed on the pixel signals from the pixel array 110, and through this processing operation, the color pattern of the pixel array 110 is converted into a specific pattern such as a Bayer pattern.

[0014] The pixel array 110 outputs an electrical signal caused by the light absorbed by the light sensing elements to the readout circuit 130. The loud driver 120 outputs a signal for controlling each color pixel included in the pixel array 110. For example, the loud driver 120 outputs a signal for resetting the light sensing elements provided in each color pixel or for controlling to output an electrical signal caused by the photoelectric charge accumulated by the light sensing elements to the outside.

[0015] The readout circuit 130 receives the electrical signal output from the pixel array 110 and outputs a pixel value (or pixel data). For example, the readout circuit 130 includes an ADC (analog-to-digital converter) and outputs a digital signal corresponding to the analog signal received from the pixel array 110 as pixel data.

[0016] On the other hand, although not shown in FIG. 1, pixel data from the image sensor 100 is provided to an image processing device (not shown), and processing operations such as remote mosaic are performed in the image processing device based on digital signal processing. According to one embodiment, processing operations such as remote mosaic are defined as being performed by components (e.g., a processor) within the image sensor 100 or by a separate processing device provided outside the image sensor 100.

[0017] Hereinafter, an embodiment of a color pixel provided in the pixel array 110 according to an embodiment of the present invention is shown.

[0018] FIGS. 2 and 3 show an embodiment of a pixel array according to an embodiment of the present invention.

[0019] Referring to FIGS. 1 to 3, the image sensor 100 includes a pixel array 110, and a color filter array (CFA) is arranged in the pixel array 110 so that a color of a predetermined component is sensed for each pixel. Hereinafter, in describing embodiments of the present invention, the terms color filter, color pixel, filter array, and pixel array are variously defined. For example, the CFA is defined as a separate configuration arranged on a pixel array including light sensing elements, or as a configuration provided in the pixel array. That is, each color pixel is defined as including a corresponding color filter. Also, in describing the following embodiments, each of the CFA cells, CFA blocks, and sub-blocks is defined as including a color pixel.

[0020] The pixel array 110 includes a plurality of CFA cells 111 defined in a predetermined unit. As an example, the pixel array 110 includes a plurality of CFA cells 111 repeatedly arranged along the horizontal and vertical directions, and each CFA cell 111 includes a color pixel having a predetermined size.

[0021] On the one hand, the CFA cell 111 is defined as comprising a plurality of CFA blocks, and also refers to the structure of the minimum size in which the same CFA block is repeated. In FIG. 2, an example is illustrated in which, as a plurality of CFA blocks, M CFA blocks are arranged in the horizontal direction and N CFA blocks are arranged in the vertical direction in the CFA cell 111, and the CFA cell 111 includes M*N CFA blocks. On the other hand, in a high-quality image sensor such as a CIS, the number of color pixels increases, and as a result, the sizes of the CFA cell 111 and the CFA blocks increase.

[0022] At this time, when each CFA block includes the same type of color pixels (or when all CFA blocks include pixels that sense the same color), when the raw video output from the image sensor is converted into an RGB video, during video processing such as interpolation and extrapolation with each color pixel (for example, the center color pixel), as the distance to the adjacent color pixel to be referenced increases, loss of video information occurs. In particular, the larger the size of the CFA block, the greater the distance to the adjacent color pixel to be referenced, resulting in a problem that the video loss rate also increases.

[0023] In the pixel array 110 according to an embodiment of the present invention, the color pixels (or color filters) of the CFA cell 111 and the CFA blocks have a pattern that reduces their loss rate during the video processing. As an embodiment, in the embodiment described below, each CFA cell 111 includes 4 (2*2) CFA blocks, and each CFA block senses at least two or more colors instead of a single color, or an embodiment is proposed in which one CFA block senses all the colors applied to the pixel array 110, and an embodiment is proposed in which the loss of video information is reduced through a reduction in the interpolation distance.

[0024] According to one embodiment, as shown in FIG. 3, the CFA block includes a plurality of color pixels. As an example, c color pixels are arranged horizontally and d color pixels are arranged vertically, and one CFA block may include c*d color pixels. Assume that the pixel array 110 includes red color pixels, blue color pixels, and green color pixels (hereinafter referred to as red pixels, blue pixels, and green pixels respectively). In one embodiment of the present invention, each CFA block includes all of the above-described red pixels, blue pixels, and green pixels.

[0025] In one embodiment, a unit including a certain number of pixels inside (or the center region) of the CFA block is referred to as a sub-block, and the sub-block includes a*b color pixels. For example, as shown in FIG. 3, in the sub-block, a red pixel R, a blue pixel B, and a green pixel G are all arranged.

[0026] Also, in each CFA block, a plurality of color pixels are arranged in the remaining region excluding the sub-block (for example, the boundary (or outer contour) region of the CFA block). As an example, in the outer contour region, pixels for sensing a specific color are mainly arranged, and pixels for sensing a small number (for example, one or two) of other colors are also arranged. That is, the outer contour region of each CFA block includes a plurality of first color pixels P1 for sensing a specific color and at least one second pixel P2 for sensing other colors.

[0027] In one embodiment, depending on the type of color pixels included in the CFA block, the CFA block is referred to as a red CFA block, a blue CFA block, or a green CFA block. As an example, in the CFA block, if the number of red pixels is the largest, the CFA block is referred to as a red CFA block. Alternatively, in the CFA block, if the first pixel P1 in the outer region is a red pixel, the CFA block is referred to as a red CFA block.

[0028] In one embodiment, in the outer region of the red CFA block, the first pixel (or red pixel) is mainly arranged, and at least one second pixel (for example, a blue pixel) is further arranged in the outer region. For example, in the outer region of the red CFA block, generally red pixels are arranged, but blue pixels are arranged at at least one corner of the outer region. Similarly, in the outer region of the blue CFA block, generally blue pixels are arranged, but red pixels are arranged at at least one corner of the outer region. Also, depending on various embodiments, only green pixels are arranged in the outer region of the green CFA block, or at least one pixel for sensing other colors is also arranged in the outer region of the green CFA block.

[0029] On the other hand, in the above embodiment, the red pixels, blue pixels, and green pixels applied to the pixel array 110 are one exemplary matter, and the present invention is not limited thereto. For example, color pixels using various types of filters such as cyan filters and RGBW filters may be applied to the embodiments of the present invention, and the embodiments of the present invention are not limited to a pattern for sensing a specific color.

[0030] Hereinafter, a specific embodiment of the pixel array of the image sensor according to an embodiment of the present invention will be described. In the following embodiments, the case where the above-described second pixel P2 is disposed at one corner of the outer peripheral region of the CFA block is exemplified, but the embodiments of the present invention are not limited thereto. That is, as described above, the embodiments of the present invention are variously modified such that two or more second pixels P2 are disposed in the outer peripheral region and the second pixel P2 is disposed at a position other than the corner of the outer peripheral region.

[0031] FIG. 4 is a diagram showing a pixel array and a remosaicked color pattern according to an embodiment of the present invention. Hereinafter, a CFA cell including 8×8 pixels will be described, but the size of the CFA cell may be variously set in the embodiments of the present invention.

[0032] According to one embodiment, the CFA cell includes four CFA blocks, and each CFA block includes color pixels of 4×4 size. As an example, when the above-described pixel array includes red, green, and blue pixels, each of the CFA blocks includes all of red, green, and blue pixels, and the CFA cell shown in FIG. 4(e) includes the red CFA block shown in FIG. 4(a), the green CFA block shown in FIG. 4(b), and the blue CFA block shown in FIG. 4(c). Further, the CFA cell of 8×8 size is converted into a Bayer pattern as shown in FIG. 4(f) through a remosaicking process, and the above-described CFA cell and the converted Bayer pattern are referred to as a pattern CFA pair (or a CFA pattern pair).

[0033] Further, each of the red CFA block, the green CFA block, and the blue CFA block includes sub-blocks of a predetermined size as shown in FIG. 4(d). As an example, the sub-blocks include color pixels of 2*2 size. Further, according to an embodiment, the sub-blocks include both red, green, and blue pixels, so that red, green, and blue colors are sensed together through the sub-blocks. Further, the pixel arrangement within the sub-blocks is embodied in various ways. As an example, in FIG. 4(d), the case where green pixel - blue pixel - red pixel - green pixel (GBRG) are arranged in the order of upper left - upper right - lower left - lower right is illustrated.

[0034] On the other hand, various forms of CFA cells are embodied based on the arrangement of the four CFA blocks. As an example, in the CFA cell shown in FIG. 4(e), the case where green - red - blue - green CFA blocks are arranged in the order of upper left - upper right - lower left - lower right is illustrated. In the embodiments of the present invention, for convenience of explanation, the 8*8 CFA cell as shown in FIG. 4(e) is referred to as a GR-GB pattern cell using the first two characters of the GRBG CFA block and the GBRG pixels of the sub-blocks, respectively.

[0035] On the other hand, in order to remosaic the 8*8 CFA cell into a Bayer pattern by applying the pattern pairs of FIGS. 4(e) and 4(f), interpolation of red, green, and blue colors is required. When performing interpolation of the green color, since the green color contains more edge information of the video than the red color and the blue color, relatively more adjacent green pixels are selected during the interpolation process. For example, for each of the four directions (as an example, the slash / reverse slash direction, the horizontal / vertical direction), the adjacent green pixels required for interpolation are selected, and the average of the distances between the pixel for which the interpolation is calculated (for example, the center pixel) and the adjacent green pixels is calculated as the interpolation distance at the center pixel. On the other hand, when performing interpolation of the blue and red colors, one or more adjacent pixels are selected regardless of the direction around the pixel for which the interpolation is calculated.

[0036] According to one embodiment of the present invention, at least one of a plurality of CFA blocks is provided with pixels that sense at least two colors in an outer peripheral region. For example, as shown in FIG. 4(a), the outer peripheral region of the red CFA block mainly includes red pixels, but may also include one second pixel (for example, the blue pixel P2 at the lower left corner). Further, as shown in FIG. 4(c), the outer peripheral region of the blue CFA block mainly includes blue pixels, but may also include one second pixel (for example, the red pixel P2 at the upper right corner). On the other hand, in the case of the green CFA block, an example is shown in which all the pixels in the outer peripheral region are green pixels, but at least one second pixel P2 may also be arranged in the outer peripheral region of the green CFA block.

[0037] Referring to FIGS. 4(e) and 4(f), the arrangement of the color pixels in the sub-blocks within each CFA block is the same as the arrangement of the color pixels at the positions corresponding to the Bayer pattern. Thereby, in one CFA cell, the interpolation distance of each of the color pixels in the four sub-blocks is calculated to be 0, and thereby the effect of reducing the interpolation distance is obtained.

[0038] Further, with respect to the four CFA blocks provided in one CFA cell, a first region DR including one color pixel of each of the four CFA blocks is defined. The first region DR includes the second pixel P2 of the blue CFA block, the second pixel P2 of the red CFA block, and two green pixels in the two green CFA blocks. Further, the arrangement of the color pixels in the first region DR is the same as the arrangement of the color pixels at the positions corresponding to the Bayer pattern. Thereby, in one CFA cell, the number of color pixels having an interpolation distance of 0 increases, and through this, the effect of further reducing the interpolation distance of the CFA cell is obtained.

[0039] That is, according to one embodiment of the present invention, due to the pixel arrangement of each CFA cell, there are color pixels having an interpolation distance of 0, and as in the above-described embodiment, a plurality of color pixels having an interpolation distance of 0, such as the color pixels of the sub-block and the color pixels of the first region DR, are further ensured. As a result, the number of color pixels having an interpolation distance of 0 in each CFA cell can be increased, which prevents an increase in the loss of video information.

[0040] FIG. 5 is a diagram showing an embodiment of a CFA cell according to various embodiments of the present invention. In FIG. 5, the CFA cell includes 2×2 CFA blocks, and as shown in FIGS. 5(a) to 5(d), examples in which a red CFA block, a green CFA block, a blue CFA block, and a sub-block are embodied in the same manner as in the case shown in FIG. 4 are illustrated. In addition, each of the CFA cells shown in FIG. 5 is converted into the Bayer pattern shown in FIG. 5(h) through remosaicking processing.

[0041] Referring to the CFA cell of FIG. 5(e), it includes a green CFA block arranged at the upper left, a blue CFA block arranged at the upper right, a red CFA block arranged at the lower left, and a green CFA block arranged at the lower right. In addition, each of the green CFA block, the red CFA block, and the blue CFA block includes a sub-block, and the sub-block includes green pixels, blue pixels, red pixels, and green pixels in the order of upper left - upper right - lower left - lower right.

[0042] In addition, the CFA cell is variously embodied depending on the arrangement form of the CFA blocks. As an example, in the example of FIG. 5(f), the CFA cell includes a red CFA block arranged at the upper left, a green CFA block arranged at the upper right, a green CFA block arranged at the lower left, and a blue CFA block arranged at the lower right. In addition, in the example of FIG. 5(g), the CFA cell includes a blue CFA block arranged at the upper left, a green CFA block arranged at the upper right, a green CFA block arranged at the lower left, and a red CFA block arranged at the lower right.

[0043] According to the above-described embodiment, the interpolation distance of the color pixels provided in each sub-block of the CFA block is zero. Also, a first region DR composed of one color pixel located in each outer peripheral region of the four CFA blocks is defined. The first region DR includes a second pixel (for example, a red pixel) arranged in the outer peripheral region of the blue CFA block, a second pixel (for example, a blue pixel) arranged in the outer peripheral region of the red CFA block, and green pixels arranged in the outer peripheral regions of the two green CFA blocks. Also, the interpolation distance of the color pixels provided in the first region DR has a value of zero.

[0044] According to the embodiment example of the CFA cell shown in FIG. 5, in the example of FIG. 5(e), the four color pixels located at the four corners of the CFA cell constitute the first region DR. In the example of FIG. 5(f), the four color pixels located at the centers of the left and right sides of the CFA cell constitute the first region DR. In the example of FIG. 5(g), the four color pixels located at the centers of the upper and lower sides of the CFA cell constitute the first region DR.

[0045] FIG. 6 is a diagram showing an example of the interpolation distance of each of the color pixels provided in the CFA cell shown in FIG. 4.

[0046] According to an embodiment of the present invention, when the above-exemplified calculation method is applied to the color pixels for which the remosaic process is performed to calculate the value of the interpolation distance, the values shown in FIG. 6 are calculated. As an example, when interpolating a green color, in each of the four directions (for example, slash / reverse slash / horizontal / vertical directions), the two adjacent green pixels closest to each other are selected. When interpolating each of the red and blue colors, regardless of the specific direction, the two adjacent red pixels or blue pixels closest to each other are selected.

[0047] As shown in FIG. 6, for the color pixels in the sub-block SB of the CFA cell and the color pixels in the first region DR, their interpolation distances are calculated to be 0. Also, for the remaining color pixels, by using the pattern pairs in FIGS. 4(e) and 4(f), the distances to the adjacent color pixels that must be referenced for interpolation are reduced, thereby minimizing the overall average of the interpolation distances calculated in the CFA cell and minimizing the loss of video information.

[0048] Hereinafter, examples of various types of color filter arrays implemented according to an embodiment of the present invention are shown. The embodiments of the present invention do not necessarily have to be limited to the specific examples of the color filter arrays shown below. As long as the effects intended by the present invention are achieved, the specific pixel arrangements of the color filter arrays can be partially deformed.

[0049] FIGS. 7 to 9 are diagrams showing an embodiment of a pixel array according to various embodiments of the present invention. FIGS. 7 to 9 illustrate various arrangement examples of color pixels in sub-blocks and various arrangement examples of color pixels in the outer regions of the respective CFA blocks.

[0050] The red CFA block, the green CFA block, and the blue CFA block are implemented as shown in FIGS. 7(a) to 7(c), and each CFA block includes the sub-block shown in FIG. 7(d). Also, each of the CFA cells shown in FIG. 7 is converted into the Bayer pattern shown in FIG. 7(i) through demosaicing.

[0051] In the embodiment of FIG. 7, the sub-blocks of each CFA block include green pixels, red pixels, blue pixels, and green pixels in the order of top left - top right - bottom left - bottom right. Also, as shown in FIG. 7(a), red pixels are mainly arranged in the outer contour region of the red CFA block. The outer contour region of the red CFA block includes one blue pixel as the second pixel, and the blue pixel is arranged at the upper right corner of the outer contour region of the red CFA block. Also, as shown in FIG. 7(b), green pixels are arranged in the outer contour region of the green CFA block. As an example, no pixels for sensing other colors are arranged in the outer contour region of the green CFA block. Also, as shown in FIG. 7(c), blue pixels are mainly arranged in the outer contour region of the blue CFA block. The outer contour region of the blue CFA block includes one red pixel as the second pixel, and the red pixel is arranged at the lower left corner of the outer contour region of the blue CFA block.

[0052] With the embodiments of the CFA block as described above, in each of the red CFA block and the blue CFA block, the color pixels have a predetermined pattern. For example, in the direction from the upper right corner to the lower left corner of each of the red CFA block and the blue CFA block, blue - red - blue - red pixels are sequentially arranged.

[0053] On one hand, various forms of CFA cells are realized based on the arrangement of CFA blocks. As an example, referring to Fig. 7(e), the CFA cell includes a green CFA block arranged in the upper left, a red CFA block arranged in the upper right, a blue CFA block arranged in the lower left, and a green CFA block arranged in the lower right. Or, referring to Fig. 7(f), the CFA cell includes a green CFA block arranged in the upper left, a blue CFA block arranged in the upper right, a red CFA block arranged in the lower left, and a green CFA block arranged in the lower right. Or, referring to Fig. 7(g), the CFA cell includes a red CFA block arranged in the upper left, a green CFA block arranged in the upper right, a green CFA block arranged in the lower left, and a blue CFA block arranged in the lower right. Or, referring to Fig. 7(h), the CFA cell includes a blue CFA block arranged in the upper left, a green CFA block arranged in the upper right, a green CFA block arranged in the lower left, and a red CFA block arranged in the lower right.

[0054] Similar to the above-described embodiments, the interpolation distance of the color pixels provided in each sub-block of the CFA block has a value of 0, and a first region DR having color pixels with an interpolation distance of 0 is further ensured. For example, according to the illustrated examples of the CFA cells in Fig. 7, in the example of Fig. 7(e), the four color pixels located at the four corners of the CFA cell constitute the first region DR; in the example of Fig. 7(f), the four color pixels located at the center of the CFA cell constitute the first region DR; in the example of Fig. 7(g), the four color pixels located at the centers of the upper and lower sides of the CFA cell constitute the first region DR; and in the example of Fig. 7(h), the four color pixels located at the centers of the left and right sides of the CFA cell constitute the first region DR.

[0055] On the one hand, FIG. 8 shows another embodiment of the CFA cell. The red CFA block, the green CFA block, and the blue CFA block are embodied as shown in FIGS. 8(a) to 8(c), and each CFA block includes the sub-blocks shown in FIG. 8(d). Also, each of the CFA cells shown in FIG. 8 is converted into the Bayer pattern shown in FIG. 8(i) through the remosaic process.

[0056] In the embodiment of FIG. 8, the sub-blocks of each CFA block include blue pixels, green pixels, green pixels, and red pixels in the order of top left - top right - bottom left - bottom right. Also, as shown in FIG. 8(a), red pixels are mainly arranged in the outer region of the red CFA block. The outer region of the red CFA block includes one blue pixel as the second pixel, and the blue pixel is arranged at the bottom right corner of the outer region of the red CFA block. Also, as shown in FIG. 8(b), green pixels are arranged in the outer region of the green CFA block, and no pixels for sensing other colors are arranged in the outer region of the green CFA block. Also, as shown in FIG. 8(c), blue pixels are mainly arranged in the outer region of the blue CFA block. The outer region of the blue CFA block includes one red pixel as the second pixel, and the red pixel is arranged at the top left corner of the outer region of the blue CFA block.

[0057] On the one hand, CFA cells in various forms are embodied similar to the above-described embodiments. As an example, referring to Fig. 8(e), the CFA cell includes a green CFA block arranged in the upper left, a red CFA block arranged in the upper right, a blue CFA block arranged in the lower left, and a green CFA block arranged in the lower right. Also, referring to Fig. 8(f), the CFA cell includes a green CFA block arranged in the upper left, a blue CFA block arranged in the upper right, a red CFA block arranged in the lower left, and a green CFA block arranged in the lower right. Or, referring to Fig. 8(g), the CFA cell includes a red CFA block arranged in the upper left, a green CFA block arranged in the upper right, a green CFA block arranged in the lower left, and a blue CFA block arranged in the lower right. Or, referring to Fig. 8(h), the CFA cell includes a blue CFA block arranged in the upper left, a green CFA block arranged in the upper right, a green CFA block arranged in the lower left, and a red CFA block arranged in the lower right.

[0058] Also, according to the illustrated examples of the CFA cell in Fig. 8, in the example of Fig. 8(e), four color pixels located at the centers on the left and right sides of the CFA cell constitute the first region DR. In the example of Fig. 8(f), four color pixels located at the centers on the upper and lower sides of the CFA cell constitute the first region DR. In the example of Fig. 8(g), four color pixels located at the center of the CFA cell constitute the first region DR. In the example of Fig. 8(h), an example in which four color pixels located at the four corners of the CFA cell constitute the first region DR is illustrated.

[0059] On the other hand, Fig. 9 shows still other illustrated examples of the CFA cell. The red CFA block, green CFA block, and blue CFA block are embodied as shown in Figs. 9(a) to 9(c), and each CFA block includes sub-blocks shown in Fig. 9(d). Also, each of the CFA cells shown in Fig. 9 is converted into the Bayer pattern shown in Fig. 9(i) through remosaic processing.

[0060] In the embodiment of FIG. 9, the sub-blocks of each CFA block include a red pixel, a green pixel, a green pixel, and a blue pixel in the order of top left - top right - bottom left - bottom right. Also, as shown in FIG. 9(a), red pixels are mainly arranged in the outer region of the red CFA block. The outer region of the red CFA block includes one blue pixel as the second pixel, and the blue pixel is arranged at the upper left corner of the outer region of the red CFA block. Also, as shown in FIG. 9(b), green pixels are arranged in the outer region of the green CFA block, and no pixels for sensing other colors are arranged in the outer region of the green CFA block. Also, as shown in FIG. 9(c), blue pixels are mainly arranged in the outer region of the blue CFA block. The outer region of the blue CFA block includes one red pixel as the second pixel, and the red pixel is arranged at the lower right corner of the outer region of the blue CFA block.

[0061] On the other hand, various forms of CFA cells are implemented similar to the above-described embodiments. As an example, referring to FIG. 9(e), the CFA cell includes a green CFA block arranged at the upper left, a red CFA block arranged at the upper right, a blue CFA block arranged at the lower left, and a green CFA block arranged at the lower right. Also, referring to FIG. 9(f), the CFA cell includes a green CFA block arranged at the upper left, a blue CFA block arranged at the upper right, a red CFA block arranged at the lower left, and a green CFA block arranged at the lower right. Or, referring to FIG. 9(g), the CFA cell includes a red CFA block arranged at the upper left, a green CFA block arranged at the upper right, a green CFA block arranged at the lower left, and a blue CFA block arranged at the lower right. Or, referring to FIG. 9(h), the CFA cell includes a blue CFA block arranged at the upper left, a green CFA block arranged at the upper right, a green CFA block arranged at the lower left, and a red CFA block arranged at the lower right.

[0062] Also, according to the embodiment example of the CFA cell shown in FIG. 9, in the example of FIG. 9(e), four color pixels located at the centers of the upper and lower sides of the CFA cell constitute the first region DR. In the example of FIG. 9(f), four color pixels located at the centers of the left and right sides of the CFA cell constitute the first region DR. In the example of FIG. 9(g), four color pixels located at the four corners of the CFA cell constitute the first region DR. In the example of FIG. 9(h), an example in which four color pixels located at the center of the CFA cell constitute the first region DR is illustrated.

[0063] According to an embodiment of the present invention by the above-described embodiment, the color pattern of the pixel array is configured in various ways. As an example, in a CFA block embodied in a predetermined size, a plurality of colors (for example, R, G, B colors) are sensed together, so the distance to adjacent color pixels used during interpolation and / or extrapolation processing is reduced, and video loss is reduced. Also, in each CFA block, by configuring the color pixels in the outer region according to the above-described embodiment, the number of color pixels having an interpolation distance of 0 within the CFA cell can be increased, which has the effect of reducing video loss during the color conversion process because it reduces the overall interpolation distance of the CFA cell. Also, in the above-described embodiment, it was explained that the outer region of one CFA block includes one different color pixel, but in order to achieve the effects of the embodiments of the present invention, various other modification examples are possible for the outer region of the CFA block. Also, in the above-described embodiment, an example in which no second pixel for sensing other colors is arranged in the outer region of the green CFA block was explained, but according to various embodiments of the present invention, the CFA cell may be embodied such that a second pixel for sensing other colors is also arranged in the outer region of the green CFA block.

[0064] FIG. 10 is a block diagram showing an example of a video processing apparatus including an image sensor according to the present invention.

[0065] Referring to FIG. 10, the video processing apparatus 200 includes a pixel array 210 having CFA cells, and also includes a video processor 220 that performs video processing using pixel values from the pixel array 210. According to one embodiment, the above-described image sensor is defined as including the pixel array 210 and at least a part of the configuration provided in the video processor 220. For example, a configuration for binning processing and / or a configuration for remosaic processing are defined as being provided in the image sensor.

[0066] The pixel array 210 includes CFA cells 211 having various patterns according to the above-described embodiments, and provides pixel values of color pixels arranged in the pixel array 210 to the video processor 220. Further, the video processor 220 includes various types of logic devices that process pixel values to generate image data. As an example, the video processor 220 includes a binning processor 221 and a remosaic processor 222. When the high-quality mode is selected under the control of the binning processor 221, video processing is performed using pixel values having a relatively high resolution, or when the low-quality mode is selected, video processing is performed using pixel values having a relatively low resolution. Also, the remosaic processor 222 performs interpolation processing at each pixel in relation to remosaic processing, and according to the above-described embodiments, pixel values corresponding to the Bayer pattern are generated using the pixel values of the CFA cells.

[0067] According to an embodiment of the present invention, the pixel array 210 includes the CFA cells (or CFA blocks) described in the above-described embodiments, and the binning processing is performed based on the arrangement of the color pixels described in the embodiments of the present invention. As a result of performing the binning processing, the size of the image data processed by the video processor 220 is reduced, and the data size per frame is reduced, so that a high frame rate is maintained in the video mode.

[0068] According to an embodiment of the present invention, the binning process is performed based on various methods. For example, the video processor 220 performs the binning process after digitally summing the pixel values of the color pixels of the pixel array 210. Alternatively, the binning process may be performed based on a method in which the electrical signals of at least two color pixels of the pixel array 210 are analogously summed.

[0069] Also, according to an embodiment of the present invention, in the binning process, the color pixels whose signals are summed with each other are variously selected. As an example, by the intra-block method, color pixels that sense the same color are selected from any one CFA block, and the signals are summed analogously or digitally. For example, all the color pixels that sense the same color included in any one CFA block are selected, or some of the color pixels are selected.

[0070] Alternatively, by the inter-block method, color pixels that sense the same color in at least two CFA blocks are selected, and the signals are summed analogously or digitally. As an example, the signal of a color pixel of any one CFA block and the signals of pixels that sense the same color in one or more rows or columns of an adjacent CFA block are summed with each other.

[0071] Hereinafter, an example in which the binning process is performed through analog signal summation according to an embodiment of the present invention will be described. FIGS. 11A to 11C are diagrams showing an embodiment example of a pixel array according to the present invention.

[0072] Referring to FIG. 11A, the pixel array includes a plurality of color pixels, and an example is illustrated in which n color pixels CP1 to CPn share one floating diffusion region (FD). As an example, in one CFA block, at least a part of the color pixels sensing the same color are shared by one floating diffusion region FD. According to one embodiment, in the CFA block, all the color pixels sensing the same color are shared by one floating diffusion region FD, or some of the color pixels are shared by one floating diffusion region FD.

[0073] Each color pixel includes a photodiode PD and a transfer gate TG. For example, when the transfer gates TG of the n color pixels CP1 to CPn are turned on together, the photocharges of the n color pixels CP1 to CPn are provided to the floating diffusion region FD together. When the transfer gates TG of the n color pixels CP1 to CPn are controlled separately, the photocharges of the n color pixels CP1 to CPn are provided to the floating diffusion region FD at different timings. As an example, when the binning mode is not executed, the photocharges of the n color pixels CP1 to CPn are provided to the floating diffusion region FD at different timings. On the other hand, when the binning mode is executed, the photocharges of the n color pixels CP1 to CPn are provided to the floating diffusion region FD together, so that the signals of the n color pixels CP1 to CPn are combined in the floating diffusion region FD.

[0074] Figures 11B and 11C show operation examples in the binning mode according to an embodiment of the present invention. Figures 11B and 11C illustrate a plurality of color pixels provided in one CFA cell. As an example, an example in which binning processing is performed on the CFA cell shown in Figure 4 is illustrated. Further, Figures 11B and 11C illustrate a case where signals of pixels that sense the same color are added together in any one CFA block by applying the intra-block method during binning processing. In Figure 11B, a binning example in two CFA blocks located above one CFA cell is illustrated, and in Figure 11C, a binning example in two CFA blocks located below one CFA cell is illustrated.

[0075] Referring to Figures 11B and 11C, each CFA block is classified into a plurality of groups based on the sharing method of the floating diffusion region FD. For example, referring to the green CFA block located in the upper left of the CFA cell, the green CFA block is classified into groups G00 to G11 located in the upper left - upper right - lower left - lower right, each having 2×2 color pixels, and the four color pixels of each of groups G00 to G11 share one floating diffusion region FD. By performing binning processing, the photocharges of the color pixels that sense the same color in each group are provided to the floating diffusion region FD. As an example, in the green CFA block, the photocharges of the four green pixels in group G00 are provided to the floating diffusion region FD, the photocharges of the three green pixels in group G01 are provided to the floating diffusion region FD, the photocharges of the three green pixels in group G10 are provided to the floating diffusion region FD, and the photocharges of the four green pixels in group G11 are provided to the floating diffusion region FD. Similarly, in each group of the red CFA block, the photocharges of three red pixels are provided to the floating diffusion region FD.

[0076] The signals provided from each of groups G00 to G11 (e.g., the combined signal) are provided to the readout circuit RC, and the pixel value corresponding to the combined signal is output from the readout circuit RC. Also, the pixel values of groups G00 to G11 are combined through analog or digital arithmetic processing, and through the above-described combining process, the green pixel value corresponding to the green CFA block is calculated.

[0077] On the other hand, when the binning process is not performed, the 4×4 color pixels provided in the green CFA block are individually connected to the floating diffusion region FD to transfer the photoelectric charge.

[0078] On the other hand, referring to FIG. 11C, the blue CFA block located at the lower left of the CFA cell and the green CFA block located at the lower right are illustrated. When the binning process is performed, in each group of the blue CFA blocks, the photoelectric charges from three blue pixels are provided to the floating diffusion region FD and combined.

[0079] For example, the blue CFA block at the lower left end of the CFA cell is classified into groups B00, B01, B10, and B11. This group is located at the upper left end, upper right end, lower left end, and lower right end of the blue CFA block, and each group of blocks includes 2×2 color pixels. The four color pixels of each of groups B00 to B11 share one floating diffusion region FD with each other. By performing the binning process, in each group, the photoelectric charges of the color pixels that sense the same color are provided to the floating diffusion region FD. For example, in the blue CFA block, the photoelectric charges of three blue pixels in group B00 are provided to the floating diffusion region FD, and the photoelectric charges of three blue pixels in group B01 are provided to the floating diffusion region FD. Also, the photoelectric charges of three blue pixels in group B10 are provided to the floating diffusion region FD, and the photoelectric charges of three blue pixels in group B11 are provided to the floating diffusion region FD.

[0080] In the green CFA block, for some groups, four green pixels are arranged, so that the photocharges from the four green pixels are provided to the floating diffusion region FD and summed up. For some other groups, three green pixels are arranged, so that the photocharges from the three green pixels are provided to the floating diffusion region FD and summed up.

[0081] FIG. 12 is a diagram showing an example of binning processing of the inter-block and intra-block methods according to an embodiment of the present invention. In FIG. 12, examples of binning processing for each of the red CFA block, the green CFA block, and the blue CFA block are illustrated.

[0082] In the red CFA block, three color pixels within the sub-block and one color filter in the outer region correspond to pixels sensing other colors, and binning processing is performed based on the intra method, so that the 12 red pixels of the red CFA block are summed up. Similarly, in the green CFA block, 14 green pixels are summed up, and in the blue CFA block, 12 blue pixels are summed up.

[0083] On the other hand, when the inter method is applied, when calculating the pixel value corresponding to any one CFA block, the color pixels of one or more adjacent CFA blocks are further used. As an example, in the example shown in FIG. 12, examples are illustrated in which color pixels located in one row of the adjacent CFA block are further used, or color pixels located in two rows and columns of the adjacent CFA block are further used.

[0084] For example, for the red CFA block, the signals of the 12 red pixels provided in the red CFA block and the red pixels of one row of other CFA blocks adjacent to the red CFA block are added together. Or, the signals of the 12 red pixels provided in the red CFA block and the red pixels of two rows and columns of other CFA blocks adjacent to the red CFA block are added together. Similarly, for the blue CFA block, the signals of the 12 blue pixels provided in the blue CFA block and the blue pixels located in one or two rows and columns of other adjacent CFA blocks are added together.

[0085] According to various embodiments, in performing the binning process of the green CFA block based on the inter-block method, the green CFA block is provided with relatively more green pixels than other CFA blocks. Thus, in the example of FIG. 12, an example is illustrated in which the signal of the green pixels of one row of other CFA blocks adjacent to the green CFA block is used for the binning process. However, the embodiments of the present invention are not limited thereto, and for the green CFA block, the green pixels of two or more rows and columns of other CFA blocks adjacent thereto may also be further used for signal addition.

[0086] FIGS. 13 to 15 are diagrams showing exemplary embodiments of CFA blocks according to various embodiments. FIG. 13 illustrates a red CFA block, FIG. 14 illustrates a blue CFA block, and FIG. 15 illustrates a green CFA block. Also, FIGS. 13 to 15 illustrate examples in which the interpolation distance is further reduced through the arrangement of color pixels in the outer contour region of each CFA block. On the other hand, FIGS. 13 to 15 illustrate the CFA block shown in the embodiment of FIG. 4 described above, but this embodiment is also applicable to other various types of CFA blocks described above.

[0087] Referring to FIG. 13, the red CFA block includes sub - blocks each having 2×2 color pixels as in the above - described embodiment. Although mainly red pixels are arranged in the outer - perimeter region, blue pixels are arranged at a corner on one side (e.g., the lower - left corner) of the outer - perimeter region. As a result, in a specific direction (e.g., from the upper - right to the lower - left) of the red CFA block, red - blue - red - blue pixels are arranged.

[0088] Also, pixels for sensing at least one other color are arranged at another corner of the outer - perimeter region of the red CFA block. For example, in FIG. 13(a), a green pixel arranged at the lower - right corner is illustrated, and in FIG. 13(b), two green pixels arranged at the lower - right and upper - left corners are illustrated. The green pixels in the outer - perimeter region shown in FIGS. 13(a) and (b) sense the same color (green color) as the green pixels at positions corresponding to the Bayer pattern. As a result, there is an effect that the value of the interpolation distance of the pixels in the red CFA block is further reduced.

[0089] On the other hand, referring to FIG. 14, although mainly blue pixels are arranged in the outer - perimeter region of the blue CFA block, red pixels are arranged at a corner on one side (e.g., the upper - right corner) of the outer - perimeter region. As a result, in a specific direction (e.g., from the upper - right to the lower - left) of the blue CFA block, red - blue - red - blue pixels are arranged. Also, similar to the embodiment of FIG. 13, pixels for sensing at least one other color are arranged at another corner of the outer - perimeter region of the blue CFA block. In FIG. 14(a), a green pixel arranged at the lower - right corner is illustrated, and in FIG. 14(b), two green pixels arranged at the lower - right and upper - left corners are illustrated. The embodiments shown in FIGS. 14(a) and (b) have an effect of further reducing the value of the interpolation distance of the pixels in the blue CFA block.

[0090] On the one hand, referring to FIG. 15, in the outer contour region of the green CFA block, pixels for sensing one or more other colors (for example, red pixels and / or blue pixels) are arranged. In FIG. 15(a), an example in which blue pixels are arranged at a corner on one side (for example, the lower left corner) of the outer contour region of the green CFA block is illustrated. In FIG. 15(b), a blue pixel arranged at the lower left corner of the outer contour region of the green CFA block and a red pixel arranged at the upper right corner are exemplified. The color pixels (for example, blue pixels and / or red pixels) in the outer contour region shown in FIGS. 15(a) and (b) and the color pixels at positions corresponding to the Bayer pattern correspond to pixels that sense the same color as each other, thereby having the effect of further reducing the value of the interpolation distance of the pixels in the green CFA block.

[0091] FIG. 16 is a diagram showing an embodiment example of color pixels according to the present invention.

[0092] According to the above-described embodiment, each color pixel includes a corresponding color filter and a light sensing element (for example, a photodiode) for sensing light (or color).

[0093] According to one embodiment example, one color pixel includes a plurality of sub-pixels. For example, FIG. 16 illustrates an example in which one color pixel includes 2×2 sub-pixels, but it may include even more sub-pixels. Also, the light sensing elements are arranged in units of sub-pixels, so that a plurality of light sensing elements are arranged corresponding to one color pixel. Although not shown in FIG. 16, the image sensor includes a plurality of microlenses (not shown), and the microlenses may be arranged in units of pixels or in units of sub-pixels.

[0094] When calculating pixel signals corresponding to respective color pixels, at least a part of signals generated from a plurality of sub-pixels is used. For example, assuming that the color pixel located at the upper left of FIG. 16 is a red pixel and a plurality of sub-pixels similarly sense the red color, through processing using at least a part of signals generated from the plurality of sub-pixels, a pixel signal provided from the red pixel is generated.

[0095] On the other hand, embodiments of the present invention can be variously implemented. As an example, signals sensing two or more colors at any one color pixel may be provided. As an example, color filters for sensing mutually different colors may be arranged on a plurality of sub-pixels provided in any one color pixel, and respective color pixels are variously implemented within a range of reducing the interpolation distance according to the above-described embodiments.

[0096] FIG. 17 is a block diagram of an electronic device including a multi-camera module to which an image sensor according to an embodiment of the present invention is applied. FIG. 18 is a detailed block diagram of the camera module of FIG. 17.

[0097] Referring to FIG. 17, the electronic device 1000 includes a camera module group 1100, an application processor 1200, a PMIC 1300, and an external memory 1400.

[0098] The camera module group 1100 includes a plurality of camera modules 1100a, 1100b, 1100c. Although FIG. 17 illustrates an embodiment in which three camera modules 1100a, 1100b, 1100c are arranged, the embodiment is not limited thereto. In some embodiments, the camera module group 1100 may be implemented by being modified to include only two camera modules. Also, in some embodiments, the camera module group 1100 may be implemented by being modified to include n (n is a natural number of 4 or more) camera modules.

[0099] Hereinafter, with reference to FIG. 18, the detailed configuration of the camera module 1100b will be described more specifically. The following description is also equally applicable to the other camera modules 1100a and 1100c.

[0100] Referring to FIG. 18, the camera module 1100b includes a prism 1105, an Optical Path Folding Element (hereinafter, “OPFE”) 1110, an actuator 1130, an image sensing device 1140, and a storage unit 1150.

[0101] The prism 1105 includes a reflection surface 1107 of a light-reflecting material and changes the path of the light L incident from the outside.

[0102] In some embodiments, the prism 1105 changes the path of the light L incident in the first direction (X) to the second direction (Y) perpendicular to the first direction (X). Also, the prism 1105 rotates the reflection surface 1107 of the light-reflecting material about the central axis 1106 in the A direction or rotates the central axis 1106 in the B direction to change the path of the light L incident in the first direction (X) to the second direction (Y) perpendicular to the first direction (X). At this time, the OPFE 1110 also moves in the third direction (Z) perpendicular to the first direction (X) and the second direction (Y).

[0103] In some embodiments, as shown in FIG. 18, the maximum rotation angle of the prism 1105 in the A direction is 15 degrees or less in the plus (+) A direction and greater than 15 degrees in the minus (−) A direction, but the embodiments are not limited thereto.

[0104] In some embodiments, the prism 1105 moves within a range of about 20 degrees, or 10 degrees to 20 degrees, or 15 degrees to 20 degrees in the plus (+) or minus (−) B direction, where the moving angle moves at the same angle in the plus (+) or minus (−) B direction or moves to an approximately similar angle within a range of about 1 degree.

[0105] In some embodiments, the prism 1105 moves the reflective surface 1107 of the light-reflective material in a third direction (e.g., the Z direction) parallel to the extension direction of the central axis 1106.

[0106] The OPFE 1110 includes, for example, an optical lens composed of m (where m is a natural number) groups. The m lenses move in the second direction (Y) to change the optical zoom ratio of the camera module 1100b. For example, if the basic optical zoom ratio of the camera module 1100b is Z, when the m optical lenses provided in the OPFE 1110 are moved, the optical zoom ratio of the camera module 1100b is changed to an optical zoom ratio of 3Z, 5Z, or 5Z or more.

[0107] The actuator 1130 moves the OPFE 1110 or the optical lens (hereinafter referred to as the optical lens) to a specific position. For example, the actuator 1130 adjusts the position of the optical lens so that the image sensor 1142 is located at the focal length of the optical lens for accurate sensing.

[0108] The image sensing device 1140 includes an image sensor 1142, control logic 1144, and a memory 1146. The image sensor 1142 senses an image of a sensing target using the light L provided through the optical lens. According to the embodiments of the present invention described above, the image sensor 1142 includes a pixel array, and the color pattern for the plurality of color pixels provided in the pixel array follows the patterns in the CFA cells, CFA blocks, and sub-blocks in the above-described embodiments.

[0109] The control logic 1144 controls the overall operation of the camera module 1100b. For example, the control logic 1144 controls the operation of the camera module 1100b by a control signal provided through the control signal line CSLb.

[0110] The memory 1146 stores information necessary for the operation of the camera module 1100b, such as calibration data 1147. The calibration data 1147 includes information necessary for generating image data using the light L provided from the outside by the camera module 1100b. The calibration data 1147 includes, for example, information regarding the rotation degree, information regarding the focal length, information regarding the optical axis, etc., described above. When the camera module 1100b is embodied in a multi-state camera form in which the focal length changes depending on the position of the optical lens, the calibration data 1147 includes the focal length values for each position (or state) of the optical lens and information regarding autofocusing.

[0111] The storage unit 1150 stores the image data sensed through the image sensor 1142. The storage unit 1150 is disposed outside the image sensing device 1140 and is embodied in a form laminated on the sensor chip constituting the image sensing device 1140. In some embodiments, the storage unit 1150 is embodied by an EEPROM (Electrically Erasable Programmable Read-Only memory), but the embodiments are not limited thereto.

[0112] In some embodiments, each of the plurality of camera modules 1100a, 1100b, 1100c includes an actuator 1130. Accordingly, each of the plurality of camera modules 1100a, 1100b, 1100c includes calibration data 1147 that is equal or different by the operation of the actuator 1130 provided therein.

[0113] In some embodiments, one of the plurality of camera modules 1100a, 1100b, 1100c (for example, 1100b) is a camera module in a folded lens form including the prism 1105 and the OPFE 1110 described above, and the remaining camera modules (for example, 1100a and 1100c) are camera modules in a vertical form that do not include the prism 1105 and the OPFE 1110, but the embodiments are not limited thereto.

[0114] In some embodiments, one of the plurality of camera modules 1100a, 1100b, 1100c (for example, 1100c) is a depth camera in a vertical form that extracts depth information using, for example, IR (Infrared Ray). In this case, the application processor 1200 merges the image data provided from such a depth camera and the image data provided from other camera modules (for example, 1100a or 1100b) to generate a three-dimensional depth image.

[0115] In some embodiments, at least two of the plurality of camera modules 1100a, 1100b, 1100c (for example, 1100a and 1100b) have different observation fields of view (Field of View, viewing angle). In this case, for example, the optical lenses of at least two of the plurality of camera modules 1100a, 1100b, 1100c (for example, 1100a and 1100b) are different from each other, but it is not limited thereto.

[0116] Also, in some embodiments, the viewing angles of the plurality of camera modules 1100a, 1100b, 1100c are different from each other. In this case, the optical lenses provided in each of the plurality of camera modules 1100a, 1100b, 1100c are also different from each other, but it is not limited thereto.

[0117] In some embodiments, each of the plurality of camera modules 1100a, 1100b, 1100c is physically separated from each other and arranged. That is, the sensing area of one image sensor 1142 is not divided and used by the plurality of camera modules 1100a, 1100b, 1100c, but an independent image sensor 1142 is arranged inside each of the plurality of camera modules 1100a, 1100b, 1100c.

[0118] Referring to FIG. 17 again, the application processor 1200 includes an image processing device 1210, a memory controller 1220, and an internal memory 1230. The application processor 1200 is implemented separately from the plurality of camera modules 1100a, 1100b, 1100c. For example, the application processor 1200 and the plurality of camera modules 1100a, 1100b, 1100c are implemented separately from each other by separate semiconductor chips.

[0119] The image processing device 1210 includes a plurality of sub-image processors 1212a, 1212b, 1212c, an image generator 1214, and a camera module controller 1216.

[0120] The image processing device 1210 includes a plurality of sub-image processors 1212a, 1212b, 1212c corresponding to the number of the plurality of camera modules 1100a, 1100b, 1100c.

[0121] The image data generated from each of the camera modules 1100a, 1100b, and 1100c is provided to the corresponding sub-image processors 1212a, 1212b, and 1212c through the mutually separated image signal lines ISLa, ISLb, and ISLc. For example, the image data generated from the camera module 1100a is provided to the sub-image processor 1212a through the image signal line ISLa, the image data generated from the camera module 1100b is provided to the sub-image processor 1212b through the image signal line ISLb, and the image data generated from the camera module 1100c is provided to the sub-image processor 1212c through the image signal line ISLc. Such transmission of image data is performed, for example, using a camera serial interface (CSI) based on MIPI (Mobile Industry Processor Interface), but the embodiments are not limited thereto.

[0122] On the other hand, in some embodiments, one sub-image processor may be arranged to correspond to a plurality of camera modules. For example, the sub-image processor 1212a and the sub-image processor 1212c are not embodied separately from each other as shown in the figure, but are integrated into one sub-image processor, and the image data provided from the camera module 1100a and the camera module 1100c is provided to the integrated sub-image processor after being selected through a selection element (e.g., a multiplexer).

[0123] The image data provided to each of the sub-image processors 1212a, 1212b, and 1212c is provided to the image generator 1214. The image generator 1214 generates an output image using the image data provided from each of the sub-image processors 1212a, 1212b, and 1212c according to image generation information or a mode signal.

[0124] Specifically, the image generator 1214 merges at least a part of the image data generated from the camera modules 1100a, 1100b, and 1100c having different viewing angles from each other according to the image generation information or the mode signal to generate an output image. Further, the image generator 1214 selects any one of the image data generated from the camera modules 1100a, 1100b, and 1100c having different viewing angles from each other according to the image generation information or the mode signal to generate an output image.

[0125] In some embodiments, the image generation information includes a zoom signal (or a zoom factor). Also, in some embodiments, the mode signal is a signal based on, for example, a mode selected by the user.

[0126] When the image generation information is a zoom signal (zoom factor) and each of the camera modules 1100a, 1100b, and 1100c has a different observation field of view (viewing angle) from each other, the image generator 1214 performs different operations according to the type of the zoom signal. For example, when the zoom signal is a first signal, after merging the image data output from the camera module 1100a and the image data output from the camera module 1100c, an output image is generated using the merged image signal and the image data output from the camera module 1100b that was not used for the merging. If the zoom signal is a second signal different from the first signal, the image generator 1214 does not perform such image data merging and selects any one of the image data output from the camera modules 1100a, 1100b, and 1100c to generate an output image. However, the embodiments are not limited to this, and the method of processing the image data can be deformed in any way as needed.

[0127] In some embodiments, the image generator 1214 receives a plurality of image data with different exposure times from at least one of the plurality of sub-image processors 1212a, 1212b, 1212c, and performs HDR (high dynamic range) processing on the plurality of image data to generate merged image data with an increased dynamic range.

[0128] The camera module controller 1216 provides control signals to the respective camera modules 1100a, 1100b, 1100c. The control signals generated from the camera module controller 1216 are provided to the corresponding camera modules 1100a, 1100b, 1100c through the control signal lines CSLa, CSLb, CSLc that are separated from each other.

[0129] Any one of the plurality of camera modules 1100a, 1100b, 1100c is designated as a master camera (e.g., 1100b) by image generation information or a mode signal including a zoom signal, and the remaining camera modules (e.g., 1100a and 1100c) are designated as slave cameras. Such information is included in the control signal and is provided to the corresponding camera modules 1100a, 1100b, 1100c through the control signal lines CSLa, CSLb, CSLc that are separated from each other.

[0130] The camera modules operating as master and slave change depending on the zoom factor or the operation mode signal. For example, when the field of view angle of the camera module 1100a is wider than that of the camera module 1100b and the zoom factor indicates a low zoom magnification, the camera module 1100b operates as the master and the camera module 1100a operates as the slave. Conversely, when the zoom factor indicates a high zoom magnification, the camera module 1100a operates as the master and the camera module 1100b operates as the slave.

[0131] In some embodiments, the control signals provided from the camera module controller 1216 to the respective camera modules 1100a, 1100b, 1100c include a sync enable signal. For example, when the camera module 1100b is the master camera and the camera modules 1100a and 1100c are slave cameras, the camera module controller 1216 transmits a sync enable signal to the camera module 1100b. The camera module 1100b provided with such a sync enable signal generates a sync signal based on the provided sync enable signal, and provides the generated sync signal to the camera modules 1100a and 1100c through the sync signal line SSL. The camera modules 1100b, 1100a, and 1100c are synchronized with such a sync signal and transmit image data to the application processor 1200.

[0132] In some embodiments, the control signals provided from the camera module controller 1216 to the plurality of camera modules 1100a, 1100b, 1100c include mode information by a mode signal. Based on such mode information, the plurality of camera modules 1100a, 1100b, 1100c operate in a first operation mode and a second operation mode with respect to the sensing speed.

[0133] The plurality of camera modules 1100a, 1100b, 1100c generate an image signal at a first speed (for example, generate an image signal at a first frame rate) in the first operation mode, encode it at a second speed higher than the first speed (for example, encode an image signal at a second frame rate higher than the first frame rate), and transmit the encoded image signal to the application processor 1200. At this time, the second speed is 30 times or less of the first speed.

[0134] The application processor 1200 stores the received image signal, that is, the encoded image signal, in the internal memory 1230 provided therein or the external memory 1400 of the application processor 1200, and then reads out and decodes the encoded image signal from the internal memory 1230 or the external memory 1400, and displays the image data generated based on the decoded image signal. For example, among the plurality of sub-processors 1212a, 1212b, 1212c of the image processing device 1210, the corresponding sub-processor performs decoding and also performs image processing on the decoded image signal.

[0135] The plurality of camera modules 1100a, 1100b, 1100c generate an image signal at a third speed lower than the first speed (for example, generate an image signal at a third frame rate lower than the first frame rate) in the second operation mode, and transmit the image signal to the application processor 1200. The image signal provided to the application processor 1200 is an unencoded signal. The application processor 1200 performs image processing on the received image signal or stores the image signal in the internal memory 1230 or the external memory 1400.

[0136] The PMIC 1300 supplies power, for example, a power supply voltage, to each of the plurality of camera modules 1100a, 1100b, 1100c. For example, under the control of the application processor 1200, the PMIC 1300 supplies first power to the camera module 1100a through the power signal line PSLa, supplies second power to the camera module 1100b through the power signal line PSLb, and supplies third power to the camera module 1100c through the power signal line PSLc.

[0137] PMIC1300 generates power corresponding to each of the plurality of camera modules 1100a, 1100b, 1100c in response to the power control signal PCON from the application processor 1200, and also adjusts the power level. The power control signal PCON includes power adjustment signals for different operation modes of the plurality of camera modules 1100a, 1100b, 1100c. For example, the operation mode includes a low power mode, and at this time, the power control signal PCON includes information regarding the camera module operating in the low power mode and the set power level. The power levels provided to each of the plurality of camera modules 1100a, 1100b, 1100c are equal or different. Also, the power level changes dynamically.

[0138] On the other hand, various components in the above-described embodiments are implemented by a processing circuit such as hardware including a logic circuit, a processor that executes software, or a combination of hardware and software. In some exemplary embodiments, the processing circuit includes a central processing unit (CPU), an application processor (AP), an arithmetic logic unit (ALU), a graphics processing unit (GPU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a system-on-chip (SoC), a programmable logic device, a microprocessor, or an application-specific integrated circuit (ASIC).

[0139] As described above, exemplary embodiments have been disclosed in the drawings and the specification. Although specific terms have been used in this specification to describe the embodiments, these are merely used for the purpose of explaining the technical idea of the present invention and are not used for meaning limitation or for limiting the technical scope of the present invention. Therefore, it is understood by those skilled in the art that various modifications and equivalent other embodiments will be possible in the future.

Description of Reference Numerals

[0140] 100 Image Sensor 110, 210 Pixel Array 111, 211 CFA cells 120 Loudspeaker 130 Readout circuit 140 Controller 200 Image processing device 220 Image processor 221 Binning processor 222 Remosaic processor 1000 Electronic device 1100 Camera module group 1100a, 1100b, 1100c Camera modules 1105 Prism 1106 Central axis 1107 Reflective surface 1110 OPFE 1130 Actuator 1140 Image sensing device 1142 Image sensor 1144 Control logic 1146 Memory 1147 Calibration data 1150 Storage unit 1200 Application processor 1210 Image processing device 1212a, 1212b, 1212c Sub-image processors 1214 Image generator 1216 Camera module controller 1220 Memory controller 1230 Internal memory 1300 PMIC 1400 External memory

Claims

1. In the pixel array of an image sensor, the pixel array includes a plurality of color filter array (CFA) cells each having a predetermined size, and each CFA cell includes a plurality of CFA blocks arranged in horizontal and vertical directions, each of the plurality of CFA blocks includes a sub-block located in a center region and having m*n color pixels, and an outer region in which color pixels other than the sub-block are arranged (where m and n are integers of 2 or more respectively), the m*n color pixels of the sub-block include color pixels sensing first to third colors, and the color pixels sensing the first to third colors are arranged at positions corresponding to a Bayer pattern, the outer region includes a first pixel sensing the first color and a second pixel sensing at least one of the second color or the third color, the number of the first pixels is larger than the number of the second pixels, the first to third colors include a red color, a green color, and a blue color, the CFA cell includes 2*2 CFA blocks arranged in horizontal and vertical directions, the 2*2 CFA blocks include two green CFA blocks, one red CFA block, and one blue CFA block corresponding to the color of the first pixels included in the outer region of each CFA block, The pixel array of the image sensor, wherein the outer region of the red CFA block includes a blue pixel arranged at at least one corner of the red CFA block as the second pixel and includes a plurality of red pixels as the first pixels.

2. The pixel array of the image sensor according to claim 1, wherein each of the 2*2 CFA blocks includes 4*4 color pixels, and the sub-block of each CFA block includes 2*2 color pixels.

3. The pixel array of the image sensor according to claim 1, wherein in the red CFA block, red pixels, blue pixels, red pixels, and blue pixels are alternately arranged in a diagonal direction from any one corner.

4. The outer contour region of the blue CFA block includes a red pixel disposed at at least one corner of the blue CFA block as the second pixel, and a plurality of blue pixels as the first pixels, and the pixel array of the image sensor according to claim 1 is characterized in that.

5. The outer contour region of the green CFA block includes green pixels as the first pixels, and the first pixels are arranged in the second pixels, and the pixel array of the image sensor according to claim 1 is characterized in that.

6. In the CFA cell, four color pixels each including one color pixel in the outer contour region of each of the 2×2 CFA blocks constitute a first region. The first region includes the second pixel of the red CFA block, the second pixel of the blue CFA block, and one green pixel in the outer contour region of each of the two green CFA blocks. When the CFA cell is converted into a Bayer pattern by remosaicking processing, each interpolation distance of the color pixels in the first region has a value of 0, and the pixel array of the image sensor according to claim 1 is characterized in that.

7. Binning processing is performed on the color pixels of the pixel array. In the process of the binning processing, signals of color pixels that sense the same color within any one CFA block are selectively added together, and the pixel array of the image sensor according to claim 1 is characterized in that.

8. Binning processing is performed on the color pixels of the pixel array. In the process of the binning processing, signals of color pixels that sense the same color located in at least one line between any one CFA block and an adjacent CFA block are added together, and the pixel array of the image sensor according to claim 1 is characterized in that.

9. In the pixel array of the image sensor. The pixel array includes a plurality of color filter arrays (CFA) cells each having a predetermined size. Each CFA cell includes 2×2 CFA blocks arranged in the horizontal and vertical directions. Each CFA block includes 4×4 color pixels. Each of the CFA blocks is located in a center region and includes a sub-block in which 2×2 color pixels are arranged and an outer region located on the outer periphery in which 12 color pixels are arranged. The 2×2 color pixels arranged in the sub-block are arranged at positions corresponding to a Bayer pattern. The 2×2 CFA blocks include a red CFA block, a green CFA block, and a blue CFA block. Each of the sub-blocks of the CFA blocks includes one red pixel, one blue pixel, and two green pixels. A pixel array of an image sensor, wherein the 12 color pixels in the outer region of the red CFA block are composed of one blue pixel and the remaining 11 red pixels.

10. The pixel array of the image sensor according to claim 9, wherein the blue pixel arranged in the outer region of the red CFA block is arranged at one of the corners of the red CFA block.

11. The pixel array of the image sensor according to claim 10, wherein blue pixels, red pixels, blue pixels, and red pixels are alternately arranged in a diagonal direction from the corner where the blue pixel is arranged in the red CFA block.

12. The pixel array of the image sensor according to claim 9, wherein the 12 color pixels in the outer region of the blue CFA block are composed of one red pixel and the remaining 11 blue pixels.

13. The CFA cell includes one red CFA block, two green CFA blocks, and one blue CFA block. One blue pixel in the outer region of the red CFA block, one red pixel in the outer region of the blue CFA block, and green pixels in the outer regions of the two green CFA blocks constitute a first region. The pixel array of the image sensor according to claim 12, wherein when the CFA cell is converted into a Bayer pattern by remosaicking processing, each interpolation distance of the color pixels in the first region has a value of 0.

14. The 12 color pixels in the outer contour region of the green CFA block are composed of two or fewer color pixels that sense at least one of the red color and the blue color, and a plurality of remaining green pixels. The pixel array of the image sensor according to claim 9, characterized in that.

15. Comprising a plurality of color filter array (CFA) cells, each CFA cell comprising a plurality of CFA blocks arranged in the horizontal and vertical directions, each of the plurality of CFA blocks being located in a center region and having n*n color pixels A pixel array comprising a sub-block and an outer contour region in which color pixels other than the sub-block are arranged (where n is an integer of 2 or more), A reading circuit for reading pixel values from the color pixels of the pixel array, The n*n color pixels of the sub-block include color pixels that sense the first to third colors, and the color pixels that sense the first to third colors are arranged at positions corresponding to the Bayer pattern, The outer contour region includes a first pixel that senses any one of the first to third colors and a second pixel that senses another one of the first to third colors, The number of the first pixels is larger than the number of the second pixels, The first to third colors include a red color, a green color, and a blue color, The CFA cell includes 2*2 CFA blocks arranged in the horizontal and vertical directions, The 2*2 CFA blocks include two green CFA blocks, one red CFA block, and one blue CFA block corresponding to the color of the first pixel included in the outer contour region of each CFA block, The outer contour region of the red CFA block includes one blue pixel arranged at a corner of the red CFA block and 11 remaining red pixels, The outer contour region of the blue CFA block includes one red pixel arranged at a corner of the blue CFA block and 11 remaining blue pixels. An image sensor characterized by that.

16. The image sensor according to claim 15, wherein each of the 2×2 CFA blocks includes 4×4 color pixels, and each sub-block of each CFA block includes 2×2 color pixels.

17. The image sensor further includes a binning processor that performs binning processing based on pixel values from the pixel array, The image sensor according to claim 15, wherein in the process of the binning processing, signals of color pixels that sense the same color within any one CFA block are selectively added together.

18. The image sensor further includes a binning processor that performs binning processing based on pixel values from the pixel array, The image sensor according to claim 15, wherein in the process of the binning processing, it senses the same color located on at least one line between any one CFA block and an adjacent CFA block.

Images