Pixel array for reducing image information loss and image sensor including the same

Abstract

A pixel array for reducing image information loss and an image sensor including the pixel array are provided. The pixel array includes: a plurality of color filter array (CFA) units having a specific size, each CFA unit including a plurality of CFA blocks in the width direction and the length direction of the CFA unit, wherein each CFA block includes a sub-block and a far-center region different from the sub-block, the sub-block is at the center region of each CFA block and includes m×n color pixels, the far-center region includes color pixels, wherein the m×n color pixels of the sub-block include color pixels sensing first to third colors, and the far-center region includes a relatively large number of first pixels sensing the first color and a relatively small number of second pixels sensing at least one color selected from the second color and the third color.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2020-0072618 filed in the Korean Intellectual Property Office on June 15, 2020, and Korean Patent Application No. 10-2021-0042232 filed in the Korean Intellectual Property Office on March 31, 2021, the disclosures of which are incorporated herein in their entirety by reference. Technical Field

[0003] The inventive concept relates to an image sensor, and more particularly, to a pixel array for reducing image information loss and an image sensor including the pixel array. Background Art

[0004] An image sensor captures a two-dimensional (2D) or three-dimensional (3D) image of an object. An image sensor generates an image of an object using a photoelectric conversion element that reacts to the intensity of light reflected from the object. With the recent development of complementary metal oxide semiconductor (CMOS) technology, CMOS image sensors (CIS) using CMOS have been widely used.

[0005] The image sensor may include a pixel array. When the image sensor has a higher pixel count, the pixel array may include more color pixels. In order to convert the raw image output from the image sensor into a specific pattern such as an RGB image, a remosaic process based on interpolation and / or extrapolation may be performed. When the distance to the adjacent color pixel (which may be referenced in the process) increases, image information loss may occur. Summary of the invention

[0006] The inventive concept provides a pixel array for preventing or reducing an increase in image information loss during processing of an image captured by an image sensor.

[0007] According to one aspect of the inventive concept, a pixel array of an image sensor is provided. The pixel array includes: a plurality of color filter array (CFA) units having a specific size, each of the plurality of CFA units including a plurality of CFA blocks in a width direction of the CFA unit and a length direction of the CFA unit, wherein each of the plurality of CFA blocks includes a sub-block and a far-center region different from the sub-block, the sub-block is at a central region of each of the plurality of CFA blocks and includes m×n color pixels, and the far-center region includes other color pixels, the m×n color pixels of the sub-block include color pixels for sensing a first color, a second color, and a third color, and the far-center region includes a first number of first pixels for sensing the first color and a second number of second pixels for sensing at least one color selected from the second color and the third color, and the first number of the first pixels is greater than the second number of the second pixels.

[0008] According to one aspect of the inventive concept, a pixel array of an image sensor is provided. The pixel array includes a plurality of color filter array (CFA) units having a specific size, each of the plurality of CFA units includes 2×2 CFA blocks in the width direction of the CFA unit and in the length direction of the CFA unit, and each of the 2×2 CFA blocks includes 4×4 color pixels. Each of the 2×2 CFA blocks includes a sub-block and an off-center area, the sub-block is at the center area of ​​each of the 2×2 CFA blocks and includes 2×2 color pixels, and the off-center area is an area of ​​each of the 2×2 CFA blocks other than the sub-block and includes 12 color pixels. The 2×2 CFA blocks include a red CFA block, a green CFA block, and a blue CFA block, and the sub-block includes 1 red pixel, 1 blue pixel, and 2 green pixels, wherein the 12 color pixels in the off-center area of ​​the red CFA block include 1 blue pixel and 11 red pixels.

[0009] According to one aspect of the inventive concept, an image sensor is provided. The image sensor includes: a pixel array including a plurality of color filter array (CFA) units, each CFA unit including a plurality of CFA blocks in the width direction of the CFA unit and in the length direction of the CFA unit, each of the plurality of CFA blocks including a sub-block and a far-center region different from the sub-block, the sub-block being at the center region of each of the plurality of CFA blocks and including n×n color pixels, and the far-center region including other color pixels, where "n" is an integer of at least 2. The pixel array also includes a reading circuit configured to read pixel values ​​from the color pixels of the pixel array. The n×n color pixels of the sub-block include color pixels sensing a first color, a second color, and a third color, and the far-center region includes a first number of first pixels sensing one of the first color, the second color, and the third color, and a second number of second pixels sensing another color of the first color, the second color, and the third color, the first number of the first pixels being greater than the second number of the second pixels. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Embodiments of the present inventive concept will be more clearly understood from the following detailed description in conjunction with the accompanying drawings, in which:

[0011] Figure 1 is a block diagram of an image sensor including a pixel array according to some example embodiments;

[0012] Figure 2 and Figure 3 are diagrams of some example embodiments of pixel arrays according to example embodiments;

[0013] Figure 4 shows a pixel array and a rearranged color pattern according to some example embodiments;

[0014] Figure 5 Some example embodiments of a color filter array (CFA) unit according to some example embodiments are shown;

[0015] Figure 6 Shows Figure 4 An example of an interpolation distance for each of the color pixels included in the CFA unit;

[0016] Figures 7 to 9 Some example embodiments of pixel arrays according to some example embodiments are shown;

[0017] Fig.10 is a block diagram of an image processing unit according to some example embodiments;

[0018] FIG. 11A to FIG. 11C are diagrams of some example embodiments of pixel arrays according to some example embodiments;

[0019] Fig.12 An example of a binning process using an inter-block method and a binning process using an intra-block method according to some example embodiments is shown;

[0020] Figures 13 to 15 Some example embodiments of CFA blocks according to some example embodiments are shown;

[0021] Fig.16 Some example embodiments of color pixels according to some example embodiments are shown; and

[0022] Fig.17 and Fig.18 is a block diagram of an electronic device including a multi-camera module using an image sensor according to some example embodiments. DETAILED DESCRIPTION

[0023] Hereinafter, example embodiments will be described in detail with reference to the accompanying drawings.

[0024] Figure 1 is a block diagram of an image sensor including a pixel array according to some example embodiments.

[0025] refer to Figure 1 , the image sensor 100 may include a pixel array 110, a row driver 120, a read circuit 130, and a controller 140. The image sensor 100 may include a complementary metal oxide semiconductor (CMOS) image sensor (CIS).

[0026] The controller 140 may control the row driver 120 and the read circuit 130. The pixel array 110 may include a plurality of pixels (e.g., color pixels). Each of the pixels may include at least one photosensitive element (not shown). The photosensitive element may sense light in each pixel and generate an electrical signal according to the intensity of the sensed light. The photosensitive element may include a photodiode, a photogate, a phototransistor, etc. According to example embodiments, the pixel array 110 may include color pixels of various patterns. Each of the color pixels may generate an electrical signal associated with at least one color as a pixel signal. Although in Figure 1 Although not shown in the figure, processing such as rearrangement may be performed on the pixel signals of the pixel array 110, and the color pattern of the pixel array 110 may be converted into a specific pattern such as a Bayer pattern through the processing.

[0027] The pixel array 110 may output an electrical signal corresponding to light absorbed by the photosensitive element to the read circuit 130. The row driver 120 may output a signal for controlling each color pixel of the pixel array 110. For example, the row driver 120 may output a signal that resets the photosensitive element of each color pixel or controls the photosensitive element to output an electrical signal corresponding to light charges accumulated therein.

[0028] Read circuit 130 may receive an electrical signal from pixel array 110 and output a pixel value (or pixel data). For example, read circuit 130 may include an analog-to-digital converter (ADC) and output a digital signal corresponding to an analog signal received from pixel array 110 as pixel data.

[0029] Despite Figure 1 Although not shown in the figure, the pixel data of the image sensor 100 may be provided to an image processing unit (not shown), and the image processing unit may perform processing operations such as rearrangement based on digital signal processing. According to example embodiments, it may be defined that processing operations such as rearrangement are performed by an element (e.g., a processor) of the image sensor 100 or by a separate processing unit outside the image sensor 100.

[0030] Hereinafter, some example embodiments of color pixels of the pixel array 110 will be illustrated according to example embodiments.

[0031] Figure 2 and Figure 3 are diagrams of some example embodiments of pixel arrays according to example embodiments.

[0032] refer to Figures 1 to 3 , the image sensor 100 may include a pixel array 110, and a color filter array (CFA) may be provided in the pixel array 110 to allow each pixel to sense a specific color. In the description of the example embodiments below, the terms "color filter", "color pixel", "filter array", and "pixel array" may be defined differently. For example, a CFA may be defined as a separate element provided on a pixel array including a photosensitive element, or defined as included in a pixel array. In other words, a color pixel may be defined as including a corresponding color filter. In the description of the example embodiments below, each of a CFA unit, a CFA block, and a sub-block may be defined as including a color pixel.

[0033] The pixel array 110 may include a plurality of CFA units 111 defined in a specific unit. For example, the pixel array 110 may include a plurality of CFA units 111 in a length direction and a width direction. Each of the CFA units 111 may include a color pixel having a specific size.

[0034] Each of CFA units 111 may be defined as including a plurality of CFA blocks, and may refer to a minimum structure of the same CFA block. Figure 2 Some example embodiments are shown in which the CFA unit 111 includes M×N CFA blocks, such that the CFA unit 111 includes M CFA blocks in the width direction and N CFA blocks in the length direction (M and N are positive integers). In a high-definition image sensor such as a CIS, the number of color pixels may increase, and thus, the size of the CFA unit 111 and the size of the CFA blocks may increase.

[0035] At this time, in some example embodiments where each CFA block includes the same kind of color pixels (or pixels sensing the same color), when the original image output from the image sensor is converted into an RGB image, the distance between the color pixel (e.g., the center color pixel) and the adjacent color pixels (which can be referenced during image processing such as interpolation or extrapolation) increases, resulting in image information loss. Specifically, the larger the size of the CFA block, the larger the distance to the adjacent color pixels to be referenced, and therefore, the image loss rate also increases.

[0036] In example embodiments, the color pixels (or color filters) of the CFA unit 111 and the CFA blocks of the pixel array 110 may have a pattern that may reduce an image loss rate during image processing. In some example embodiments, the CFA unit 111 includes 4 (or 2×2) CFA blocks, and each CFA block senses at least two colors or one CFA block senses all colors applied to the pixel array 110, and some example embodiments may reduce image information loss by reducing an interpolation distance.

[0037] According to some example embodiments, Figure 3 As shown, the CFA block may include a plurality of color pixels. For example, "c" color pixels are arranged in the width direction and "d" color pixels are arranged in the length direction, so that the CFA block includes c×d color pixels (c and d are positive integers). In some example embodiments where the pixel array 110 includes red color pixels, blue color pixels, and green color pixels (referred to as red pixels, blue pixels, and green pixels, respectively), the CFA block may include red pixels, blue pixels, and green pixels in example embodiments.

[0038] In some example embodiments, a unit including a certain number of pixels in the interior (e.g., the center area or another location) of a CFA block may be referred to as a sub-block. A sub-block may include a×b color pixels (a and b are positive integers). Figure 3 As shown, red pixels R, blue pixels B, and green pixels G are all arranged together in a sub-block.

[0039] In addition, a plurality of color pixels may be arranged in the remaining region (e.g., the boundary or outer region of the CFA block) other than the sub-block in the CFA block. For example, most pixels sensing a specific color and a few pixels sensing other colors (e.g., one or two pixels) may be arranged together in the outer region. In other words, the outer region of the CFA block may include a plurality of first pixels P1 sensing a specific color and at least one second pixel P2 sensing another color.

[0040] According to some example embodiments, the CFA block may be referred to as a red CFA block, a blue CFA block, or a green CFA block according to the kind of color pixels included in the CFA block. For example, when the number of red pixels is the largest in the CFA block, the CFA block may be referred to as a red CFA block. When the first pixel P1 in the far-from-center region of the CFA block is a red pixel, the CFA block may be referred to as a red CFA block.

[0041] In some example embodiments, the first pixel P1 (or red pixel) may be mainly arranged in the out-of-center area of ​​the red CFA block, and at least one second pixel P2 (e.g., blue pixel) may be further arranged in the out-of-center area of ​​the red CFA block. For example, most of the red pixels may be arranged in the out-of-center area of ​​the red CFA block, and the blue pixels may be arranged in at least one corner of the out-of-center area of ​​the red CFA block. Similarly, most of the blue pixels may be arranged in the out-of-center area of ​​the blue CFA block, and the red pixels may be arranged in at least one corner of the out-of-center area of ​​the blue CFA block. According to some example embodiments, only green pixels may be arranged in the out-of-center area of ​​the green CFA block, or at least one pixel sensing another color different from green may also be arranged in the out-of-center area of ​​the green CFA block.

[0042] The red pixels, blue pixels, and green pixels applied to the pixel array 110 in some of the above-described example embodiments are merely examples, and the embodiments are not limited thereto. For example, color pixels involved in various filters such as a cyan filter and an RGBW filter may be applied to some example embodiments, and some example embodiments are not limited to sensing a pattern of a specific color.

[0043] Specific some example embodiments of the pixel array of the image sensor according to the example embodiments will be described below. Although in the example embodiments below the second pixel P2 is arranged in a corner away from the central area of ​​the CFA block, some example embodiments are not limited thereto. In other words, as described above, at least two second pixels P2 may be arranged in the away-from-center area of ​​the CFA block. The embodiments may be variously modified. For example, the second pixel P2 may be arranged in another position different from the corner away from the central area.

[0044] Figure 4 A pixel array and a rearranged color pattern according to some example embodiments are shown. Hereinafter, a CFA unit including 8×8 pixels is described, but in some example embodiments, the size of the CFA unit may be set differently.

[0045] According to some example embodiments, the CFA unit includes 4 CFA blocks, and thus each CFA block may include 4×4 color pixels. For example, when the above pixel array includes red pixels, green pixels, and blue pixels, each CFA block may include red pixels, green pixels, and blue pixels. Figure 4 The CFA unit shown in (e) may include Figure 4 The red CFA block shown in (a) Figure 4 The green CFA block shown in (b) and Figure 4 The blue CFA block shown in (c) of FIG. 8×8 CFA units can be transformed into Figure 4 The above-mentioned CFA unit and the Bayer pattern obtained by conversion may be referred to as a pattern CFA pair or a CFA pattern pair.

[0046] like Figure 4 As shown in (d), each of the red CFA block, the green CFA block, and the blue CFA block may include a sub-block having a specific size. For example, the sub-block may include 2×2 color pixels. According to some example embodiments, the sub-block includes a red pixel, a green pixel, and a blue pixel, and thus, red, green, and blue may all be sensed by the sub-block together. The pixel arrangement in the sub-block may vary. For example, a green pixel, a blue pixel, a red pixel, and a green pixel (or GBRG) are sequentially arranged at the upper left, upper right, lower left, and lower right of the sub-block, as shown in FIG. Figure 4 As shown in (d).

[0047] Various types of CFA units can be implemented based on the arrangement of 4 CFA blocks. For example, Figure 4The CFA unit shown in (e) of FIG. 1 shows an example, in which a green CFA block, a red CFA block, a blue CFA block, and a green CFA block are sequentially located at the upper left, upper right, lower left, and lower right of the CFA unit. For ease of description, the first two letters of the GRBG CFA block and the first two letters of the GBRG pixels of the sub-block can be used to represent the CFA unit. Figure 4 The 8×8 CFA unit shown in (e) is called a GR-GB pattern unit.

[0048] Interpolation of red, green, and blue may be required to use Figure 4 A pair of patterns shown in (e) and (f) in FIG. 8 rearranges the 8×8 CFA unit into a Bayer pattern. When interpolation of green is performed, relatively more adjacent green pixels can be selected because green includes more image edge information than red or blue. For example, adjacent green pixels required for interpolation can be selected in each of four directions (e.g., diagonal direction, reverse diagonal direction, horizontal direction, and vertical direction), and the average value of the distances from the pixel for which interpolation is calculated (e.g., the center pixel) to the adjacent green pixels can be calculated as the interpolation distance of the center pixel. Conversely, when interpolation of blue or red is performed, at least one adjacent pixel can be selected regardless of the direction relative to the pixel for which interpolation is calculated.

[0049] According to some example embodiments, at least one CFA block may have pixels sensing at least two colors in its remote-center region. Figure 4 As shown in (a), the area away from the center of the red CFA block may mainly include red pixels and one second pixel P2 (eg, a blue pixel in the lower left corner). Figure 4 As shown in (c), the area away from the center of the blue CFA block may mainly include blue pixels and one second pixel P2 (eg, a red pixel in the upper right corner). Figure 4 2 shows that the pixels in the out-of-center region of the green CFA block are all green pixels, but at least one second pixel P2 may be arranged in the out-of-center region of the green CFA block.

[0050] refer to Figure 4 In (e) and (f), the arrangement of the color pixels in the sub-blocks of each CFA block can be the same as the arrangement of the color pixels in the corresponding positions in the Bayer pattern. Therefore, the interpolation distance of each color pixel of the four sub-blocks of one CFA unit can be calculated as 0, and thus, the reduction of the interpolation distance can be achieved.

[0051] With respect to the four CFA blocks of the CFA unit, a first region DR including a color pixel of each of the four CFA blocks may be defined. The first region DR may include a second pixel P2 of a blue CFA block, a second pixel P2 of a red CFA block, and two green pixels of corresponding two green CFA blocks. The arrangement of the color pixels in the first region DR may be the same as the arrangement of the color pixels at corresponding positions in the Bayer pattern. Therefore, the number of color pixels having an interpolation distance of 0 in the CFA unit may be increased, and thus, the interpolation distance of the CFA unit may be further reduced.

[0052] In other words, according to some example embodiments, there are color pixels having an interpolation distance of 0 according to the pixel arrangement of the CFA unit, and many color pixels having an interpolation distance of 0 (e.g., the color pixels of the above-mentioned sub-blocks and the color pixels of the first region DR) may be protected. Therefore, the number of color pixels having an interpolation distance of 0 in the CFA unit may be increased, and thus, image information loss may be reduced or prevented.

[0053] Figure 5 Some example embodiments of CFA units according to some example embodiments are shown. Figure 5 In some example embodiments shown, the CFA unit includes 2×2 CFA blocks, and Figure 5 The red CFA block, green CFA block, blue CFA block and sub-block shown in (a) to (d) are respectively Figure 4 Same as those shown. Figure 5 The CFA unit shown can be converted into Figure 5 The Bayer pattern shown in (h).

[0054] Figure 5 The CFA unit shown in (e) may include a green CFA block at its upper left, a blue CFA block at its upper right, a red CFA block at its lower left, and another green CFA block at its lower right. Each of the green CFA block, the red CFA block, and the blue CFA block may include a sub-block. The sub-block may include a green pixel, a blue pixel, a red pixel, and a green pixel in sequence at its upper left, upper right, lower left, and lower right.

[0055] The CFA unit can be implemented differently depending on the arrangement of the CFA block. For example, Figure 5 The CFA unit shown in (f) may include a red CFA block at its upper left, a green CFA block at its upper right, another green CFA block at its lower left, and a blue CFA block at its lower right. Figure 5The CFA unit shown in (g) may include a blue CFA block at its upper left, a green CFA block at its upper right, another green CFA block at its lower left, and a red CFA block at its lower right.

[0056] According to some of the above example embodiments, the interpolation distance of the color pixels of the sub-block of each CFA block may be 0. The first region DR may be defined as being formed by the color pixels respectively located in the out-of-center regions of the corresponding four CFA blocks. The first region DR may include a second pixel (e.g., a red pixel) in the out-of-center region of the blue CFA block, a second pixel (e.g., a blue pixel) in the out-of-center region of the red CFA block, and a green pixel in the out-of-center region of each of the two green CFA blocks. The interpolation distance of the color pixels of the first region DR may have a value of 0.

[0057] according to Figure 5 Some example embodiments of the CFA unit in Figure 5 In some example embodiments shown in (e) of FIG. 1 , four color pixels in corresponding four corners of the CFA unit may form a first region DR. Figure 5 In some example embodiments shown in (f) of FIG. 1 , two color pixels at the middle of each of the left and right sides of the CFA unit may form a first region DR, and in the example Figure 5 In some example embodiments shown in (g), 2 color pixels at the middle of each of the top and bottom sides of the CFA unit may form the first region DR.

[0058] Figure 6 Shows Figure 4 An example of an interpolation distance for each color pixel included in a CFA unit.

[0059] According to some example embodiments, when the above method is used to calculate the interpolation distance of each color pixel to be rearranged, it is possible to obtain Figure 6 For example, in some example embodiments of the interpolation of green, two adjacent green pixels that are most adjacent to each other in each of four directions (e.g., diagonal direction, reverse diagonal direction, horizontal direction, and vertical direction) may be selected. In some example embodiments of the interpolation of red or blue, two adjacent red or blue pixels that are most adjacent to each other regardless of direction may be selected.

[0060] like Figure 6 As shown, the interpolation distance between each color pixel of the sub-block of each CFA unit and the color pixel of the first region DR can be calculated as 0. In some example embodiments of other color pixels, the interpolation distance between each color pixel of the sub-block of each CFA unit and the color pixel of the first region DR can be calculated as 0. Figure 4A pair of patterns (e) and (f) of FIG. 4 is used to reduce the distance to the adjacent color pixel to be referenced for interpolation. Therefore, the overall average value of the interpolation distance calculated in the CFA unit can be minimized, and therefore, the image information loss can also be minimized.

[0061] Hereinafter, examples of various color filter arrays that can be implemented according to example embodiments will be described. Embodiments are not limited to the specific examples of color filter arrays described below, and the specific arrangement of pixels of the color filter arrays may be partially modified as long as the effects of some example embodiments are achieved.

[0062] Figures 7 to 9 Some example embodiments of pixel arrays according to some example embodiments are shown. Figures 7 to 9 Various examples of arrangements of color pixels of sub-blocks and various examples of arrangements of color pixels in areas away from the center of the CFA block are shown.

[0063] You can Figure 7 As shown in (a) to (c) of FIG. 1 , a red CFA block, a green CFA block, and a blue CFA block are implemented respectively, and each CFA block may include Figure 7 In addition, Figure 7 Each CFA unit shown can be converted into Figure 7 The Bayer pattern shown in (i).

[0064] exist Figure 7 In some example embodiments, each sub-block of the CFA block may include a green pixel, a red pixel, a blue pixel, and a green pixel in sequence at its upper left, upper right, lower left, and lower right. Figure 7 As shown in (a) of FIG. 1 , the red pixel may be mainly arranged in the out-of-center region of the red CFA block, and a blue pixel may be further arranged as a second pixel in the out-of-center region of the red CFA block. The blue pixel may be arranged in the upper right corner of the out-of-center region of the red CFA block. Figure 7 As shown in (b), the green pixels may be arranged in the out-of-center area of ​​the green CFA block. For example, pixels sensing colors other than green may not be arranged in the out-of-center area of ​​the green CFA block. Figure 7 As shown in (c), the blue pixel can be mainly arranged in the out-of-center area of ​​the blue CFA block, and a red pixel can be further arranged as the second pixel in the out-of-center area of ​​the blue CFA block. The red pixel can be arranged in the lower left corner of the out-of-center area of ​​the blue CFA block.

[0065] As described above, according to some example embodiments of the CFA block, the color pixels of each of the red CFA block and the blue CFA block may have a specific pattern. For example, a blue pixel, a red pixel, a blue pixel, and a red pixel may be sequentially arranged from the upper right corner to the lower left corner of each of the red CFA block and the blue CFA block.

[0066] Various types of CFA units can be implemented based on the arrangement of CFA blocks. Figure 7 (e), the CFA unit may include a green CFA block at its upper left, a red CFA block at its upper right, a blue CFA block at its lower left, and another green CFA block at its lower right. Figure 7 (f), the CFA unit may include a green CFA block at its upper left, a blue CFA block at its upper right, a red CFA block at its lower left, and another green CFA block at its lower right. Figure 7 (g), the CFA unit may include a red CFA block at its upper left, a green CFA block at its upper right, another green CFA block at its lower left, and a blue CFA block at its lower right. Figure 7 (h), the CFA unit may include a blue CFA block at its upper left, a green CFA block at its upper right, another green CFA block at its lower left, and a red CFA block at its lower right.

[0067] Similar to some of the above-described example embodiments, the interpolation distance of the color pixels in the sub-block of each CFA block may have a value of 0, and the first region DR including the color pixels having the interpolation distance of 0 may be more protected. Figure 7 Some example embodiments of the CFA unit in Figure 7 In some example embodiments shown in (e) of FIG. 1 , four color pixels in corresponding four corners of the CFA unit may form a first region DR. Figure 7 In some example embodiments shown in (f) of FIG. 1 , four color pixels at the center of the CFA unit may form a first region DR. Figure 7 In some example embodiments shown in (g) of FIG. 1 , two color pixels at the middle of each of the top and bottom sides of the CFA unit may form a first region DR, and in the example Figure 7 In some example embodiments shown in (h) of FIG. 5 , 2 color pixels at the middle of each of the left and right sides of the CFA unit may form the first region DR.

[0068] Figure 8 Some other example embodiments of CFA units are shown. Figure 8As shown in (a) to (c) of FIG. 1 , a red CFA block, a green CFA block, and a blue CFA block are implemented respectively, and each CFA block may include Figure 8 The sub-block shown in (d) is shown in FIG. Figure 8 Each CFA unit shown can be converted into Figure 8 The Bayer pattern shown in (i).

[0069] exist Figure 8 In some example embodiments, each sub-block of the CFA block may include a blue pixel, a green pixel, a green pixel, and a red pixel in the order of its upper left, upper right, lower left, and lower right. Figure 8 As shown in (a), the red pixels may be mainly arranged in the out-of-center region of the red CFA block, and a blue pixel may be further arranged as a second pixel in the out-of-center region of the red CFA block. The blue pixel may be arranged in the lower right corner of the out-of-center region of the red CFA block. Figure 8 As shown in (b), the green pixels may be arranged in the out-of-center area of ​​the green CFA block. Pixels sensing colors other than green may not be arranged in the out-of-center area of ​​the green CFA block. Figure 8 As shown in (c), the blue pixel can be mainly arranged in the out-of-center area of ​​the blue CFA block, and a red pixel can be further arranged as the second pixel in the out-of-center area of ​​the blue CFA block. The red pixel can be arranged in the upper left corner of the out-of-center area of ​​the blue CFA block.

[0070] Similar to the above example embodiments, various types of CFA units may be implemented. Figure 8 (e), the CFA unit may include a green CFA block at its upper left, a red CFA block at its upper right, a blue CFA block at its lower left, and another green CFA block at its lower right. Figure 8 (f), the CFA unit may include a green CFA block at its upper left, a blue CFA block at its upper right, a red CFA block at its lower left, and another green CFA block at its lower right. Figure 8 (g), the CFA unit may include a red CFA block at its upper left, a green CFA block at its upper right, another green CFA block at its lower left, and a blue CFA block at its lower right. Figure 8 (h), the CFA unit may include a blue CFA block at its upper left, a green CFA block at its upper right, another green CFA block at its lower left, and a red CFA block at its lower right.

[0071] according to Figure 8 Some example embodiments of the CFA unit in Figure 8 In some example embodiments shown in (e) of FIG. 1 , two color pixels in the middle of each of the left and right sides of the CFA unit may form a first region DR. Figure 8 In some example embodiments shown in (f) of FIG. 1 , two color pixels at the middle of each of the top and bottom sides of the CFA unit may form a first region DR. Figure 8 In some example embodiments shown in (g) of FIG. 1 , four color pixels at the center of the CFA unit may form a first region DR, and Figure 8 In some example embodiments shown in (h), 4 color pixels in corresponding 4 corners of the CFA unit may form the first region DR.

[0072] Fig. 9 Some other example embodiments of CFA units are shown. Fig. 9 As shown in (a) to (c) of FIG. 1 , a red CFA block, a green CFA block, and a blue CFA block are implemented respectively, and each CFA block may include Fig. 9 The sub-block shown in (d) is shown in FIG. Fig. 9 Each CFA unit shown can be converted into Fig. 9 The Bayer pattern shown in (i).

[0073] exist Fig. 9 In some example embodiments, each sub-block of the CFA block may include a red pixel, a green pixel, a green pixel, and a blue pixel in the order of its upper left, upper right, lower left, and lower right. Fig. 9 As shown in (a), the red pixels may be mainly arranged in the out-of-center region of the red CFA block, and a blue pixel may be further arranged as a second pixel in the out-of-center region of the red CFA block. The blue pixel may be arranged in the upper left corner of the out-of-center region of the red CFA block. Fig. 9 As shown in (b), the green pixels may be arranged in the out-of-center area of ​​the green CFA block. Pixels sensing colors other than green may not be arranged in the out-of-center area of ​​the green CFA block. Fig. 9 As shown in (c), the blue pixel can be mainly arranged in the out-of-center area of ​​the blue CFA block, and a red pixel can be further arranged as the second pixel in the out-of-center area of ​​the blue CFA block. The red pixel can be arranged in the lower right corner of the out-of-center area of ​​the blue CFA block.

[0074] Similar to some of the above-mentioned example embodiments, various types of CFA units can be implemented. Fig. 9(e), the CFA unit may include a green CFA block at its upper left, a red CFA block at its upper right, a blue CFA block at its lower left, and another green CFA block at its lower right. Fig. 9 (f), the CFA unit may include a green CFA block at its upper left, a blue CFA block at its upper right, a red CFA block at its lower left, and another green CFA block at its lower right. Fig. 9 (g), the CFA unit may include a red CFA block at its upper left, a green CFA block at its upper right, another green CFA block at its lower left, and a blue CFA block at its lower right. Fig. 9 (h), the CFA unit may include a blue CFA block at its upper left, a green CFA block at its upper right, another green CFA block at its lower left, and a red CFA block at its lower right.

[0075] according to Fig. 9 Some example embodiments of the CFA unit in Fig. 9 In some example embodiments shown in (e) of FIG. 1 , two color pixels at the middle of each of the top and bottom sides of the CFA unit may form a first region DR. Fig. 9 In some example embodiments shown in (f) of FIG. 1 , two color pixels in the middle of each of the left and right sides of the CFA unit may form a first region DR. Fig. 9 In some example embodiments shown in (g) of FIG. 1 , four color pixels in corresponding four corners of the CFA unit may form a first region DR, and in the example Fig. 9 In some example embodiments shown in (h), 4 color pixels at the center of the CFA unit may form the first region DR.

[0076] According to some of the above-described example embodiments, the color pattern of the pixel array may be formed differently, and a plurality of colors (e.g., red (R), green (G), and blue (B)) may all be sensed together by a CFA block having a specific size. Therefore, the distance from the adjacent color pixel for interpolation and / or extrapolation may be reduced, and image information loss may also be reduced. When the color pixels in the far-center region of the CFA block are arranged according to some of the above-described example embodiments, the number of color pixels having an interpolation distance of 0 may be increased in the CFA unit, and thus the total interpolation distance of the CFA unit may be reduced. As a result, image information loss during color conversion may be reduced. Although it has been described in the above embodiments that the far-center region of the CFA block includes a different color pixel, various modifications may be made to the far-center region of the CFA block to achieve the effects of some of the example embodiments. Although it has been described in the above embodiments that the second pixel sensing a color different from green is not arranged in the far-center region of the green CFA block, it may also be arranged in the far-center region of the green CFA block according to some of the example embodiments.

[0077] Fig.10 is a block diagram of an example of an image processing unit including an image sensor according to some example embodiments.

[0078] refer to Fig.10 , the image processing unit (or image processing device 200) may include: a pixel array 210 including a CFA unit 211; and an image processor 220 that performs image processing using pixel values ​​from the pixel array 210. According to some example embodiments, the above-mentioned image sensor may be defined as including at least some configurations of the pixel array 210 and the image processor 220. For example, the image sensor may be defined as including a configuration for binning processing and / or a configuration for rearrangement processing.

[0079] The pixel array 210 may include a CFA unit 211 having various patterns according to some of the above-described example embodiments, and the pixel values ​​of the color pixels arranged in the pixel array 210 may be provided to the image processor 220. The image processor 220 may include various logic units that process the pixel values ​​to generate image data. For example, the image processor 220 may include a merging processor 221 and a rearrangement processor 222. Under the control of the merging processor 221, image processing may be performed using pixel values ​​with a relatively high resolution when the high-definition mode is selected, or image processing may be performed using pixel values ​​with a relatively low resolution when the low-definition mode is selected. The rearrangement processor 222 may perform interpolation of pixels related to the rearrangement process. According to some of the above-described example embodiments, pixel values ​​corresponding to the Bayer pattern may be generated using the pixel values ​​of the CFA unit.

[0080] According to some example embodiments, pixel array 210 may include the CFA unit (or CFA block) described in the above embodiments, and may perform a merging process based on the arrangement of color pixels that has been described in some example embodiments. As a result of performing the merging process, the size of image data processed by image processor 220 may be reduced, and the size of data per frame may also be reduced, so that a high frame rate may be maintained in video mode.

[0081] According to some example embodiments, the merging process may be performed based on various methods. For example, the merging process may be performed based on the following method: the image processor 220 performs digital addition of pixel values ​​of color pixels of the pixel array 210. Alternatively or additionally, the merging process may be performed based on the following method: the electrical signals of at least two color pixels of the pixel array 210 are added in an analog manner.

[0082] According to some example embodiments, color pixels whose signals are added in the merging process may be selected differently. For example, according to the intra-block method, color pixels sensing the same color in one CFA block may be selected, and the signals of the color pixels may be added in an analog or digital manner. For example, all or some color pixels sensing the same color in one CFA block may be selected.

[0083] According to the inter-block method, color pixels sensing the same color in at least two CFA blocks may be selected, and the signals of the color pixels may be added in an analog or digital manner. For example, the signals of color pixels sensing the same color in one CFA block and in at least one row or column of an adjacent CFA block may be added.

[0084] An example of performing a combining process by adding analog signals according to some example embodiments will be described below. FIG. 11A to FIG. 11C is a diagram of some example embodiments of pixel arrays according to some example embodiments.

[0085] Fig.11A An example is shown where the pixel array includes a plurality of color pixels, and "n" color pixels CP1 to CPn share one floating diffusion region FD with each other (n is a positive integer). For example, at least some color pixels sensing the same color in one CFA block may share the floating diffusion region FD with each other. According to some example embodiments, all or some color pixels sensing the same color in a CFA block may share the floating diffusion region FD with each other.

[0086] Each color pixel may include a photodiode PD and a transfer gate TG. For example, when the transfer gates TG of each "n" color pixel CP1 to CPn are all turned on, the photocharges of the "n" color pixels CP1 to CPn can be provided to the floating diffusion region FD at the same time. When the transfer gates TG of each "n" color pixel CP1 to CPn are controlled respectively, the photocharges of the "n" color pixels CP1 to CPn can be provided to the floating diffusion region FD at different times. For example, when the merge mode is not performed, the photocharges of the "n" color pixels CP1 to CPn are provided to the floating diffusion region FD at different times. When the merge mode is performed, the photocharges of the "n" color pixels CP1 to CPn are provided to the floating diffusion region FD at the same time, so that the signals of the "n" color pixels CP1 to CPn can be added in the floating diffusion region FD.

[0087] Fig. 11B and Fig. 11C Example operations in merge mode according to some example embodiments are shown. Fig. 11B and Fig. 11C A plurality of color pixels of a CFA unit are shown. Fig. 11B and Fig. 11C The Figure 4 An example of a CFA unit in which a merge process is performed. Fig. 11B and Fig. 11C In the example of , the intra-block method is applied to the merging process, and thus, signals of pixels sensing the same color in the CFA block are added. Fig. 11B An example of performing a merge process on the upper two CFA blocks of a CFA unit is shown. Fig. 11C An example of performing a merge process on the next two CFA blocks of the CFA unit is shown.

[0088] refer to Fig. 11B and Fig. 11C, each CFA block can be divided into a plurality of groups based on a method of sharing a floating diffusion region FD. For example, the green CFA block at the upper left of the CFA unit can be divided into groups G00, G01, G10, and G11, which are located at the upper left, upper right, lower left, and lower right of the green CFA block, respectively, and each group includes 2×2 color pixels. The 4 color pixels in each of the groups G00 to G11 can share one floating diffusion region FD with each other. When the merging process is performed, the photocharges of the color pixels sensing the same color in each group can be provided to the floating diffusion region FD. For example, in some example embodiments of the green CFA block, the photocharges of the 4 green pixels in group G00 can be provided to the floating diffusion region FD, the photocharges of the 3 green pixels in group G01 can be provided to the floating diffusion region FD, the photocharges of the 3 green pixels in group G10 can be provided to the floating diffusion region FD, and the photocharges of the 4 green pixels in group G11 can be provided to the floating diffusion region FD. Similarly, the photocharges of the 3 pixels in each group of the red CFA block can be provided to the floating diffusion region FD.

[0089] A signal (e.g., a sum signal) provided from each of the groups G00 to G11 may be provided to a read circuit RC, and a pixel value corresponding to the sum signal may be output from the read circuit RC. The pixel values ​​of the groups G00 to G11 may be added by analog or digital calculation processing, and a green pixel value corresponding to the green CFA block may be calculated by addition.

[0090] When the binning process is not performed, each of the 4×4 color pixels of the green CFA block is respectively connected to the floating diffusion region FD and transfers photo charges to the floating diffusion region FD.

[0091] Fig. 11CA blue CFA block at the bottom left of the CFA unit and a green CFA block at the bottom right of the CFA unit are shown. When a merging process is performed, the photocharges of the 3 blue pixels in each group in the blue CFA block can be provided to the floating diffusion region FD and added. For example, the blue CFA block at the bottom left of the CFA unit can be divided into groups B00, B01, B10, and B11, which are located at the top left, top right, bottom left, and bottom right of the blue CFA block, respectively, and each group includes 2×2 color pixels. The 4 color pixels in each of the groups B00 to B11 can share one floating diffusion region FD with each other. When a merging process is performed, the photocharges of the color pixels sensing the same color in each group can be provided to the floating diffusion region FD. For example, in some example embodiments of the blue CFA block, the photo charges of the three blue pixels in group B00 may be provided to the floating diffusion region FD, the photo charges of the three blue pixels in group B01 may be provided to the floating diffusion region FD, the photo charges of the three blue pixels in group B10 may be provided to the floating diffusion region FD, and the photo charges of the three blue pixels in group B11 may be provided to the floating diffusion region FD.

[0092] In some example embodiments of the green CFA block, some groups each include 4 green pixels, and the photocharges of the 4 green pixels may be provided to the floating diffusion region FD and added together. Other groups each include 3 green pixels, and the photocharges of the 3 green pixels may be provided to the floating diffusion region FD and added together.

[0093] Fig.12 Examples of a merge process using an inter-block method and a merge process using an intra-block method according to some example embodiments are illustrated. Fig.12 An example of a merging process for each of the red CFA block, the green CFA block, and the blue CFA block is shown.

[0094] In the red CFA block, 3 color pixels in the sub-block and 1 color pixel in the area away from the center sense colors other than red. Since the merging process is performed based on the intra-block method, the 12 red pixels of the red CFA block can be added. Similarly, the 14 green pixels of the green CFA block can be added, and the 12 blue pixels of the blue CFA block can be added.

[0095] In some example embodiments of the inter-block method, when calculating the pixel value corresponding to a CFA block, the color pixels of at least one neighboring CFA block may also be used. Fig.12 In the example shown, color pixels in one row of adjacent CFA blocks or color pixels in two rows of adjacent CFA blocks may also be used.

[0096] For example, for a red CFA block, the signals corresponding to 12 red pixels of the red CFA block and the signals of red pixels in a row of CFA blocks adjacent to the red CFA block may be added. Alternatively or additionally, the signals corresponding to 12 red pixels of the red CFA block and the signals of red pixels in two rows of CFA blocks adjacent to the red CFA block may be added. Similarly, for a blue CFA block, the signals corresponding to 12 blue pixels of the blue CFA block and the signals of blue pixels in one or two rows of CFA blocks adjacent to the blue CFA block may be added.

[0097] According to some example embodiments, when a merging process is performed on a green CFA block based on an inter-block method, the green CFA block includes relatively more green pixels, and thus, Fig.12 An example is shown in which the signals of green pixels in one row of a CFA block adjacent to the green CFA block are used in the merging process. However, example embodiments are not limited thereto. Green pixels in at least two rows of a CFA block adjacent to the green CFA block may be further used for signal addition.

[0098] Figures 13 to 15 Some example embodiments of a CFA block according to some example embodiments are shown. Fig.13 The red CFA block is shown, Fig.14 The blue CFA block is shown, and Fig.15 A green CFA block is shown. Figures 13 to 15 An example of further reducing the interpolation distance by arranging color pixels in areas away from the center of each CFA block is shown. Figures 13 to 15 Shows Figure 4 The CFA block shown in FIG. 1 is a block of FIG. 1 , but the example embodiments may also be applied to the other various CFA blocks described above.

[0099] refer to Fig.13 As described above, the red CFA block may include a sub-block including 2×2 color pixels. Although the red pixels are mainly arranged in a region away from the center of the red CFA block, the blue pixels may be arranged in a corner (e.g., the lower left corner) away from the center. Therefore, the red pixel, the blue pixel, the red pixel, and the blue pixel may be sequentially arranged in a specific direction (e.g., from the upper right to the lower left) of the red CFA block.

[0100] In addition, pixels sensing at least one other color other than red may be arranged in another corner of the red CFA block away from the central area. Fig.13 (a) shows that the green pixel is arranged in the lower right corner. Fig.13 (b) shows that two green pixels are arranged in the lower right corner and the upper left corner respectively. Fig.13The green pixels in the far-center area shown in (a) and (b) sense the same color (ie, green) as the green pixels in the corresponding positions in the Bayer pattern, so the interpolation distance of the pixels of the red CFA block can be further reduced.

[0101] refer to Fig.14 , although the blue pixels are mainly arranged in the away-from-center area of ​​the blue CFA block, the red pixels may be arranged in the corners (e.g., the upper right corner) away from the center area. Therefore, the red pixels, the blue pixels, the red pixels, and the blue pixels may be arranged sequentially in a specific direction (e.g., from the upper right to the lower left) of the blue CFA block. In addition, similar to Fig.13 In some example embodiments, pixels sensing at least one other color other than blue may be arranged in another corner of the blue CFA block away from the central area. Fig.14 (a) shows that the green pixel is arranged in the lower right corner. Fig.14 (b) shows that two green pixels are arranged in the lower right corner and the upper left corner respectively. Fig.14 The embodiments shown in (a) and (b) can further reduce the interpolation distance of the pixels of the blue CFA block.

[0102] refer to Fig.15 , pixels sensing at least one other color other than green (e.g., red pixels and / or blue pixels) may be arranged in a region away from the center of the green CFA block. Fig.15 (a) of FIG. 5 shows that the blue pixel is arranged in a corner (eg, the lower left corner) of the green CFA block away from the central area. Fig.15 (b) shows that the blue pixel is arranged at the lower left corner of the green CFA block away from the center area, and the red pixel is arranged at the upper right corner of the green CFA block away from the center area. Fig.15 The color pixels (e.g., blue pixels and / or red pixels) in the away-from-center areas shown in (a) and (b) sense the same color as the color pixels in the corresponding positions in the Bayer pattern, so the interpolation distance of the pixels of the green CFA block can be further reduced.

[0103] Fig.16 Some example embodiments of color pixels according to some example embodiments are shown.

[0104] According to some example embodiments described above, a color pixel may include a corresponding color filter and a photosensitive element (eg, a photodiode) that senses light (or color).

[0105] According to some example embodiments, a color pixel may include a plurality of sub-pixels. Fig.16An example is shown in which a color pixel includes 2×2 sub-pixels, but the color pixel may include more sub-pixels. A photosensitive element is provided for each sub-pixel, and therefore, a plurality of photosensitive elements may be provided for a color pixel. Fig.16 Although not shown in the figure, the image sensor may include a plurality of micro lenses (not shown), and the micro lenses may be provided for each pixel or each sub-pixel.

[0106] When calculating the pixel signal corresponding to the color pixel, at least some of the signals generated by the plurality of sub-pixels may be used. Fig.16 If the upper left color pixel is a red pixel and a plurality of sub-pixels sense red, a pixel signal for the red pixel may be generated by processing at least some of the signals generated by the sub-pixels.

[0107] The embodiments may be implemented differently. For example, a color pixel may provide a signal generated by sensing at least two colors. For example, a color filter sensing different colors may be arranged for a plurality of sub-pixels of a color pixel, and the color pixel may be implemented differently according to some of the above-described example embodiments, as long as the interpolation distance can be reduced.

[0108] Fig.17 is a block diagram illustrating an electronic device including a multi-camera module using an image sensor according to some example embodiments. Fig.18 yes Fig.17 Detailed block diagram of the camera module.

[0109] refer to Fig.17 , the electronic device 1000 may include a camera module group 1100 , an application processor 1200 , a power management integrated circuit (PMIC) 1300 , and an external memory 1400 .

[0110] The camera module group 1100 may include a plurality of camera modules 1100a, 1100b, and 1100c. Fig.17 1100a, 1100b, and 1100c are shown, but example embodiments are not limited thereto. In some example embodiments, camera module group 1100 may be modified to include only two camera modules. In some example embodiments, camera module group 1100 may be modified to include "n" camera modules, where "n" is a natural number with a minimum of 4.

[0111] The following will refer to Fig.18 A detailed configuration of the camera module 1100b is described. The following description may also be applied to the other camera modules 1100a and 1100c.

[0112] refer to Fig.18, the camera module 1100 b may include a prism 1105 , an optical path folding element (OPFE) 1110 , an actuator 1130 , an image sensing device 1140 , and a storage device 1150 .

[0113] The prism 1105 may include a reflective surface 1107 of a light reflective material, and may change a path of light L incident from the outside.

[0114] In some example embodiments, the prism 1105 may change the path of the light L incident along the first direction X to a second direction Y perpendicular to the first direction X. The prism 1105 may rotate the reflective surface 1107 of the light reflective material about the central axis 1106 in the direction A or rotate the central axis 1106 in the direction B, so that the path of the light L incident along the first direction X is changed to the second direction Y perpendicular to the first direction X. At this time, the OPFE 1110 may move in a third direction Z perpendicular to the first direction X and the second direction Y.

[0115] In some example embodiments, the A-direction maximum rotation angle of the prism 1105 may be less than or equal to 15 degrees in the positive (+) A direction and may be greater than 15 degrees in the negative (−) A direction, but the embodiment is not limited thereto.

[0116] In some example embodiments, the prism 1105 may move at an angle of about 20 degrees or within a range of about 10 degrees to about 20 degrees or about 15 degrees to about 20 degrees in the positive B direction or the negative B direction. At this time, the angle at which the prism 1105 moves in the positive B direction may be the same as the angle at which the prism 1105 moves in the negative B direction or is similar within a difference of about 1 degree.

[0117] In some example embodiments, the prism 1105 may move the reflective surface 1107 of the light reflective material in a third direction Z parallel to the extension direction of the central axis 1106 .

[0118] OPFE 1110 may include, for example, "m" optical lenses, where "m" is a natural number. The "m" lenses may move in the second direction Y and change the optical zoom ratio of camera module 1100b. For example, when the default optical zoom ratio of camera module 1100b is Z, the optical zoom ratio of camera module 1100b may be changed to 3Z, 5Z, or more by moving the "m" optical lenses included in OPFE 1110.

[0119] The actuator 1130 may move the OPFE 1110 or the optical lens to a specific position. For example, the actuator 1130 may adjust the position of the optical lens so that the image sensor 1142 is located at the focal length of the optical lens for accurate sensing.

[0120] The image sensing device 1140 may include an image sensor 1142, a control logic 1144, and a memory 1146. The image sensor 1142 may sense an image of an object using light L provided through an optical lens. According to some of the above-described example embodiments, the image sensor 1142 may include a pixel array, and a color pattern of a plurality of color pixels of the pixel array may follow the patterns of the CFA unit, CFA block, and sub-block in some of the above-described example embodiments.

[0121] The control logic 1144 may generally control the operation of the camera module 1100b. For example, the control logic 1144 may control the operation of the camera module 1100b according to a control signal provided through the control signal line CSLb.

[0122] The memory 1146 may store information required for the operation of the camera module 1100b, such as calibration data 1147. The calibration data 1147 may include information required for the camera module 1100b to generate image data using the light L provided from the outside. For example, the calibration data 1147 may include information about the angle of the above-mentioned rotation, information about the focal length, information about the optical axis, etc. When the camera module 1100b is implemented as a multi-state camera in which the focal length varies with the position of the optical lens, the calibration data 1147 may include a value of the focal length for each position (or state) of the optical lens and information about autofocus.

[0123] The storage device 1150 may store image data sensed by the image sensor 1142. The storage device 1150 may be disposed outside the image sensing device 1140, and may form a stack with a sensor chip of the image sensing device 1140. In some embodiments, the storage device 1150 may include an electrically erasable programmable read-only memory (EEPROM), but the embodiment is not limited thereto.

[0124] In some example embodiments, each of the camera modules 1100a, 1100b, and 1100c may include an actuator 1130. Therefore, the camera modules 1100a, 1100b, and 1100c may include calibration data 1147 that is the same or different in the camera modules 1100a, 1100b, and 1100c according to the operation of the actuator 1130 included in each of the camera modules 1100a, 1100b, and 1100c.

[0125] In some example embodiments, one of the camera modules 1100a, 1100b, and 1100c (e.g., camera module 1100b) may be a folded lens type including the prism 1105 and the OPFE 1110, and the other camera modules (e.g., camera modules 1100a and 1100c) may be a vertical type not including the prism 1105 and the OPFE 1110. However, example embodiments are not limited thereto.

[0126] In some example embodiments, one of the camera modules 1100a, 1100b, and 1100c (e.g., camera module 1100c) may include a vertical depth camera that extracts depth information using infrared (IR) rays. In this case, the application processor 1200 may generate a three-dimensional (3D) depth image by merging image data provided from the depth camera with image data provided from another camera module (e.g., camera module 1100a or 1100b).

[0127] In some example embodiments, at least two camera modules (e.g., 1100a and 1100b) of the camera modules 1100a, 1100b, and 1100c may have different fields of view. In this case, two camera modules (e.g., 1100a and 1100b) of the camera modules 1100a, 1100b, and 1100c may have different optical lenses, respectively, but the embodiment is not limited thereto.

[0128] In some example embodiments, the camera modules 1100a, 1100b, and 1100c may have different fields of view from each other. In this case, the camera modules 1100a, 1100b, and 1100c may have different optical lenses, respectively, but the embodiment is not limited thereto.

[0129] In some example embodiments, the camera modules 1100a, 1100b, and 1100c may be physically separated from each other. In other words, the sensing area of ​​the image sensor 1142 is not divided and used by the camera modules 1100a, 1100b, and 1100c, but the image sensor 1142 may be independently included in each of the camera modules 1100a, 1100b, and 1100c.

[0130] Return to reference Fig.17 , the application processor 1200 may include an image processing unit 1210, a memory controller 1220, and an internal memory 1230. The application processor 1200 may be implemented separately from the camera modules 1100a, 1100b, and 1100c. For example, the application processor 1200 and the camera modules 1100a, 1100b, and 1100c may be implemented in different semiconductor chips.

[0131] The image processing unit 1210 may include a plurality of sub-processors 1212 a , 1212 b , and 1212 c , an image generator 1214 , and a camera module controller 1216 .

[0132] The image processing unit 1210 may include as many sub-processors 1212a, 1212b, and 1212c as the camera modules 1100a, 1100b, and 1100c.

[0133] The image data generated from each camera module 1100a, 1100b, or 1100c may be provided to a corresponding one of the sub-processors 1212a, 1212b, and 1212c through a corresponding one of the separate image signal lines ISLa, ISLb, and ISLc. For example, the image data generated from the camera module 1100a may be provided to the sub-processor 1212a through the image signal line ISLa, the image data generated from the camera module 1100b may be provided to the sub-processor 1212b through the image signal line ISLb, and the image data generated from the camera module 1100c may be provided to the sub-processor 1212c through the image signal line ISLc. Such image data transmission may be performed using, for example, a camera serial interface (CSI) based on a mobile industry processor interface (MIPI), but the embodiment is not limited thereto.

[0134] In some example embodiments, a single sub-processor may be provided for multiple camera modules. Fig.13 Differently, sub-processors 1212a and 1212c may not be separated but may be integrated into a single sub-processor, and image data provided from camera module 1100a or camera module 1100c may be selected by a selection element (e.g., a multiplexer) and then provided to the integrated sub-processor.

[0135] The image data provided to each of the sub-processors 1212a, 1212b, and 1212c may be provided to the image generator 1214. The image generator 1214 may generate an output image using the image data provided from each of the sub-processors 1212a, 1212b, and 1212c according to the image generation information or the mode signal.

[0136] In detail, the image generator 1214 can generate an output image by merging at least a portion of the respective image data generated from the camera modules 1100a, 1100b, and 1100c having different fields of view, respectively, according to the image generation information or the mode signal. Alternatively or additionally, the image generator 1214 can generate an output image by selecting one of the plurality of image data generated from the camera modules 1100a, 1100b, and 1100c having different fields of view, respectively, according to the image generation information or the mode signal.

[0137] In some example embodiments, the image generation information may include a zoom signal or a zoom factor.In some example embodiments, the mode signal may be based on a mode selected by a user.

[0138] When the image generation information includes a zoom signal or a zoom factor and the camera modules 1100a, 1100b, and 1100c have different fields of view, the image generator 1214 may perform different operations according to different kinds of zoom signals. For example, when the zoom signal is a first signal, the image generator 1214 may merge the image data output from the camera module 1100a and the image data output from the camera module 1100c, and then use the merged image signal and the image data output from the camera module 1100b (not used in the merge) to generate an output image. When the zoom signal is a second signal different from the first signal, the image generator 1214 may not perform the image data merge, but select one of the multiple image data output from the camera modules 1100a to 1100c to generate an output image. However, the example embodiments are not limited thereto, and the method of processing image data may be changed when necessary.

[0139] In some example embodiments, the image generator 1214 may receive a plurality of image data having different exposure times from at least one of the sub-processors 1212a, 1212b, and 1212c, and perform high dynamic range (HDR) processing on the plurality of image data to generate merged image data having an increased dynamic range.

[0140] The camera module controller 1216 may provide a control signal to each of the camera modules 1100a, 1100b, and 1100c. The control signal generated by the camera module controller 1216 may be provided to a corresponding one of the camera modules 1100a, 1100b, and 1100c through a corresponding one of the control signal lines CSLa, CSLb, and CSLc separated from each other.

[0141] One of the camera modules 1100a, 1100b, and 1100c (e.g., camera module 1100b) may be designated as a master camera and the other camera modules (e.g., 1100a and 1100c) may be designated as slave cameras according to a mode signal or an image generation signal including a zoom signal. Such designation information may be included in a control signal and provided to a corresponding one of the camera modules 1100a, 1100b, and 1100c through a corresponding one of control signal lines CSLa, CSLb, and CSLc separated from each other.

[0142] The camera module operating as a master camera or a slave camera may be changed according to a zoom factor or an operation mode signal. For example, when the field of view of the camera module 1100a is greater than the field of view of the camera module 1100b and the zoom factor indicates a low zoom ratio, the camera module 1100b may operate as a master camera, and the camera module 1100a may operate as a slave camera. On the contrary, when the zoom factor indicates a high zoom ratio, the camera module 1100a may operate as a master camera, and the camera module 1100b may operate as a slave camera.

[0143] In some example embodiments, the control signal provided from the camera module controller 1216 to each of the camera modules 1100a, 1100b, and 1100c may include a synchronization enable signal. For example, when the camera module 1100b is a master camera and the camera modules 1100a and 1100c are slave cameras, the camera module controller 1216 may send the synchronization enable signal to the camera module 1100b. The camera module 1100b provided with the synchronization enable signal may generate a synchronization signal based on the synchronization enable signal, and may provide the synchronization signal to the camera modules 1100a and 1100c through the synchronization signal line SSL. The camera modules 1100a, 1100b, and 1100c may be synchronized by the synchronization signal, and may send the image data to the application processor 1200.

[0144] In some example embodiments, the control signal provided from the camera module controller 1216 to each of the camera modules 1100a, 1100b, and 1100c may include mode information according to the mode signal. The camera modules 1100a, 1100b, and 1100c may operate in the first operation mode or the second operation mode with respect to the sensing speed based on the mode information.

[0145] In the first operation mode, the camera modules 1100a, 1100b, and 1100c may generate image signals at a first speed (e.g., at a first frame rate), encode the image signals at a second speed higher than the first speed (e.g., at a second frame rate higher than the first frame rate), and transmit the encoded image signals to the application processor 1200. At this time, the second speed may be up to 30 times the first speed.

[0146] The application processor 1200 may store the received image signal (e.g., the encoded image signal) in the internal memory 1230 therein or in the external memory 1400 outside the application processor 1200. Thereafter, the application processor 1200 may read the encoded image signal from the internal memory 1230 or the external memory 1400, decode the encoded image signal, and display image data generated based on the decoded image signal. For example, a corresponding one of the sub-processors 1212a, 1212b, and 1212c of the image processing unit 1210 may perform decoding, and may also perform image processing on the decoded image signal.

[0147] In the second operation mode, camera modules 1100a, 1100b, and 1100c may generate image signals at a third speed lower than the first speed (e.g., at a third frame rate lower than the first frame rate) and transmit the image signals to application processor 1200. The image signals provided to application processor 1200 may not have been encoded. Application processor 1200 may perform image processing on the image signals or store the image signals in internal memory 1230 or external memory 1400.

[0148] The PMIC 1300 may provide power, such as a power supply voltage, to each of the camera modules 1100a, 1100b, and 1100c. For example, under the control of the application processor 1200, the PMIC 1300 may provide a first power to the camera module 1100a through a power signal line PSLa, provide a second power to the camera module 1100b through a power signal line PSLb, and provide a third power to the camera module 1100c through a power signal line PSLc.

[0149] The PMIC 1300 may generate power corresponding to each of the camera modules 1100a, 1100b, and 1100c and adjust the power level in response to the power control signal PCON from the application processor 1200. The power control signal PCON may include a power adjustment signal for each operating mode of the camera modules 1100a, 1100b, and 1100c. For example, the operating mode may include a low power mode. At this time, the power control signal PCON may include information related to the camera module operating in the low power mode and the power level to be set. The same or different levels of power may be provided to the camera modules 1100a, 1100b, and 1100c, respectively. The power level may be changed dynamically.

[0150] It will be understood that any or all of the devices, controllers, generators, decoders, units, modules, circuits, processors, etc. according to any example embodiments described herein (including Figure 1Any or all of the components of the controller 140 and the read circuit 130 shown, Fig.10 The image processor 220, the merging processor 221 and the rearrangement processor 222 are shown. Fig.17 The application processor 1200 and its subcomponents shown, Fig.18 The image sensing device 1140 shown and its sub-elements, any combination thereof, etc.) may be included in one or more instances of processing circuitry (e.g., hardware including logic circuitry, a hardware / software combination such as a processor executing software, or a combination thereof), may include one or more instances of processing circuitry, and / or may be implemented by one or more instances of processing circuitry. In some example embodiments, the one or more instances of processing circuitry may include, but are not limited to, a central processing unit (CPU), an application processor (AP), an arithmetic logic unit (ALU), a graphics processing unit (GPU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a system on a chip (SoC), a programmable logic unit, a microprocessor, or an application specific integrated circuit (ASIC), etc. In some example embodiments, any memory, memory unit, etc. described herein may include a non-transitory computer-readable storage device (e.g., a solid-state drive (SSD)) storing an instruction program, and one or more instances of processing circuitry may be configured to execute the instruction program to implement any or all of the functions of the device, controller, decoder, unit, module, etc. according to any example embodiment described herein, including any operation method that is the same as any operation method described herein.

[0151] although Figure 1 Each element of the controller 140 and the read circuit 130 shown, Fig.10 The image processor 220, the merging processor 221 and the rearrangement processor 222 are shown. Fig.17 The application processor 1200 and its subcomponents shown, Fig.18 The image sensing device 1140 and its sub-elements shown are shown as being different, but the example embodiments are not limited thereto, and certain functions of one of the above may be performed by other features of the related figure. This may also be the case with additional elements within the above items described in the example embodiments herein.

[0152] While the inventive concept has been particularly shown and described with reference to embodiments thereof, it will be understood that various changes in form and details may be made therein without departing from the spirit and scope of the appended claims.

Claims

1. A pixel array of an image sensor, the pixel array comprising: a plurality of color filter array (CFA) units having a specific size, each of the plurality of CFA units including a plurality of CFA blocks in a width direction of the CFA unit and in a length direction of the CFA unit, wherein each of the plurality of CFA blocks includes a sub-block and an out-of-center region different from the sub-block, the sub-block is at a central region of each of the plurality of CFA blocks and includes m×n color pixels, and the out-of-center region includes other color pixels, wherein “m” and “n” are integers of at least 2, The m×n color pixels of the sub-block include color pixels for sensing a first color, a second color, and a third color, and the far-from-center area includes a first number of first pixels for sensing the first color and a second number of second pixels for sensing at least one color selected from the second color and the third color, the first number of the first pixels is greater than the second number of the second pixels, and each color pixel on the boundary of the sub-block is adjacent to at least one first pixel of the far-from-center area in a horizontal direction or a vertical direction, and Wherein, at least two CFA blocks in one CFA unit have different first colors.

2. The pixel array according to claim 1, wherein: The first color, the second color, and the third color include red, green, and blue; and Each of the plurality of CFA units includes 2×2 CFA blocks in a width direction and a length direction of the CFA unit, wherein a color of the CFA block is determined by the first pixel, wherein the 2×2 CFA blocks include 2 green CFA blocks, 1 red CFA block, and 1 blue CFA block, and Each of the 2×2 CFA blocks includes 4×4 color pixels, and a sub-block of each of the 2×2 CFA blocks includes 2×2 color pixels.

3. The pixel array according to claim 2, wherein: The off-center region of the red CFA block includes a blue pixel in at least one corner of the red CFA block as the second pixel and includes a plurality of red pixels as the first pixel.

4. The pixel array according to claim 3, wherein: A red pixel, a blue pixel, a red pixel, and a blue pixel are sequentially arranged in a diagonal direction from one corner of the red CFA block.

5. The pixel array according to claim 2, wherein: The out-of-center region of the blue CFA block includes a red pixel in at least one corner of the blue CFA block as the second pixel and includes a plurality of blue pixels as the first pixel.

6. The pixel array according to claim 2, wherein: The out-of-center region of each of the two green CFA blocks includes only green pixels.

7. The pixel array according to claim 2, wherein: Four color pixels including one color pixel in a region away from a center of each of the 2×2 CFA blocks form a first region in each of the plurality of CFA units, The first area includes the second pixel of the red CFA block, the second pixel of the blue CFA block, and the green pixels of each of the two green CFA blocks that are away from the center area; and When each of the plurality of CFA units is converted into a Bayer pattern through a rearrangement process, a value of an interpolation distance of each of four color pixels of the first region is 0.

8. The pixel array according to claim 1, wherein: performing a merging process on the color pixels of the pixel array, and Signals sensing color pixels of the same color in one of the plurality of CFA blocks are selectively added during the merging process.

9. The pixel array according to claim 1, wherein: performing a merging process on the color pixels of the pixel array, and Signals of color pixels sensing the same color in at least one row or column of one CFA block among the plurality of CFA blocks and a CFA block adjacent to the one CFA block are added during the merging process.

10. A pixel array of an image sensor, the pixel array comprising: a plurality of color filter array (CFA) units having a specific size, each of the plurality of CFA units including 2×2 CFA blocks in a width direction of the CFA unit and in a length direction of the CFA unit, each of the 2×2 CFA blocks including 4×4 color pixels, Each of the 2×2 CFA blocks includes a sub-block and an out-of-center area, the sub-block is located at the center area of ​​each of the 2×2 CFA blocks and includes 2×2 color pixels, and the out-of-center area is an area of ​​each of the 2×2 CFA blocks excluding the sub-block and includes 12 color pixels; Among them, the color of the CFA block is determined by the color pixel with the largest number. wherein the 2×2 CFA blocks include a red CFA block, a green CFA block, and a blue CFA block, and the sub-block includes 1 red pixel, 1 blue pixel, and 2 green pixels, The 12 color pixels in the area away from the center of the red CFA block include 1 blue pixel and 11 red pixels.

11. The pixel array according to claim 10, wherein: The one blue pixel in the out-of-center region of the red CFA block is in a corner of the red CFA block.

12. The pixel array according to claim 11, wherein: A blue pixel, a red pixel, a blue pixel, and a red pixel are sequentially arranged in a diagonal direction from the one corner of the red CFA block where the one blue pixel is arranged.

13. The pixel array according to claim 10, wherein: The 12 color pixels in the out-of-center region of the blue CFA block include 1 red pixel and 11 blue pixels.

14. The pixel array according to claim 13, wherein: Each of the plurality of CFA units includes one red CFA block, two green CFA blocks, and one blue CFA block, wherein the one blue pixel in the out-of-center region of the red CFA block, the one red pixel in the out-of-center region of the blue CFA block, and the two green pixels in the respective out-of-center regions of the two green CFA blocks form a first region, When each of the plurality of CFA units is converted into a Bayer pattern through a rearrangement process, a value of an interpolation distance of each color pixel in the first area is 0.

15. The pixel array according to claim 10, wherein: The 12 color pixels in the off-center region of the green CFA block include at most two color pixels other than the green pixel sensing at least one color selected from red and blue and the remaining green pixels.

16. An image sensor, comprising: A pixel array, comprising a plurality of color filter array (CFA) units, each CFA unit comprising a plurality of CFA blocks in a width direction of the CFA unit and a length direction of the CFA unit, each of the plurality of CFA blocks comprising a sub-block and an out-of-center region different from the sub-block, the sub-block being at a center region of each of the plurality of CFA blocks and comprising n×n color pixels, and the out-of-center region comprising other color pixels, wherein “n” is an integer of at least 2; as well as a read circuit configured to read pixel values ​​from color pixels of the pixel array, The n×n color pixels of the sub-block include color pixels for sensing a first color, a second color, and a third color, and the far-from-center area includes a first number of first pixels for sensing one of the first color, the second color, and the third color, and a second number of second pixels for sensing the other of the first color, the second color, and the third color, the first number of the first pixels being greater than the second number of the second pixels, wherein each color pixel on the boundary of the sub-block is adjacent to at least one first pixel of the far-from-center area in a horizontal direction or a vertical direction, and Wherein, at least two CFA blocks in one CFA unit have different first colors.

17. The image sensor according to claim 16, wherein: The first color, the second color and the third color include red, green and blue; Each of the plurality of CFA units includes 2×2 CFA blocks in the width direction and the length direction, wherein a color of the CFA block is determined by the first pixel, wherein the 2×2 CFA blocks include 2 green CFA blocks, 1 red CFA block, and 1 blue CFA block; and Each of the 2×2 CFA blocks includes 4×4 color pixels, and a sub-block of each of the 2×2 CFA blocks includes 2×2 color pixels.

18. The image sensor according to claim 17, wherein: The off-center region of the red CFA block includes 1 blue pixel and 11 red pixels in a corner of the red CFA block; and The off-center region of the blue CFA block includes 1 red pixel and 11 blue pixels in the corners of the blue CFA block.

19. The image sensor of claim 16, further comprising a merging processor configured to perform a merging process based on pixel values ​​from the pixel array, in, Signals sensing color pixels of the same color in one of the plurality of CFA blocks are selectively added during the merging process.

20. The image sensor of claim 16, further comprising a merging processor configured to perform a merging process based on pixel values ​​from the pixel array, in, Signals of color pixels sensing the same color in at least one row or column of one CFA block among the plurality of CFA blocks and a CFA block adjacent to the one CFA block are added during the merging process.

Images