Image sensing device and operating method thereof

Abstract

The present application discloses an image sensing device and an operating method thereof, wherein the image sensing device includes: an image sensor adapted to generate a first image signal corresponding to some pixels among a plurality of pixels in a merging mode; and an image processor adapted to generate a second image signal corresponding to other pixels by performing an interpolation operation based on the first image signal in the merging mode.

Description

Technical Field

[0001] Various embodiments of the present disclosure relate to semiconductor design technology, and more particularly, to an image sensing device and an operating method thereof. Background Art

[0002] An image sensing device is a device for capturing an image using the properties of a semiconductor that reacts to light. Image sensing devices can be roughly classified into charge coupled device (CCD) image sensing devices and complementary metal oxide semiconductor (CMOS) image sensing devices. Recently, CMOS image sensing devices are widely used because they can allow both analog and digital control circuits to be directly implemented on a single integrated circuit (IC). Summary of the invention

[0003] Various embodiments of the present disclosure are directed to an image sensing device including a circuit that is used in a normal mode and can be applied in a binning mode without changing a circuit design to achieve a high frame rate, and an operating method of the image sensing device.

[0004] According to an embodiment, an image sensing device may include: an image sensor adapted to generate a first image signal corresponding to some of a plurality of pixels in a merging mode; and an image processor adapted to generate a second image signal corresponding to other pixels by performing an interpolation operation based on the first image signal in a merging mode.

[0005] According to an embodiment, an operating method of an image sensing device may include the following steps: entering a merge mode; generating first image signals corresponding to some rows of a plurality of rows in which a plurality of pixels are arranged by performing an analog merge operation; and generating second image signals corresponding to other rows by performing an interpolation operation and a digital merge operation based on the first image signal.

[0006] According to an embodiment, an operating method of an image sensing device may include the following steps: selecting a set number of rows from a plurality of rows in an image matrix having m rows and n columns, wherein m and n are positive integers; selecting a set number of pixels from a plurality of pixels arranged in two or more rows adjacent to each other in the selected rows; performing an analog merging operation on pixel signals of the selected pixels to generate an analog image signal; converting the analog image signal into a digital image signal; and performing an interpolation operation on the digital image signal to generate a second image signal corresponding to pixels arranged in the remaining rows that were not selected in the step of selecting the set number of rows. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1is a block diagram illustrating an image sensing device according to an embodiment of the present disclosure.

[0008] Figure 2 is an example Figure 1 Block diagram of the image sensor illustrated in .

[0009] Figure 3 is an example Figure 2 A configuration diagram of a portion of the pixel array illustrated in FIG.

[0010] Figure 4 is an example used to describe Figure 1 A configuration diagram of a portion of a pixel array illustrating the operation of an image sensing device illustrated in FIG.

[0011] Figure 5 is an example used to describe Figure 1 A configuration diagram of a portion of a pixel array of another operation of the image sensing device illustrated in FIG.

[0012] Figure 6 is an example used to describe Figure 1 0 is a configuration diagram of a portion of a pixel array of yet another operation of the image sensing device illustrated in . DETAILED DESCRIPTION

[0013] Various embodiments are described below with reference to the accompanying drawings so that a person having ordinary skill in the art to which the present disclosure pertains can easily perform the technical concept of the present disclosure.

[0014] It should be understood that when an element is referred to as being "connected to" or "coupled to" another element, the element may be directly connected to or coupled to the other element, or electrically connected to or coupled to the other element with one or more elements interposed therebetween. In addition, it should also be understood that, unless otherwise stated, the terms "include", "comprise", "have" and "contain" when used in this specification do not exclude the presence of one or more other elements, but may further include one or more other elements. In the description throughout the specification, some components are described in the singular, but the present disclosure is not limited thereto, and it should be understood that the components may be formed in plurality.

[0015] Figure 1 is a block diagram illustrating an image sensing device according to an embodiment of the present disclosure.

[0016] Reference Figure 1 , the image sensing device may include an image sensor 100 and an image processor 200 .

[0017] The image sensor 100 may generate an image signal DPX corresponding to incident light in a normal mode and a binning mode.

[0018] For example, the image sensor 100 may operate all pixels arranged in a matrix row by row during one frame period in the normal mode and generate an image signal DPX corresponding to an original image. The image signal DPX generated in the normal mode may correspond to pixel signals (e.g., pixel values) of the plurality of pixels, respectively.

[0019] For example, the image sensor 100 may operate only some of the plurality of pixels for each group of selected rows during one frame period in the merging mode, and generate an image signal DPX corresponding to the processed image. The processed image may be smaller in size than the original image. The size may refer to the resolution of the processed image, and the processed image may be a high-definition (HD) image or a full HD image, etc. Each image signal DPX generated in the merging mode may be obtained by calculating the average value of the pixel signals read out from at least two of the above-mentioned pixels. Hereinafter, the image signal DPX generated from the image sensor 100 in the merging mode will be referred to as the first image signal DPX. The image sensor 100 may generate the first image signal DPX by performing an analog binning operation in the merging mode. The analog binning operation may include an operation of calculating the average value of at least two analog pixel signals.

[0020] The image processor 200 may process the image signal DPX during the one frame period in the normal mode and the merge mode.

[0021] For example, the image processor 200 may generate a normal image corresponding to the original image based on the image signal DPX in the normal mode.

[0022] In addition, the image processor 200 can generate a merged image corresponding to the processed image based on the first image signal DPX in the merge mode. Specifically, the image processor 200 can generate a second image signal corresponding to other pixels by performing an interpolation operation and a digital binning operation on the first image signal DPX in the merge mode. For example, the image processor 200 can generate a third image signal by performing a digital binning operation on the first image signal DPX, and generate a second image signal by performing an interpolation operation on the third image signal.

[0023] Figure 2 is an example Figure 1 00 is a block diagram of an image sensor 100 illustrated in FIG.

[0024] Reference Figure 2, the image sensor 100 may include a row controller 110 , a pixel array 120 , and a signal converter 130 .

[0025] The row controller 110 may generate row control signals CTRL for controlling all rows in which a plurality of pixels are arranged row by row in the normal mode. The row controller 110 may generate row control signals CTRL for controlling some rows in the plurality of rows for each group of rows in the binning mode.

[0026] The pixel array 120 may include a plurality of pixels arranged in a row direction and a column direction. The pixel array 120 may generate an analog pixel signal VPX under the control of the row controller 110. For example, the pixel array 120 may generate a pixel signal VPX row by row in a normal mode, and generate a pixel signal VPX for each group of rows in a merging mode. According to the analog merging operation, each pixel signal VPX generated in the merging mode may be a signal corresponding to an average value of at least two pixel signals.

[0027] The signal converter 130 may generate a digital image signal DPX based on the pixel signal VPX. For example, the signal converter 130 may generate an image signal DPX corresponding to the pixel signal VPX row by row in a normal mode, and generate a first image signal DPX corresponding to the pixel signal VPX for each group of rows in a merge mode. The signal converter 130 may include an analog-to-digital converter.

[0028] Figure 3 is an example Figure 2 0048. A diagram of a portion of pixel array 120 is illustrated in FIG.

[0029] Reference Figure 3 , the pixel array 120 may be arranged in a set pattern. For example, the pixel array 120 may be arranged in a Bayer pattern. The Bayer pattern may consist of a repeating unit of 2×2 pixels. In each unit, two pixels Gr and Gb, each having a green filter, may be arranged to face each other diagonally at the corners of the unit, and a pixel R having a red filter and a pixel B having a blue filter may be arranged to face each other diagonally at the remaining corners.

[0030] In the following, reference will be made to Figures 4 to 6 The operation of the image sensing device having the above configuration according to the present embodiment is described. In the present embodiment, only the operation according to the merging mode will be described.

[0031] First, the overall operation method of the present disclosure may include the following steps: entering a merge mode; generating a first image signal DPX corresponding to some rows of a plurality of rows in which a plurality of pixels are arranged by performing an analog merge operation; and generating a second image signal corresponding to other rows by performing an interpolation operation and a digital merge operation based on the first image signal DPX. Hereinafter, the analog merge operation, the digital merge operation, and the interpolation operation will be described according to examples.

[0032] Figure 4 is an example used to describe Figure 1 0 is a diagram of a portion of pixel array 120 illustrating the operation of an image sensing device.

[0033] Reference Figure 4 , the above-mentioned some rows may include the first to fourth rows and the ninth to twelfth rows, and the other rows may include the fifth to eighth rows and the thirteenth to sixteenth rows. It can be seen that the above-mentioned some rows and the other rows are regularly mixed. Hereinafter, for the convenience of description, a process of generating a second image signal corresponding to 2×2 pixels (i.e., R20, R21, R30, and R31) each having a red filter among the pixels included in the other rows will be representatively described.

[0034] The second image signal corresponding to the 2×2 pixels R20, R21, R30, and R31 may be generated by performing an interpolation operation. More specifically, the image value of the second image signal corresponding to the 2×2 pixels R20, R21, R30, and R31 may be calculated according to the following Equation 1.

[0035] [Formula 1]

[0036] (R20+R21+R30+R31) / 4=1 / 2*(R00+R01+R10+R11) / 4+1 / 2*(R40+R41+R50+R51) / 4

[0037] Herein, “(R20+R21+R30+R31) / 4” may be an image value of a second image signal corresponding to the 2×2 pixels R20, R21, R30, and R31, “(R00+R01+R10+R11) / 4” may be an image value of a first upper image signal corresponding to the 2×2 pixels R00, R01, R10, and R11 arranged in the first and third rows among the some rows, and “(R40+R41+R50+R51) / 4” may be an image value of a first lower image signal corresponding to the 2×2 pixels R40, R41, R50, and R51 arranged in the ninth and eleventh rows among the some rows.

[0038] The sum of image values ​​arranged in the same column among image values ​​used to generate the first upper image signal (i.e., each of “(R00+R10) / 2” and “(R01+R11) / 2”) can be obtained from the analog merging operation, and the total image value used to generate the first upper image signal (i.e., “{(R00+R01)+(R10+R11)} / 4”) can be obtained from the digital merging operation. The sum of image values ​​arranged in the same column among image values ​​used to generate the first lower image signal (i.e., each of “(R40+R50) / 2” and “(R41+R51) / 2”) can be obtained from the analog merging operation, and the total image value used to generate the first lower image signal (i.e., “{(R40+R41)+(R50+R51)} / 4”) can be obtained from the digital merging operation.

[0039] Figure 5 is an example used to describe Figure 1 0 is a diagram of a portion of pixel array 120 of another operation of the image sensing device illustrated in FIG.

[0040] Reference Figure 5 , the above-mentioned some rows may include the first row, the third row, the sixth row, the eighth row, the ninth row, the eleventh row, the fourteenth row and the sixteenth row, and the other rows may include the second row, the fourth row, the fifth row, the seventh row, the tenth row, the twelfth row, the thirteenth row and the fifteenth row. It can be seen that the some rows and the other rows are irregularly mixed. Hereinafter, for the convenience of description, a process of generating a second image signal corresponding to 2×2 pixels (i.e., Gr20, Gr21, Gr30 and Gr31) each having a green filter among the pixels arranged in the other rows will be representatively described.

[0041] The second image signal corresponding to the 2×2 pixels Gr20, Gr21, Gr30, and Gr31 may be generated by performing an interpolation operation. More specifically, the image value of the second image signal corresponding to the 2×2 pixels Gr20, Gr21, Gr30, and Gr31 may be calculated according to Equation 2 below.

[0042] [Formula 2]

[0043] (Gr20+Gr21+Gr30+Gr31) / 4=1 / 2*(Gr00+Gr01+Gr10+Gr11) / 4+1 / 2*(Gr40+Gr41+Gr50+Gr51) / 4

[0044] Herein, “(Gr20+Gr21+Gr30+Gr31) / 4” may be an image value of a second image signal corresponding to the 2×2 pixels Gr20, Gr21, Gr30, and Gr31, “(Gr00+Gr01+Gr10+Gr11) / 4” may be an image value of a first upper image signal corresponding to the 2×2 pixels Gr00, Gr01, Gr10, and Gr11 arranged in the first and third rows among the some rows, and “(Gr40+Gr41+Gr50+Gr51) / 4” may be an image value of a first lower image signal corresponding to the 2×2 pixels Gr40, Gr41, Gr50, and Gr51 arranged in the ninth and eleventh rows among the some rows.

[0045] The sum of image values ​​arranged in the same column among image values ​​used to generate the first upper image signal (i.e., each of “(Gr00+Gr10) / 2” and “(Gr01+Gr11) / 2”) can be obtained from the analog merging operation, and the total image value used to generate the first upper image signal (i.e., “{(Gr00+Gr01)+(Gr10+Gr11)} / 4”) can be obtained from the digital merging operation. The sum of image values ​​arranged in the same column among image values ​​used to generate the first lower image signal (i.e., each of “(Gr40+Gr50) / 2” and “(Gr41+Gr51) / 2”) can be obtained from the analog merging operation, and the total image value used to generate the first lower image signal (i.e., “{(Gr40+Gr41)+(Gr50+Gr51)} / 4”) can be obtained from the digital merging operation.

[0046] Figure 6 is an example used to describe Figure 1 0 is a diagram of a portion of pixel array 120 of yet another operation of an image sensing device illustrated in FIG.

[0047] Reference Figure 6 , the above-mentioned some rows may include the first row, the third row, the sixth row, the eighth row, the ninth row, the eleventh row, the fourteenth row and the sixteenth row, and the other rows may include the second row, the fourth row, the fifth row, the seventh row, the tenth row, the twelfth row, the thirteenth row and the fifteenth row. It can be seen that the some rows and the other rows are irregularly mixed. Hereinafter, for the convenience of description, a process of generating a second image signal corresponding to 2×2 pixels (i.e., Gr20, Gr21, Gr30 and Gr31) each having a green filter among the pixels arranged in the other rows will be representatively described.

[0048] The second image signal corresponding to the 2×2 pixels Gr20, Gr21, Gr30, and Gr31 may be generated by performing an interpolation operation. More specifically, an image value of the second image signal corresponding to the 2×2 pixels Gr20, Gr21, Gr30, and Gr31 may be calculated according to Equation 3 below.

[0049] [Formula 3]

[0050] (Gr20+Gr21+Gr30+Gr31) / 4=p(1 / 2*(Gr00+Gr01+Gr10+Gr11) / 4+1 / 2*(G r40+Gr41+Gr50+Gr51) / 4)+(1-p)*f(Gb20,Gb30,Gb21,Gb31,Gb22,Gb32)

[0051] Herein, "(Gr20+Gr21+Gr30+Gr31) / 4" may be an image value of a second image signal corresponding to 2×2 pixels Gr20, Gr21, Gr30, and Gr31, "(Gr00+Gr01+Gr10+Gr11) / 4" may be an image value of a first upper image signal corresponding to 2×2 pixels Gr00, Gr01, Gr10, and Gr11 arranged in the first and third rows among the some rows, and "(Gr40+Gr41+Gr50+Gr51) / 4" may be an image value of a first lower image signal corresponding to 2×2 pixels Gr40, Gr41, Gr50, and Gr51 arranged in the ninth and eleventh rows among the some rows. "p" as a weight may have a value between "0" and "1" (0≤p≤1). "f(Gb20, Gb30, Gb21, Gb31, Gb22, Gb32)" as a function for interpolation may be an image value of an image signal corresponding to the 3×2 pixels Gb20, Gb30, Gb21, Gb31, Gb22, and Gb32. The 3×2 pixels Gb20, Gb30, Gb21, Gb31, Gb22, and Gb32 may be arranged closest to the 2×2 pixels Gr20, Gr21, Gr30, and Gr31, and each of the 3×2 pixels Gb20, Gb30, Gb21, Gb31, Gb22, and Gb32 has a green filter. Since the function for interpolation is applied to generate the second image signal, Figure 5 In comparison, it may be more advantageous in terms of brightness.

[0052] The sum of image values ​​arranged in the same column among image values ​​used to generate the first upper image signal (i.e., each of “(Gr00+Gr10) / 2” and “(Gr01+Gr11) / 2”) can be obtained from the analog merging operation, and the total image value used to generate the first upper image signal (i.e., “{(Gr00+Gr01)+(Gr10+Gr11)} / 4”) can be obtained from the digital merging operation. The sum of image values ​​arranged in the same column among image values ​​used to generate the first lower image signal (i.e., each of “(Gr40+Gr50) / 2” and “(Gr41+Gr51) / 2”) can be obtained from the analog merging operation, and the total image value used to generate the first lower image signal (i.e., “{(Gr40+Gr41)+(Gr50+Gr51)} / 4”) can be obtained from the digital merging operation. “f(Gb20, Gb30, Gb21, Gb31, Gb22, Gb32)” can be calculated according to the following formula 4.

[0053] [Formula 4]

[0054] (Gb20+Gb30) / 2=A,

[0055] (Gb21+Gb31) / 2=B,

[0056] (Gb22+Gb32) / 2=C,

[0057] f((Gb20, Gb30, Gb21, Gb31, Gb22, Gb32) = 1 / 3*(A+B+C) or = 1 / 4*A+1 / 2*B+1 / 4*C

[0058] In this article, each of "(Gb20+Gb30) / 2", "(Gb21+Gb31) / 2" and "(Gb22+Gb32) / 2" can be obtained from the analog merging operation, and "1 / 3*(A+B+C)" or "1 / 4*A+1 / 2*B+1 / 4*C" can be obtained from the digital merging operation.

[0059] According to the embodiments of the present disclosure, since the readout operation and the signal conversion operation for other rows can be omitted in the binning mode, a high frame rate can be achieved.

[0060] According to an embodiment of the present disclosure, a high frame rate can be achieved while applying a circuit used in a normal mode in a merge mode without changing the design, thereby reducing design costs and obtaining an image with improved image quality at a high speed in the merge mode.

[0061] Although the present disclosure has been illustrated and described with respect to specific embodiments, the disclosed embodiments are provided for description and are not intended to be limiting. In addition, it should be noted that, as will be appreciated by those skilled in the art from this disclosure, the present disclosure may be implemented in various ways by substitutions, changes and modifications falling within the scope of the appended claims.

[0062] CROSS-REFERENCE TO RELATED APPLICATIONS

[0063] This application claims priority to Korean Patent Application No. 10-2020-0127124, filed on September 29, 2020, the disclosure of which is incorporated herein by reference in its entirety.

Claims

1. An image sensing device, comprising: an image sensor adapted to generate, in a binning mode, first image signals corresponding to some of a plurality of pixels and not generate second image signals corresponding to other pixels of the plurality of pixels; as well as an image processor adapted to generate the second image signal by performing an interpolation operation based on the first image signal in the merging mode, The image sensor generates the first image signal by performing an analog merging operation on pixels in different rows, wherein the some of the plurality of pixels are selected from a plurality of pixels arranged in two or more rows adjacent to each other among a set number of rows selected from the plurality of rows, The second image signal corresponds to pixels arranged in remaining rows that are not selected when the set number of rows are selected.

2. The image sensing device according to claim 1, wherein: The other pixels include pixels arranged in the remaining rows.

3. The image sensing device according to claim 2, wherein: Some of the plurality of rows include a first group of rows and a second group of rows, and The remaining rows include a third group of rows between the first group of rows and the second group of rows.

4. The image sensing device according to claim 1, wherein: The image processor generates the second image signal by performing a digital merging operation.

5. The image sensing device according to claim 4, wherein: The analog binning operation includes an operation of calculating an average value of analog pixel signals.

6. The image sensing device according to claim 4, wherein: The digital combining operation includes an operation of calculating an average value of the first image signal of a digital type.

7. A method for operating an image sensing device, the method comprising the following steps: Enter merge mode; generating first image signals corresponding to some of a plurality of rows in which a plurality of pixels are arranged by performing an analog binning operation on pixels in different rows; as well as generating a second image signal corresponding to other lines by performing an interpolation operation and a digital merging operation based on the first image signal, wherein the some of the plurality of rows include a first group of rows and a second group of rows, and The other rows include a third group of rows between the first group of rows and the second group of rows, wherein the pixels of the plurality of pixels are selected from a plurality of pixels arranged in two or more rows adjacent to each other among a set number of rows selected from a plurality of rows, The second image signal corresponds to pixels arranged in remaining rows that are not selected when the set number of rows are selected.

8. The operating method according to claim 7, wherein: The step of generating the second image signal comprises the following steps: generating a third image signal by performing the digital merging operation based on the first image signal; and The second image signal is generated by performing the interpolation operation based on the third image signal.

9. The operating method according to claim 7, wherein: The analog binning operation includes an operation of calculating an average value of analog pixel signals.

10. The operating method according to claim 7, wherein: The digital combining operation includes an operation of calculating an average value of the first image signal of a digital type.

11. The operating method according to claim 7, wherein: The first to third groups of rows are regularly mixed.

12. The operating method according to claim 7, wherein: The first to third groups of rows have the same color filter pattern.

13. The operating method according to claim 7, wherein: The some of the plurality of rows include a first group of rows, a second group of rows, and a third group of rows between the first group of rows and the second group of rows, and The other rows include a fourth group of rows that is different from the third group of rows and between the first group of rows and the second group of rows.

14. The operating method according to claim 13, wherein: The first to fourth groups of rows are irregularly mixed.

15. A method for performing a merging operation in an image sensing device, the method comprising the steps of: Selecting a set number of rows from a plurality of rows in an image matrix having m rows and n columns, wherein m and n are positive integers; selecting a set number of pixels from a plurality of pixels arranged in two or more rows adjacent to each other in the selected row; performing an analog merging operation on pixel signals of selected pixels in different rows to generate an analog image signal; converting the analog image signal into a digital image signal; and An interpolation operation is performed on the digital image signal to generate a second image signal corresponding to pixels arranged in remaining rows that are not selected in the step of selecting the set number of rows.

16. The method according to claim 15, wherein: The step of selecting the set number of pixels comprises: selecting a first group of pixels that are horizontally and vertically adjacent to each other from among a plurality of pixels arranged in two or more rows that are adjacent to each other in the selected row; as well as A second group of pixels that are horizontally and vertically adjacent to each other are selected from a plurality of pixels arranged in two or more rows that are adjacent to each other in the selected row.

17. The method according to claim 16, wherein: The analog binning operation is performed on the pixel signals of both the first group of pixels and the second group of pixels.

18. The method according to claim 16, wherein: The two or more rows having the first set of pixels are different from the two or more rows having the second set of pixels.

19. The method according to claim 18, wherein: The remaining rows are arranged between the two or more rows having the first group of pixels and the two or more rows having the second group of pixels.

Images