Image sensor, operating method of image sensor, and electronic device comprising image sensor

Abstract

The present invention relates to an image sensor. The image sensor of the present invention comprises a pixel array including pixels and reference pixels, an analog sensing circuit configured to detect signals from the pixels and reference pixels, and a digital logic circuit configured to receive signals from the analog sensing circuit and to compensate the signals of the pixels among the signals using the signals of the reference pixels among the signals. Each of the reference pixels is located between the pixels.

Description

Technology Field

[0001] The present invention relates to an electronic device, and more specifically, to an image sensor that improves image quality, a method of operating an image sensor, and an electronic device including an image sensor. Background Technology

[0002] Image sensors can capture incident light to generate image data. Image sensors are installed in various mobile devices, such as smartphones and smartpads, and are used to create multimedia content.

[0003] As image sensor manufacturing technology advances, image sensor resolution is increasing. This increase in resolution is raising the complexity of the circuits that constitute the image sensors. Furthermore, with the advancement of manufacturing technology, various functions are being added to image sensors, along with circuits capable of performing these diverse functions.

[0004] Increasing the complexity of the circuits constituting the image sensor, and adding functions and circuits to the image sensor, can improve the quality of the image data captured by the image sensor. However, increasing the complexity of the circuits constituting the image sensor, and adding functions and circuits to the image sensor, can introduce new forms of noise into the image data captured by the image sensor. The problem to be solved

[0005] The object of the present invention is to provide an image sensor that eliminates noise generated in image data by increasing the complexity of the image sensor and added circuits, a method of operating the image sensor, and an electronic device including the image sensor. means of solving the problem

[0006] An image sensor according to an embodiment of the present invention comprises a pixel array including pixels and reference pixels, an analog sensing circuit configured to detect signals from the pixels and reference pixels, and a digital logic circuit configured to receive signals from the analog sensing circuit and compensate the signals of the pixels among the signals using the signals of the reference pixels among the signals. Each of the reference pixels is located between the pixels.

[0007] A method of operation of an image sensor according to an embodiment of the present invention, comprising pixels and reference pixels, comprises the steps of detecting signals from the pixels and reference pixels, compensating the signals of the pixels based on the signals of the reference pixels among the detected signals, and outputting the compensated signals. The reference pixels are located between the pixels.

[0008] An electronic device according to an embodiment of the present invention includes a processor and an image sensor that generates image data under the control of the processor and provides the image data to the processor. The image sensor includes a pixel array comprising pixels and reference pixels, an analog sensing circuit configured to detect signals from the pixels and reference pixels, a digital logic circuit configured to receive signals from the analog sensing circuit and compensate the signals of the pixels among the signals using the signals of the reference pixels among the signals, and an interface circuit configured to output the compensated signals as image data to the processor. Each of the reference pixels includes a photodiode, a reflective material on the photodiode, a color filter on the reflective material, and a microlens on the color filter. Effects of the invention

[0009] According to the present invention, an image sensor can detect dark level leakages according to locations on a pixel array and compensate image data based on the dark level leakages. Accordingly, an image sensor that provides image data of improved quality, a method of operating the image sensor, and an electronic device including the image sensor are provided. Brief explanation of the drawing

[0010] FIG. 1 shows an image sensor according to an embodiment of the present invention. Figure 2 shows an example of an image sensor implemented. FIG. 3 shows an example of an operation method of an image sensor according to an embodiment of the present invention. Figure 4 shows an example of pixels and a reference pixel. Figure 5 shows an example where detection is performed on pixels and a reference pixel. FIG. 6 shows an example of a method in which an image sensor according to an embodiment of the present invention compensates for dark level leakage. Figure 7 shows a first example in which compensation is performed using pixels and reference pixels. Figure 8 shows a second example in which compensation is performed using pixels and reference pixels. Figure 9 shows a third example in which compensation is performed using pixels and reference pixels. Figure 10 shows a fourth example in which compensation is performed using pixels and reference pixels. Figure 11 shows a fifth example in which compensation is performed using pixels and reference pixels. Figure 12 shows another example of the operation method of an image sensor. Figure 13 shows another example of the operation method of an image sensor. FIG. 14 is a block diagram of an electronic device including a camera module group. FIG. 15 is a detailed block diagram of one of the camera modules of the camera module group of FIG. 14. Specific details for implementing the invention

[0011] In the following, embodiments of the present invention will be described clearly and in detail so that a person skilled in the art can easily practice the present invention. In the following, 'and / or' is interpreted to include any one of the items listed in association with the term, and a combination of some of the items listed in association.

[0012] FIG. 1 shows an image sensor (100) according to an embodiment of the present invention. Referring to FIG. 1, the image sensor (100) may include a pixel array (110), a scan driver circuit (120), an analog sensing circuit (130), a digital logic circuit (140), an interface circuit (150), and a digital control circuit (160).

[0013] A pixel array (110) may include pixels (PX) and reference pixels (PXr). The pixels (PX) and reference pixels (PXr) may be arranged in rows and columns. Each of the reference pixels (PXr) may be located between the pixels (PX) or surrounded by the pixels (PX).

[0014] Pixels (PX) can generate a voltage or current corresponding to the amount of incident light. Reference pixels (PXr) can generate a voltage or current regardless of the amount of incident light. For example, reference pixels (PXr) can generate a voltage or current corresponding to the leakage of the dark level.

[0015] Dark level leakage can correspond to current (or electrons) leaking from a pixel (PX) or a reference pixel (PXr) when there is no incident light. For example, the reference pixels (PXr) are exposed to incident light along with the pixels (PX), but unlike the pixels (PX), they can generate a voltage or current corresponding to dark level leakage regardless of the amount of incident light.

[0016] The scan driver circuit (120) can be connected to rows of pixels (PX) and reference pixels (PXr) through the first to nth scan lines (S1 to Sn). The scan driver circuit (120) can sequentially (or alternately) select the first to nth scan lines (S1 to Sn) in response to the control of the digital control circuit (160).

[0017] In response to one of the first to n scan lines (S1 to Sn) being selected, the scan driver circuit (120) can reset the pixels (PX) and / or reference pixels (PXr) connected to the selected row.

[0018] After the pixels (PX) connected to the selected row and / or reference pixels (PXr) are reset, the pixels (PX) can adjust the voltage or current at the reset level in response to the amount of incident light. After the pixels (PX) connected to the selected row and / or reference pixels (PXr) are reset, the reference pixels (PXr) can adjust the voltage or current at the reset level in response to the leakage of the dark level.

[0019] In response to a specific time elapsed after the pixels (PX) and / or reference pixels (PXr) connected to the selected row are reset (in response to the control of the digital control circuit (160)), the scan driver circuit (120) can electrically connect the pixels (PX) and / or reference pixels (PXr) of the selected row to the first to m data lines (D1 to Dm).

[0020] In FIG. 1, the scan driver circuit (120) is shown connected through one row of pixels (PX) and reference pixels (PXr) and one scan line. However, the scan driver circuit (120) may be connected through one row of pixels (PX) and reference pixels (PXr) and two or more lines.

[0021] For example, one of the two or more lines may be used to reset the pixels (PX) and / or reference pixels (PXr) of the corresponding row. At least one of the two or more lines may be used to electrically connect the pixels (PX) and / or reference pixels (PXr) of the corresponding row to the first to m data lines (D1 to Dm).

[0022] The analog sensing circuit (130) can be connected to columns of pixels (PX) and reference pixels (PXr) through the first to m data lines (D1 to Dm). In response to the control of the digital control circuit (160), at the timing when the pixels (PX) and / or reference pixels (PXr) of a selected row are connected to the first to m data lines (D1 to Dm), the analog sensing circuit can detect signals (voltages or currents) of the first to m data lines (D1 to Dm).

[0023] The analog detection circuit (130) can output the detected signals as first image data (ID1). The first image data (ID1) may be based on the signals of pixels (PX) of a selected row and / or reference pixels (PXr).

[0024] For example, the analog sensing circuit (130) may include a ramp signal generator that generates a ramp signal having a level that increases or decreases sequentially. The analog sensing circuit (130) may further include comparators that compare the signals of pixels (PX) of a selected row and / or reference pixels (PXr) with the ramp signal.

[0025] The analog detection circuit (130) may further include counters that convert the timings of the outputs of the comparators into digital values. The analog detection circuit (130) may output the digital values ​​output from the counters as first image data (ID1). For example, the first image data (ID1) output from the analog detection circuit (130) may be signals of pixels (PX) and / or reference pixels (PXr) that are detected by the analog detection circuit (130).

[0026] The digital logic circuit (140) can receive first image data (ID1) from the analog sensing circuit (130). In response to the control of the digital control circuit (160), the digital logic circuit (140) can receive signals detected from the first image data (ID1), i.e., pixels (PX) and / or reference pixels (PXr).

[0027] The digital logic circuit (140) can compensate the signals of the pixels (PX) using the signals of the reference pixels (PXr). For example, the digital logic circuit (140) can perform compensation by adding the digital value of the signals of the reference pixels (PXr) (or a value derived therefrom) to the digital value of the signals of the pixels (PX), or by subtracting the digital value of the signals of the reference pixels (PXr) (or a value derived therefrom) from the digital value of the signals of the pixels (PX).

[0028] The digital logic circuit (140) can output compensated signals as second image data (ID2) in response to the control of the digital control circuit (160). The interface circuit (150) can receive the second image data (ID2) from the digital logic circuit (140). In response to the control of the digital control circuit (160) or in response to the reception of the second image data (ID2), the interface circuit (150) can output the second image data (ID2) to an external device as third image data (ID3).

[0029] For example, the interface circuit (150) can output third image data (ID3) to an external device based on a C-PHY or D-PHY determined by MIPI (Mobile Industry Processor Interface).

[0030] The digital control circuit (160) can control the timings at which the scan driver circuit (120) selects the first to n scan lines (S1 to Sn), the timing at which the analog detection circuit (130) detects signals of the first to m data lines (D1 to Dm) and outputs the first image data (ID1), the timing at which the digital logic circuit (140) performs compensation and outputs the second image data (ID2), and / or the timing at which the interface circuit (150) outputs the third image data (ID3).

[0031] FIG. 2 shows an example in which an image sensor (200) is implemented. The image sensor (200) may be an example in which the image sensor (100) of FIG. 1 is implemented. Referring to FIG. 1 and FIG. 2, the image sensor (200) may be implemented as a combination of a first substrate (210) and a second substrate (220).

[0032] A pixel array (110) may be implemented on the first substrate (210). A scan driver circuit (120), an analog sensing circuit (130), a digital logic circuit (140), an interface circuit (150), and a digital control circuit (160) may be implemented on the second substrate (220). The first substrate (210) and the second substrate (220) may be electrically coupled through vertical wiring.

[0033] The pixels (PX) and reference pixels (PXr) of the pixel array (110) have the same structures and are used with the same frequency. Therefore, the heat generated by the pixels (PX) and reference pixels (PXr) on the first substrate (210) can be uniform regardless of the location on the first substrate (210), that is, the location on the pixel array (110).

[0034] The scan driver circuit (120), analog sensing circuit (130), digital logic circuit (140), interface circuit (150), and digital control circuit (160) implemented on the second substrate (220) may have different structures, implement different functions, and have different operating frequencies.

[0035] Accordingly, the heat generated by the scan driver circuit (120), analog detection circuit (130), digital logic circuit (140), interface circuit (150), and digital control circuit (160) on the second substrate (220) may vary depending on the location on the second substrate (220), that is, the location on the pixel array (110).

[0036] For example, the first heat source (221) and the second heat source (222) marked on the second substrate (220) may generate higher heat than other parts of the second substrate (220). The heat generated in the second substrate (220) may affect the first substrate (210). For example, the heat generated in the second substrate (220) may affect the amount of dark level leakage of the pixels (PX) and reference pixels (PXr) of the pixel array (110) of the first substrate (210).

[0037] If the heat generation of the second substrate (220) varies depending on the position on the pixel array (110), the dark level leakage of the pixels (PX) and reference pixels (PXr) of the pixel array (110) may vary depending on the position on the pixel array (110). Accordingly, signals detected by the pixel array (110) may be disturbed by changes in dark level leakage, and the quality of the image data (ID1, ID2, or ID3) may be degraded.

[0038] FIG. 3 shows an example of an operation method of an image sensor (100) according to an embodiment of the present invention. Referring to FIG. 1 and FIG. 3, in step S110, the image sensor (100) can detect dark signals from reference pixels (PXr). The dark signals are signals detected from reference pixels (PXr) of a selected row and can correspond to leakage of dark levels.

[0039] In step S120, the image sensor (100) can detect image signals from pixels (PX). The image signals are signals detected from pixels (PX) of a selected row and can correspond to the amount of incident light and the leakage of dark levels.

[0040] For example, step S110 may be performed in response to a scan line connected to a row containing reference pixels (PXr) being selected. Step S120 may be performed in response to each of the first to nth scan lines (S1 to Sn) being selected.

[0041] In step S130, the image sensor (100) can compensate for image signals detected from pixels (PX) based on dark signals detected from reference pixels (PXr). For example, pixels (PX) of the currently selected row can be compensated based on dark signals of reference pixels (PXr) of the currently selected row or a previously selected row.

[0042] Image signals of pixels (PX) of the currently selected row can be stored in a digital logic circuit (140). The stored image signals can be compensated based on the dark signals of reference pixels (PXr) of a row to be selected later. For example, each of the image signals of the pixels (PX) can be compensated based on the dark signal(s) of at least one (or two) adjacent reference pixels (PXr).

[0043] In step S140, the image sensor (100) can output the compensated image signals as image data, for example, third image data (ID3).

[0044] According to an embodiment of the present invention, differences in dark levels caused by differences in the position of heat generated in the second substrate (220) can be compensated. Accordingly, the quality of the third image data (ID3) generated by the image sensor (100) is improved.

[0045] FIG. 4 shows an example of pixels (PX) and a reference pixel (PXr). For example, an example of three adjacent pixels (PX) and a reference pixel (PXr) between pixels (PX) is shown in FIG. 4.

[0046] Referring to FIGS. 1 and 4, each of the pixels (PX) and the reference pixel (PXr) may include a micro lens (310) configured to concentrate incident light, a color filter (320) located below the micro lens (310), an isolation space (330) located below the color filter (320) that concentrates and transmits incident light, and a light sensing circuit (340) located below the isolation space (330) and including a photodiode and a transistor.

[0047] First reflective materials (301) may be located at the boundaries between pixels (PX) and reference pixels (PXr). The reflective materials (301) may support the concentration and transmission of incident light in the isolation space (330).

[0048] The light detection circuit (340) may include transistors connected to a corresponding scan line among the first to n scan lines (S1 to Sn) and a corresponding data line among the first to m data lines (D1 to Dm), and a photodiode that generates a current (or voltage) corresponding to the incident light according to the control of the transistors.

[0049] Unlike pixels (PX), the reference pixel (PXr) may further include a second reflective material (350) that separates the color filter (320) and the isolation space (330) between the color filter (320) and the isolation space (330). The second reflective material (350) may reflect incident light outward, thereby blocking the incident light from being transmitted to the isolation space (330).

[0050] FIG. 5 shows an example in which detection is performed on pixels (PX) and a reference pixel (PXr). The pixels (PX) and the reference pixel (PXr) in FIG. 5 are identical to the pixels (PX) and the reference pixel (PXr) described in FIG. 4, and thus redundant descriptions are omitted.

[0051] Referring to FIGS. 1 and FIGS. 5, in step S210, light incident on pixels (PX) can be transmitted to a photodiode of a light detection circuit (340). The photodiode can generate a current (or electron) in response to the incident light. The voltage (or current) of the light detection circuit (340) at a reset level can be changed by the current (or electron) generated by the photodiode.

[0052] In step S220, no light is incident on the reference pixel (PXr). Therefore, the photodiode in the reference pixel (PXr) does not generate a current (or electron) corresponding to the incident light. In the reference pixel (PXr), the voltage (or current) at the reset level of the light detection circuit (340) may not change due to the incident light.

[0053] In step S230, dark level leakage may occur in pixels (PX) and reference pixel (PXr). Due to the dark level leakage, currents (or electrons) may leak from pixels (PX) and reference pixel (PXr).

[0054] The signal detected from the pixels (PX) may indicate a value where the voltage (or current) at the reset level has changed due to incident light and dark level leakage. The signal detected from the reference pixel (PXr) may indicate a value where the voltage (or current) at the reset level has changed due to dark level leakage. Accordingly, based on the amount of dark level leakage measured from the reference pixel (PXr), the dark level leakage of the pixels (PX) can be compensated.

[0055] As described with reference to FIG. 2, the amount of dark level leakage may vary depending on the location on the pixel array (110). By distributing reference pixels (PXr) on the pixel array (110), the amount of dark level leakage according to the location on the pixel array (110) can be measured. Additionally, the dark level leakage of pixels (PX) according to the location on the pixel array (110) can be compensated.

[0056] FIG. 6 shows an example of a method in which an image sensor (100) according to an embodiment of the present invention compensates for dark level leakage. Referring to FIG. 1 and FIG. 6, a digital logic circuit (140) of the image sensor (100) can collect dark signals from reference pixels (PXr).

[0057] In step S320, the digital logic circuit (140) of the image sensor (100) can calculate dark shading from the dark signals. For example, dark shading may represent a distribution of dark levels according to location on the image data of image signals generated by the pixels (PX) of the pixel array (110).

[0058] In step S330, the digital logic circuit (140) can correct image signals based on dark shading. For example, the digital logic circuit (140) can compensate the values ​​(e.g., gray levels) of image signals of pixels (PX) adjacent to a specific reference pixel (PXr) based on each of the dark signals of a specific reference pixel (PXr).

[0059] The digital logic circuit (140) can compensate the image signals of pixels (PX) based on various values ​​such as the minimum, maximum, median, and average values ​​of dark levels corresponding to dark shading. For example, the digital logic circuit (140) can compensate the DC level and / or gain of the values ​​(e.g., gray levels) of the image signals of pixels (PX).

[0060] FIG. 7 shows a first example in which compensation is performed using pixels (PX) and reference pixels (PXr). Referring to FIG. 1 and FIG. 7, compensation can be performed based on a compensation unit (CU). The pixels (PX) and reference pixels (PXr) of the pixel array (110) can be divided into compensation units (CU).

[0061] Each of the compensation units (CU) may include a reference pixel (PXr), two or more rows of pixels (PX), and two or more columns of pixels (PX). The reference pixel (PXr) may be located at the center of the pixels (PX). The image signals of the pixels (PX) belonging to the compensation unit (CU) may be compensated based on the dark signal of the reference pixel (PXr) belonging to the same compensation unit (CU).

[0062] In each of the compensation units (CU), image signals of pixels (PX) of rows selected before the row of reference pixel (PXr) can be stored in a digital logic circuit (140). After a dark signal is detected from the reference pixel (PXr), compensation of the stored image signals can be performed.

[0063] In each of the compensation units (CU), the image signals of the pixels (PX) in rows selected after the row of the reference pixel (PXr) or the row of the reference pixel (PXr) can be compensated using the dark signal detected from the reference pixel (PXr).

[0064] FIG. 8 shows a second example in which compensation is performed using pixels (PX) and reference pixels (PXr). Referring to FIG. 1 and FIG. 8, compensation can be performed based on a compensation unit (CU). The pixels (PX) and reference pixels (PXr) of the pixel array (110) can be divided into compensation units (CU).

[0065] Each of the compensation units (CU) may include a reference pixel (PXr), two or more rows of pixels (PX), and two or more columns of pixels (PX). The reference pixel (PXr) may be located in the first row selected among the rows belonging to each of the compensation units (CU). The image signals of the pixels (PX) belonging to the compensation unit (CU) may be compensated based on the dark signal of the reference pixel (PXr) belonging to the same compensation unit (CU).

[0066] In each of the compensation units (CU), the image signals of the pixels (PX) in rows selected after the row of the reference pixel (PXr) or the row of the reference pixel (PXr) can be compensated using the dark signal detected from the reference pixel (PXr).

[0067] FIG. 9 shows a third example in which compensation is performed using pixels (PX) and reference pixels (PXr). Referring to FIG. 1 and FIG. 9, compensation can be performed based on a first compensation unit (CU1) and a second compensation unit (CU2). The pixels (PX) and reference pixels (PXr) of the pixel array (110) can be divided into first compensation units (CU1) and second compensation units (CU2).

[0068] For example, pixels (PX) and reference pixels (PXr) located at the edges of the pixel array (110) can form first compensation units (CU1). Pixels (PX) and reference pixels (PXr) located at the center of the pixel array (110) can form second compensation units (CU2).

[0069] Each of the first compensation units (CU1) and the second compensation units may include one reference pixel (PXr), two or more rows of pixels (PX), and two or more columns of pixels (PX). The reference pixel (PXr) may be located in the center of the pixels (PX) in each of the first compensation units (CU1) or the second compensation units (CU2) (see FIG. 7), or in the first row selected among the rows belonging to each of the first compensation units (CU1) or the second compensation units (CU2) (see FIG. 8). The image signals of the pixels (PX) belonging to the compensation unit (CU) may be compensated based on the dark signal of the reference pixel (PXr) belonging to the same compensation unit (CU).

[0070] For example, the number of pixels (PX) belonging to the first compensation unit (CU1) may be less than the number of pixels (PX) belonging to the second compensation unit (CU2). That is, depending on the position on the pixel array (110) of the compensation unit (CU1 or CU2), the number of pixels (PX) included in the compensation unit (CU1 or CU2) may vary.

[0071] FIG. 10 shows a fourth example in which compensation is performed using pixels (PX) and reference pixels (PXr). Referring to FIG. 1 and FIG. 10, compensation can be performed based on a first compensation unit (CU1) and a second compensation unit (CU2). The pixels (PX) and reference pixels (PXr) of the pixel array (110) can be divided into first compensation units (CU1) and second compensation units (CU2).

[0072] For example, pixels (PX) and reference pixels (PXr) located at the edges of the pixel array (110) may form first compensation units (CU1) or second compensation units (CU2). Pixels (PX) and reference pixels (PXr) located at the center of the pixel array (110) may form first compensation units (CU1).

[0073] Each of the first compensation units (CU1) and the second compensation units may include one reference pixel (PXr), two or more rows of pixels (PX), and two or more columns of pixels (PX). The reference pixel (PXr) may be located in the center of the pixels (PX) in each of the first compensation units (CU1) or the second compensation units (CU2) (see FIG. 7), or in the first row selected among the rows belonging to each of the first compensation units (CU1) or the second compensation units (CU2) (see FIG. 8). The image signals of the pixels (PX) belonging to the compensation unit (CU) may be compensated based on the dark signal of the reference pixel (PXr) belonging to the same compensation unit (CU).

[0074] For example, the number of pixels (PX) belonging to the first compensation unit (CU1) may be less than the number of pixels (PX) belonging to the second compensation unit (CU2). That is, depending on the position on the pixel array (110) of the compensation unit (CU1 or CU2), the number of pixels (PX) included in the compensation unit (CU1 or CU2) may vary.

[0075] FIG. 11 shows a fifth example in which compensation is performed using pixels (PX) and reference pixels (PXr). Referring to FIG. 1 and FIG. 11, compensation can be performed based on interpolation.

[0076] As indicated by the interpolation line (IP) in FIG. 11, compensation can be performed based on the dark signals of the two closest reference pixels (PXr) in the row, column, and / or diagonal directions. For example, the image signals of the pixels (PX) between the two closest reference pixels (PXr) in the row, column, and / or diagonal directions can be compensated based on the result of the interpolation of the two closest reference pixels (PXr) in the row, column, and / or diagonal directions.

[0077] For example, interpolation may involve performing calculations by weighting the values ​​of the dark signals of two adjacent reference pixels (PXr). For example, interpolation may be linear interpolation or Gaussian interpolation.

[0078] In the example of linear interpolation, the weight of the dark signal of one of two adjacent reference pixels (PXr) can decrease linearly as the distance from one of the reference pixels (PXr) increases. In the example of Gaussian interpolation, the weight of the dark signal of one of two adjacent reference pixels (PXr) can decrease based on a Gaussian as the distance from one of the reference pixels (PXr) increases.

[0079] For example, when a specific pixel (PX) is located in the same row as two reference pixels (PXr), the image signal of the specific pixel (PX) can be compensated based on the interpolation of the dark signals of the two reference pixels (PXr) belonging to the same row.

[0080] When a specific pixel (PX) is located in the same column as two reference pixels (PXr), the image signal of the specific pixel (PX) can be compensated based on the interpolation of the dark signals of the two reference pixels (PXr) belonging to the same column.

[0081] When a specific pixel (PX) is not located in the same column or row as two reference pixels (PXr), the image signal of the specific pixel (PX) can be compensated based on the interpolation of the dark signals of the four closest reference pixels (PXr) in the row direction, column direction, and / or diagonal direction.

[0082] FIG. 12 shows another example of the method of operation of the image sensor (100). Referring to FIG. 1 and FIG. 12, in step S410, the image sensor (100) can detect the operation mode. For example, the image sensor (100) can detect the operation mode by checking the value stored in a register among the registers of the interface circuit (150) that can be set by an external device.

[0083] For example, the first value stored in the register may indicate an active state of dark level leakage compensation. The second value stored in the register may indicate an inactive state of dark level leakage compensation.

[0084] In step S420, the image sensor (100) can determine whether the compensation for dark level leakage is in an inactive state. When the compensation for dark level leakage is in an inactive state, in step S430, the image sensor (100) can disable the compensation for dark level leakage. For example, the analog detection circuit (130) outputs first image data (ID1) containing dark signals of reference pixels (PXr), but the digital logic circuit (140) can ignore the dark signals of reference pixels (PXr).

[0085] When compensation for dark level leakage is active, in step S440, the image sensor (100) can enable compensation for dark level leakage. The image sensor can perform compensation for dark level leakage as described with reference to FIGS. 1 through 11.

[0086] FIG. 13 shows another example of the operation method of the image sensor (100). Referring to FIG. 1 and FIG. 13, in step S510, the digital logic circuit (140) of the image sensor (100) can collect image signals of pixels (PX) that are closest to each of the reference pixels (PXr) (e.g., in the row direction, column direction, and / or diagonal direction).

[0087] In step S520, the digital logic circuit (140) can generate each image signal of the reference pixels (PXr) based on the collected image signals. For example, the digital logic circuit (140) can generate each image signal of the reference pixels (PXr) by performing interpolation on the image signals of the pixels (PX) surrounding each of the reference pixels (PXr).

[0088] FIG. 14 is a block diagram of an electronic device (1000) including a camera module group (1100). FIG. 15 is a detailed block diagram of one camera module (e.g., 1100b) among the camera modules (1100a, 1100b, 1100c) of the camera module group (1100) of FIG. 14. For example, the structures of the camera modules (1100a, 1100b, 1100c) may be identical or at least similar.

[0089] Referring to FIG. 14, the electronic device (1000) may include a camera module group (1100), an application processor (1200), a PMIC (1300), and an external memory (1400).

[0090] The camera module group (1100) may include a plurality of camera modules (1100a, 1100b, 1100c). Although an embodiment in which three camera modules (1100a, 1100b, 1100c) are arranged is illustrated in the drawings, the embodiments are not limited thereto. In some embodiments, the camera module group (1100) may be modified to include only two camera modules. Also, in some embodiments, the camera module group (1100) may be modified to include n camera modules (n is a natural number greater than or equal to 4).

[0091] Hereinafter, with reference to FIG. 15, the detailed configuration of the camera module (1100b) will be described in more detail, but the following description may be applied equally to other camera modules (1100a, 1100c) according to the embodiment.

[0092] Referring to FIG. 15, the camera module (1100b) may include a prism (1105), an optical path folding element (OPFE, hereinafter referred to as "OPFE") (1110), an actuator (1130), an image sensing device (1140), and a storage unit (1150).

[0093] The prism (1105) can modify the path of light (L) incident from the outside by including a reflective surface (1107) of a light-reflecting material.

[0094] In some embodiments, the prism (1105) can change the path of light (L) incident in a first direction (X) to a second direction (Y) perpendicular to the first direction (X). Additionally, the prism (1105) can change the path of light (L) incident in the first direction (X) to a second direction (Y) perpendicular to the first direction (X) by rotating the reflective surface (1107) of the light-reflecting material in direction A around the central axis (1106) or by rotating the central axis (1106) in direction B. At this time, the OPFE (1110) can also move to a third direction (Z) perpendicular to the first direction (X) and the second direction (Y).

[0095] In some embodiments, as illustrated, the maximum rotation angle of the prism (1105) in the A direction may be 15 degrees or less in the plus (+) A direction and greater than 15 degrees in the minus (-) A direction, but the embodiments are not limited thereto.

[0096] In some embodiments, the prism (1105) can move in the plus (+) or minus (-) B direction by about 20 degrees, or between 10 and 20 degrees, or between 15 and 20 degrees, where the angle of movement can be moved by the same angle in the plus (+) or minus (-) B direction, or by a nearly similar angle within a range of about 1 degree.

[0097] In some embodiments, the prism (1105) can move the reflective surface (1107) of the light-reflecting material in a third direction (e.g., Z direction) parallel to the extension direction of the central axis (1106).

[0098] OPFE (1110) may include, for example, groups of m (where m is a natural number) optical lenses. The m lenses can be moved in a second direction (Y) to change the optical zoom ratio of the camera module (1100b). For example, when the basic optical zoom ratio of the camera module (1100b) is Z, moving the m optical lenses included in the OPFE (1110) may change the optical zoom ratio of the camera module (1100b) to 3Z or 5Z or an optical zoom ratio of 5Z or more.

[0099] The actuator (1130) can move the OPFE (1110) or the optical lens (hereinafter referred to as the optical lens) to a specific position. For example, the actuator (1130) can adjust the position of the optical lens so that the image sensor (1142) is positioned at the focal length of the optical lens for accurate sensing.

[0100] The image sensing device (1140) may include an image sensor (1142), control logic (1144), and memory (1146). The image sensor (1142) can sense an image of a sensing target using light (L) provided through an optical lens. The control logic (1144) can control the overall operation of the camera module (1100b). For example, the control logic (1144) can control the operation of the camera module (1100b) according to a control signal provided through a control signal line (CSLb).

[0101] The memory (1146) can store information necessary for the operation of the camera module (1100b), such as calibration data (1147). The calibration data (1147) may include information necessary for the camera module (1100b) to generate image data using light (L) provided from the outside. The calibration data (1147) may include, for example, information regarding the degree of rotation described above, information regarding the focal length, information regarding the optical axis, etc. If the camera module (1100b) is implemented in the form of a multi-state camera in which the focal length changes according to the position of the optical lens, the calibration data (1147) may include focal length values ​​for each position (or state) of the optical lens and information related to auto-focusing.

[0102] The storage unit (1150) can store image data sensed through the image sensor (1142). The storage unit (1150) may be placed outside the image sensing device (1140) and may be implemented in a stacked form with the sensor chip constituting the image sensing device (1140). In some embodiments, the storage unit (1150) may be implemented as an EEPROM (Electrically Erasable Programmable Read-Only Memory), but the embodiments are not limited thereto.

[0103] For example, an image sensor (1142) or an image sensing device (1140) may correspond to the image sensor (100) of FIG. 1.

[0104] Referring to FIG. 14 and FIG. 15 together, in some embodiments, each of the plurality of camera modules (1100a, 1100b, 1100c) may include an actuator (1130). Accordingly, each of the plurality of camera modules (1100a, 1100b, 1100c) may include identical or different calibration data (1147) according to the operation of the actuator (1130) included therein.

[0105] In some embodiments, one of the plurality of camera modules (1100a, 1100b, 1100c) (e.g., 1100b) is a camera module in the form of a folded lens including the previously described prism (1105) and OPFE (1110), and the remaining camera modules (e.g., 1100a, 1100c) may be camera modules in the form of a vertical camera module that do not include the prism (1105) and OPFE (1110), but the embodiments are not limited thereto.

[0106] In some embodiments, one of the plurality of camera modules (1100a, 1100b, 1100c) (e.g., 1100c) may be a vertical depth camera that extracts depth information using, for example, IR (Infrared Ray). In this case, the application processor (1200) may generate a 3D depth image by merging image data provided from this depth camera with image data provided from another camera module (e.g., 1100a or 1100b).

[0107] In some embodiments, at least two of the plurality of camera modules (1100a, 1100b, 1100c) may have different field of view angles. In this case, for example, the optical lenses of at least two of the plurality of camera modules (1100a, 1100b, 1100c) may be different from each other, but are not limited thereto.

[0108] Additionally, in some embodiments, the viewing angles of each of the plurality of camera modules (1100a, 1100b, 1100c) may differ from each other. In this case, the optical lenses included in each of the plurality of camera modules (1100a, 1100b, 1100c) may also differ from each other, but are not limited thereto.

[0109] In some embodiments, each of the plurality of camera modules (1100a, 1100b, 1100c) may be physically separated from one another. That is, instead of the plurality of camera modules (1100a, 1100b, 1100c) dividing and using the sensing area of ​​a single image sensor (1142), an independent image sensor (1142) may be placed inside each of the plurality of camera modules (1100a, 1100b, 1100c).

[0110] Referring again to FIG. 14, the application processor (1200) may include an image processing device (1210), a memory controller (1220), and an internal memory (1230). The application processor (1200) may be implemented separately from a plurality of camera modules (1100a, 1100b, 1100c). For example, the application processor (1200) and the plurality of camera modules (1100a, 1100b, 1100c) may be implemented separately from each other as separate semiconductor chips.

[0111] The image processing device (1210) may include a plurality of sub-image processors (1212a, 1212b, 1212c), an image generator (1214), and a camera module controller (1216).

[0112] The image processing device (1210) may include a plurality of sub-image processors (1212a, 1212b, 1212c) corresponding to the number of camera modules (1100a, 1100b, 1100c).

[0113] Image data generated from each camera module (1100a, 1100b, 1100c) can be provided to corresponding sub-image processors (1212a, 1212b, 1212c) via separate image signal lines (ISLa, ISLb, ISLc). For example, image data generated from camera module (1100a) can be provided to sub-image processor (1212a) via image signal line (ISLa), image data generated from camera module (1100b) can be provided to sub-image processor (1212b) via image signal line (ISLb), and image data generated from camera module (1100c) can be provided to sub-image processor (1212c) via image signal line (ISLc). Such image data transmission can be performed, for example, using a Camera Serial Interface (CSI) based on MIPI (Mobile Industry Processor Interface), but the embodiments are not limited thereto.

[0114] Meanwhile, in some embodiments, a single sub-image processor may be arranged to correspond to a plurality of camera modules. For example, the sub-image processor (1212a) and the sub-image processor (1212c) may not be implemented separately as illustrated, but rather integrated into a single sub-image processor, and image data provided from the camera module (1100a) and the camera module (1100c) may be selected through a selection element (e.g., a multiplexer) and then provided to the integrated sub-image processor.

[0115] Image data provided to each sub-image processor (1212a, 1212b, 1212c) may be provided to an image generator (1214). The image generator (1214) may generate an output image using image data provided from each sub-image processor (1212a, 1212b, 1212c) according to image generation information or a mode signal.

[0116] Specifically, the image generator (1214) can generate an output image by merging at least some of the image data generated from camera modules (1100a, 1100b, 1100c) having different viewing angles according to image generation information or a mode signal. Additionally, the image generator (1214) can generate an output image by selecting any one of the image data generated from camera modules (1100a, 1100b, 1100c) having different viewing angles according to image generation information or a mode signal.

[0117] In some embodiments, the image generation information may include a zoom signal (or zoom factor). Additionally, in some embodiments, the mode signal may be a signal based on a mode selected by a user, for example.

[0118] When the image generation information is a zoom signal (zoom factor) and each camera module (1100a, 1100b, 1100c) has a different viewing angle (angle of view), the image generator (1214) can perform different operations depending on the type of zoom signal. For example, if the zoom signal is a first signal, the image data output from the camera module (1100a) and the image data output from the camera module (1100c) can be merged, and then an output image can be generated using the merged image signal and the image data output from the camera module (1100b) that was not used for merging. If the zoom signal is a second signal different from the first signal, the image generator (1214) can generate an output image by selecting one of the image data output from each camera module (1100a, 1100b, 1100c) without performing such image data merging. However, the embodiments are not limited thereto, and the method of processing image data can be modified as needed.

[0119] In some embodiments, the image generator (1214) receives multiple image data with different exposure times from at least one of a plurality of sub-image processors (1212a, 1212b, 1212c) and performs HDR (high dynamic range) processing on the multiple image data to generate merged image data with increased dynamic range.

[0120] The camera module controller (1216) can provide control signals to each camera module (1100a, 1100b, 1100c). The control signals generated from the camera module controller (1216) can be provided to the corresponding camera modules (1100a, 1100b, 1100c) through separate control signal lines (CSLa, CSLb, CSLc).

[0121] One of the plurality of camera modules (1100a, 1100b, 1100c) may be designated as a master camera (e.g., 1100b) according to image generation information including a zoom signal or a mode signal, and the remaining camera modules (e.g., 1100a, 1100c) may be designated as slave cameras. This information may be included in a control signal and provided to the corresponding camera modules (1100a, 1100b, 1100c) through separate control signal lines (CSLa, CSLb, CSLc).

[0122] The camera module operating as a master and slave may be changed according to the zoom factor or operation mode signal. For example, if the field of view of the camera module (1100a) is wider than the field of view of the camera module (1100b) and the zoom factor indicates a low zoom magnification, the camera module (1100b) may operate as a master and the camera module (1100a) may operate as a slave. Conversely, if the zoom factor indicates a high zoom magnification, the camera module (1100a) may operate as a master and the camera module (1100b) may operate as a slave.

[0123] In some embodiments, the control signal provided from the camera module controller (1216) to each camera module (1100a, 1100b, 1100c) may include a sync enable signal. For example, if the camera module (1100b) is a master camera and the camera modules (1100a, 1100c) are slave cameras, the camera module controller (1216) may transmit a sync enable signal to the camera module (1100b). The camera module (1100b) that receives this sync enable signal may generate a sync signal based on the received sync enable signal and provide the generated sync signal to the camera modules (1100a, 1100c) through a sync signal line (SSL). The camera module (1100b) and camera modules (1100a, 1100c) can be synchronized with this sync signal to transmit image data to the application processor (1200).

[0124] In some embodiments, a control signal provided from a camera module controller (1216) to a plurality of camera modules (1100a, 1100b, 1100c) may include mode information according to a mode signal. Based on this mode information, the plurality of camera modules (1100a, 1100b, 1100c) may operate in a first operation mode and a second operation mode with respect to the sensing speed.

[0125] A plurality of camera modules (1100a, 1100b, 1100c) can, in a first operating mode, generate an image signal at a first speed (e.g., generate an image signal at a first frame rate) and encode it at a second speed higher than the first speed (e.g., encode an image signal at a second frame rate higher than the first frame rate), and transmit the encoded image signal to an application processor (1200). At this time, the second speed may be 30 times or less of the first speed.

[0126] The application processor (1200) stores the received image signal, that is, the encoded image signal, in an internal memory (1230) or an external memory (1400) of the application processor (1200), and subsequently reads the encoded image signal from the internal memory (1230) or the external memory (1400) to decode it, and can display image data generated based on the decoded image signal. For example, a corresponding sub-processor among a plurality of sub-processors (1212a, 1212b, 1212c) of the image processing device (1210) can perform decoding, and can also perform image processing on the decoded image signal.

[0127] A plurality of camera modules (1100a, 1100b, 1100c) can generate an image signal at a third speed lower than a first speed in a second operation mode (e.g., generate an image signal at a third frame rate lower than a first frame rate) and transmit the image signal to an application processor (1200). The image signal provided to the application processor (1200) may be an unencoded signal. The application processor (1200) may perform image processing on the received image signal or store the image signal in an internal memory (1230) or an external memory (1400).

[0128] The PMIC (1300) can supply power, such as power voltage, to each of the plurality of camera modules (1100a, 1100b, 1100c). For example, the PMIC (1300) can supply first power to the camera module (1100a) through a power signal line (PSLa), supply second power to the camera module (1100b) through a power signal line (PSLb), and supply third power to the camera module (1100c) through a power signal line (PSLc), under the control of the application processor (1200).

[0129] The PMIC (1300) can generate power corresponding to each of the plurality of camera modules (1100a, 1100b, 1100c) and adjust the power level in response to a power control signal (PCON) from the application processor (1200). The power control signal (PCON) may include power adjustment signals for each operating mode of the plurality of camera modules (1100a, 1100b, 1100c). For example, the operating mode may include a low power mode, and in this case, the power control signal (PCON) may include information about the camera module operating in the low power mode and the power level being set. The power levels provided to each of the plurality of camera modules (1100a, 1100b, 1100c) may be the same or different from each other. Additionally, the power level may be changed dynamically.

[0130] In the embodiments described above, components according to the technical concept of the present invention have been described using terms such as first, second, third, etc. However, terms such as first, second, third, etc. are used to distinguish the components from one another and do not limit the present invention. For example, terms such as first, second, third, etc. do not imply a sequential order or any numerical meaning.

[0131] In the embodiments described above, components according to embodiments of the present invention are referenced using blocks. The blocks may be implemented in various forms such as ICs (Integrated Circuits), ASICs (Application Specific ICs), FPGAs (Field Programmable Gate Arrays), CPLDs (Complex Programmable Logic Devices), software such as firmware or applications running on hardware devices, or a combination of hardware devices and software. Additionally, the blocks may include circuits composed of semiconductor devices within an IC or circuits registered as IPs (Intellectual Property).

[0132] The above description describes specific embodiments for implementing the present invention. The present invention will include not only the embodiments described above, but also embodiments that can be simply modified or easily modified. Furthermore, the present invention will include technologies that can be easily modified and implemented using the embodiments. Accordingly, the scope of the present invention should not be limited to the embodiments described above, but should be defined by the claims set forth below as well as equivalents to the claims of this invention. Explanation of the symbols

[0133] 100: Image sensor 110: Pixel array 120: Scan driver circuit 130: Analog sensing circuit 140: Digital Logic Circuits 150: Interface circuit 160: Digital control circuit 200: Image sensor 210: First substrate 220: Second substrate

Claims

Claim 1 An image sensor comprising: a pixel array including pixels and reference pixels; an analog sensing circuit configured to detect signals from the pixels and the reference pixels; and a digital logic circuit configured to receive the signals from the analog sensing circuit and to compensate the signals of the pixels among the signals using the signals of the reference pixels among the signals, wherein each of the reference pixels is located between the pixels, the pixel array includes compensation units, and each of the compensation units includes one of the reference pixels and two or more of the pixels, and the number of the two or more pixels included in each of the compensation units varies according to the position of each of the compensation units on the pixel array. Claim 2 An image sensor according to claim 1, wherein each of the reference pixels has the same structure as each of the pixels, and each of the reference pixels further includes a reflective material that reflects light incident from the outside to the outside. Claim 3 An image sensor according to claim 1, wherein the digital logic circuit is configured to calculate the leakage of the reference pixels based on the signals of the reference pixels and to compensate for the leakage of the pixels based on the leakage of the reference pixels. Claim 4 In paragraph 3, the leakage of the reference pixels is an image sensor that is a dark level leakage. Claim 5 In claim 1, the digital logic circuit is an image sensor that compensates the signals of two or more pixels based on the signal of one reference pixel in each of the compensation units. Claim 6 In paragraph 5, the image sensor wherein, in each of the compensation units, the two or more pixels are arranged in two or more rows and two or more columns, and the one reference pixel is located in the center of the two or more pixels. Claim 7 In paragraph 5, in each of the above compensation units, the two or more pixels are arranged in two or more rows and two or more columns, and the one reference pixel is located in the first row of the two or more rows in an image sensor. Claim 8 delete Claim 9 A method of operation of an image sensor comprising a pixel array including pixels and reference pixels, comprising: a step of detecting signals from the pixels and the reference pixels; a step of compensating the signals of the pixels based on the signals of the reference pixels among the detected signals; and a step of outputting the compensated signals, wherein the reference pixels are located between the pixels, the pixel array includes compensation units, and each of the compensation units includes one reference pixel among the reference pixels and two or more pixels among the pixels, and the number of the two or more pixels included in each of the compensation units varies according to the position of each of the compensation units on the pixel array. Claim 10 An electronic device comprising: a processor; and an image sensor that generates image data under the control of the processor and provides the image data to the processor, wherein the image sensor comprises: a pixel array including pixels and reference pixels; an analog sensing circuit configured to detect signals from the pixels and the reference pixels; a digital logic circuit configured to receive the signals from the analog sensing circuit and compensate the signals of the pixels among the signals using the signals of the reference pixels among the signals; and an interface circuit configured to output the compensated signals as image data to the processor, wherein each of the reference pixels comprises: a photodiode; a reflective material on the photodiode; a color filter on the reflective material; and a microlens on the color filter, wherein the pixel array comprises compensation units, and each of the compensation units comprises one reference pixel among the reference pixels and two or more pixels among the pixels, and wherein the number of the two or more pixels included in each of the compensation units varies according to the position of each of the compensation units on the pixel array.

Images