Imaging device and control method thereof

Abstract

The imaging device 100 includes a data acquisition unit 20 that acquires first partial binning data p1 based on a first pixel group consisting of at least one pixel by binning in a unit pixel group consisting of a plurality of grouped pixels, an analysis unit 30 that analyzes the frequency characteristics of the video signal of an area consisting of the unit pixel group based on the cross-correlation between the first partial binning data p2 and second partial binning data p2 based on a second pixel group consisting of pixels other than the first pixel group among the unit pixel group, and a moire removal unit 40 that removes moire occurring in the area consisting of the unit pixel group based on the analysis results.

Description

[Technical field]

[0001] The present disclosure relates to an imaging apparatus and a control method thereof. [Background technology]

[0002] 2. Description of the Related Art In general, there is a demand for improved performance, such as higher image quality and higher functionality, in imaging devices such as cameras, and various improvements have been made to image sensors such as CMOS sensors mounted in such imaging devices.

[0003] For example, a technique has been disclosed for achieving a high dynamic range (HDR) image by processing a plurality of pixels of an image sensor in groups (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Application Publication No. 2021 / 0385389 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the image sensor described in Patent Document 1 does not take into consideration moire that occurs in captured images, and as a result, there is a possibility that moire will occur in captured images.

[0006] Therefore, an object of the present disclosure is to provide an imaging device that appropriately removes moire and a control method thereof. [Means for solving the problem]

[0007] An imaging device according to one embodiment of the present disclosure includes a data acquisition unit that acquires first partial binning data based on a first pixel group composed of at least one pixel by binning in a unit pixel group composed of a plurality of grouped pixels, an analysis unit that analyzes the frequency characteristics of a video signal in an area composed of the unit pixel group based on the cross-correlation between the first partial binning data and second partial binning data based on a second pixel group composed of pixels other than the first pixel group in the unit pixel group, and a moiré removal unit that removes moiré that occurs in the area composed of the unit pixel group based on the analysis results.

[0008] In the above aspect, the moire remover may remove the moire by removing high frequency components of the video signal using a low pass filter.

[0009] In the above aspect, the moiré remover may remove moiré based on a video signal of an area constituted by unit pixel groups in the vicinity of the unit pixel group.

[0010] In the above aspect, the data acquisition section may acquire all binning data based on all pixels that make up the unit pixel group, and the second partial binning data may be acquired by subtracting the first partial binning data from the all binning data.

[0011] In the above aspect, the photodiodes configured to correspond to the plurality of pixels may be connected to a common floating diffusion.

[0012] In the above aspect, the floating diffusion is capable of switching between a plurality of charge-voltage conversion gains, the data acquisition unit acquires first partial binning data at a low conversion gain among the plurality of charge-voltage conversion gains, the analysis unit analyzes the frequency characteristics of the video signal of the region composed of the unit pixel group based on the cross-correlation between the first partial binning data and the second partial binning data at the low conversion gain, and the moiré removal unit removes moiré occurring in the region composed of the unit pixel group based on the analysis result at the low conversion gain.

[0013] In the above aspect, each of the multiple pixels is further composed of two or more sub-pixels, and the data acquisition unit acquires first sub-partial binning data based on a first sub-pixel group composed of at least one or more sub-pixels by binning, and second sub-partial binning data based on a second sub-pixel group composed of sub-pixels other than the first sub-pixel group, and the first sub-partial binning data and the second sub-partial binning data may be used for phase difference autofocus.

[0014] In the above aspect, each of the multiple pixels is further composed of two or more sub-pixels, and any one of the multiple pixels includes a mask pixel that masks at least one of the two or more sub-pixels, and the data acquisition unit acquires sub-partial binning data based on sub-pixels other than the mask pixel in the pixel including the mask pixel, and the sub-partial binning data may be used for phase difference autofocus.

[0015] An imaging device according to another aspect of the present disclosure includes a data acquisition unit that acquires first partial binning data based on a first pixel group composed of at least one pixel by binning in a unit pixel group composed of a plurality of grouped pixels, an analysis unit that analyzes frequency characteristics of a video signal in an area composed of the unit pixel group based on cross-correlation between the first partial binning data and second partial binning data based on a second pixel group composed of pixels other than the first pixel group in the unit pixel group, and an image generation unit that generates an image while restoring high frequency components in the area composed of the unit pixel group based on the analysis results.

[0016] A control method according to one embodiment of the present disclosure is a control method executed by a processor included in an imaging device, and includes a data acquisition step of acquiring first partial binning data based on a first pixel group composed of at least one pixel by binning in a unit pixel group composed of a plurality of grouped pixels, an analysis step of analyzing the frequency characteristics of a video signal in an area composed of the unit pixel group based on the cross-correlation between the first partial binning data and second partial binning data based on a second pixel group composed of pixels other than the first pixel group among the unit pixel group, and a moiré removal step of removing moiré occurring in the area composed of the unit pixel group based on the analysis results. Effect of the Invention

[0017] According to the present disclosure, it is possible to provide an imaging device that appropriately removes moire and a control method thereof. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a schematic diagram for explaining the configuration of an image sensor 10 according to a first embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram for explaining binning used in the image sensor 10 according to the first embodiment of the present disclosure. [Diagram 3] FIG. 2 is a diagram for explaining partial binning used in the image sensor 10 according to the first embodiment of the present disclosure. [Figure 4] FIG. 2 is a block diagram for explaining each function and data flow of the imaging device 100 according to the first embodiment of the present disclosure. [Diagram 5] FIG. 13 is a diagram showing a schematic circuit configuration relating to a signal flow for explaining an example of binning in 4 (2×2) pixels. [Figure 6] FIG. 6 is a diagram for explaining the operation of each element in the circuit configuration of 4 (2×2) pixels shown in FIG. 5. [Figure 7] 11 is a flowchart showing the flow of processing of a control method M100 executed by the imaging device 100 according to the first embodiment of the present disclosure. [Figure 8]FIG. 11 is a diagram showing another partial binning (another specific example 1) used in the image sensor 10 according to the first embodiment of the present disclosure. [Figure 9] FIG. 11 is a diagram showing another partial binning (another specific example 2) used in the image sensor 10 according to the first embodiment of the present disclosure. [Figure 10] FIG. 11 is a diagram showing another partial binning (another specific example 3) used in the image sensor 10 according to the first embodiment of the present disclosure. [Figure 11] FIG. 13 is a diagram illustrating a circuit configuration related to a signal flow for explaining an example in which dual conversion gain and full-pixel image-surface phase-difference AF are operated in combination in 4 (2×2) pixels. [Figure 12] FIG. 12 is a diagram for explaining the operation of each element in the circuit configuration of 4 (2×2) pixels shown in FIG. [Figure 13] FIG. 2 is a schematic diagram showing an image sensor in which dedicated pixels are set for acquiring signals for phase-difference AF. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that each embodiment described below is merely a specific example for implementing the present disclosure, and is not intended to limit the interpretation of the present disclosure. In addition, in order to facilitate understanding of the description, the same reference numerals are used as much as possible for the same components in each drawing, and duplicate descriptions may be omitted.

[0020] First Embodiment [About image sensors] Fig. 1 is a schematic diagram for explaining the configuration of an image sensor 10 according to a first embodiment of the present disclosure. As shown in Fig. 1, the image sensor 10 is typically a CMOS image sensor or the like, and includes a control circuit 1, a group of multiple pixels 2 arranged two-dimensionally, signal lines 3, a readout circuit 4, and a digital signal processor (DSP) 5.

[0021] Note that here, pixel group 2 is formed by grouping 4 (2 x 2) pixels into one pixel group (unit pixel group), but this is not limited to this. For example, 3 (3 x 1) pixels, 8 (4 x 2) pixels, 9 (3 x 3) pixels, and 16 (4 x 4) pixels may be used as unit pixel groups.

[0022] The control circuit 1 drives a plurality of pixel groups 2 of the image sensor 10, reads out data based on optical signals accumulated in the plurality of pixel groups 2, and controls the image sensor 10 to output the data to the outside.

[0023] The multiple pixel groups 2 are arranged two-dimensionally, and accumulate optical signals brought to the image sensor 10 based on control signals from the control circuit 1 and control signals generated by the pixel groups 2 themselves, and read out as data (electrical signals) based on the optical signals.

[0024] Electric signals read out from the pixel groups 2 are transmitted to the readout circuit 4 via signal lines 3 (typically, column signal lines parallel to the column direction), and the electric signals are converted from analog to digital.

[0025] A digital signal processor (DSP) 5 processes the digital signal that has been analog-to-digital converted by the readout circuit 4. Then, the processed digital signal is transmitted to a processor, memory, or the like included in the imaging device via a data bus.

[0026] The DSP5 is not limited to such an arrangement, and may be configured such that, for example, the image sensor 10 does not include the DSP5, and a downstream processor includes the DSP. Furthermore, a part of the digital signal processing in the image processing may be processed by the DSP5 of the image sensor 10 and a DSP included in the downstream processor. In other words, the position of the DSP in the present disclosure is not limited to a specific position.

[0027] [About binning] Fig. 2 is a diagram for explaining binning used in the image sensor 10 according to the first embodiment of the present disclosure. In Fig. 2, as an example, in a color pixel arrangement of a single-chip Bayer array, each color is composed of 4 (2 x 2) pixels.

[0028] If data from each pixel is read as if each pixel were an individual pixel, a high-resolution image based on a high sampling frequency can be obtained.On the other hand, as shown in Figure 2, if data from four pixels are read by treating four pixels as one pixel group (unit pixel group) using binning, a high SNR based on a high number of signal electrons, high sensitivity based on a wide pixel size, a high frame rate based on a small number of pixels, and low power consumption based on low readout can be achieved.

[0029] That is, there is a trade-off between resolution and other properties depending on the binning. Specifically, when data is read out from all pixels as if each pixel were an individual, independent pixel, the sampling frequency during readout is fs (full pixel readout mode). In contrast, when data is read out from four pixels as one pixel group (unit pixel group) by binning, the sampling frequency during readout is reduced to fs / 2 (binning mode).

[0030] Fig. 3 is a diagram for explaining partial binning used in the image sensor 10 according to the first embodiment of the present disclosure. In Fig. 3, as an example, a Bayer array consisting of green (G), red (R), blue (B), and green (G) is regarded as one Bayer unit, and the Bayer units are arranged in a matrix.

[0031] Note that here, it is composed of one Bayer unit and four (2 x 2) unit pixel groups of G, R, B, and G, but this is not limited to this and may be composed of, for example, nine (3 x 3) unit pixel groups or 16 (4 x 4) unit pixel groups.

[0032] In the even-numbered row group, as indicated by the number "1," for example, of a unit pixel group of 4 (2 x 2) pixels consisting of G, the left half of the two pixels (first pixel group) are subjected to partial binning and data is read out (first partial binning data). Next, the 4 (2 x 2) pixels consisting of G are subjected to full binning and data is read out (full binning data).

[0033] In the odd-numbered row group, as indicated by the number "1," for example, of a unit pixel group of 4 (2 x 2) pixels consisting of G, the upper two pixels (first pixel group) are subjected to partial binning and data is read out (first partial binning data). Next, the 4 (2 x 2) pixels consisting of G are subjected to full binning and data is read out (full binning data).

[0034] In addition, of the unit pixel group of 4 (2 x 2) pixels composed of G, the data for the right half and lower half can be generated based on the difference between the full binning data read out by full binning and the partial binning data read out by partial binning (second pixel group, second partial binning data).

[0035] Note that, here, a specific example has been given of a portion of the unit pixel groups of 4 (2 x 2) pixels composed of G, however, the same processing is performed for the other unit pixel groups of 4 (2 x 2) pixels composed of G, the unit pixel groups of 4 (2 x 2) pixels composed of R, and the unit pixel groups of 4 (2 x 2) pixels composed of B.

[0036] [About moire removal] The following describes a process for removing moire (aliasing) using the partial binning data and full binning data output from the image sensor 10. Note that moire is a type of noise that occurs in an image, but the removal of moire in this specification also includes the removal of similar aliasing, which can naturally be processed.

[0037] Fig. 4 is a block diagram for explaining each function and data flow of the imaging device 100 according to the first embodiment of the present disclosure. As shown in Fig. 4, the imaging device 100 includes an image sensor 10, a data acquisition unit 20, an analysis unit 30, and a moire removal unit 40. Note that, although an optical system, a memory, and the like are not shown and detailed description is omitted here, the imaging device 100 also includes functions and members that are generally included in imaging devices. Note that the imaging device of the present disclosure is applied to digital cameras and terminals such as smartphones, tablets, and personal computers equipped with a camera function.

[0038] The image sensor 10 is the image sensor described above with reference to FIGS. 1 to 3, and as shown in FIG. 4, the first partial binning data p1 and the entire binning data a1 are read out from the image sensor 10.

[0039] Here, the full binning data a1, the first partial binning data p1, and the second partial binning data p2 may be read and generated, for example, according to the procedure described above with reference to FIG.

[0040] Specifically, the first partial binning data p1 is data based on a first pixel group consisting of at least one pixel by binning in a unit pixel group consisting of a plurality of grouped pixels. The first partial binning data p1 corresponds to data read from two pixels indicated by the number "1" in the unit pixel group in FIG.

[0041] The total binning data a1 is data based on all pixels in a unit pixel group made up of a plurality of grouped pixels. In FIG. 3, the total binning data a1 corresponds to data read out from 4 (2×2) pixels in the unit pixel group.

[0042] Then, by subtracting the first partial binning data p1 from the entire binning data a1, the second partial binning data p2 is generated based on the difference.

[0043] The analysis section 30 analyzes the frequency characteristics of the video signal of the region formed by the unit pixel group, based on the cross-correlation between the first partial binning data p1 and the second partial binning data p2.

[0044] For example, the analysis unit 30 calculates the cross-correlation between the first partial binning data p1 and the second partial binning data p2, and if the cross-correlation is small (below a predetermined threshold), it determines that the area composed of the unit pixel group contains a large number of high-frequency components.

[0045] Furthermore, moire is highly likely to occur periodically, and taking this characteristic into consideration, the analysis section 30 may estimate an area of ​​the image sensor 10 that is composed of a group of unit pixels in which moire has occurred.

[0046] For example, in the example shown in FIG. 3, the analysis unit 30 calculates the cross-correlation between the first partial binning data based on the two pixels in the left half of the unit pixel group and the second partial binning data based on the two pixels in the right half of the unit pixel group in the even row group. Also, the analysis unit 30 calculates the cross-correlation between the first partial binning data based on the two pixels in the upper half of the unit pixel group and the second partial binning data based on the two pixels in the lower half of the unit pixel group in the odd row group. That is, although the analysis unit 30 analyzes the frequency characteristics of the video signal in the vertical direction and the horizontal direction in the unit pixel group, the binning grouping is set alternately for each row in the vertical direction and the horizontal direction. Therefore, depending on the occurrence state of the moire, it may be impossible to determine all the areas (unit pixel groups) in which the moire occurs. However, as described above, by assuming that the moire occurs periodically (with a predetermined length and period) in a striped pattern, it is possible to estimate which area (unit pixel group) the moire occurs in.

[0047] The threshold for calculating the cross-correlation between the first partial binning data p1 and the second partial binning data p2 and determining that a large amount of high frequency components is included may be preset or changeable depending on, for example, the type and performance of the imaging device including the lens and image sensor, the subject and surrounding environment, and other imaging conditions. Also, an appropriate threshold may be set by learning using AI (Artificial Intelligence). Furthermore, whether moire occurs may be determined using AI, for example, using the first partial binning data p1, the second binning data p2, and the entire binning data a1 as teacher data.

[0048] In this way, the analysis unit 30 is not particularly limited in its analysis method, but can use various analysis methods to analyze the frequency characteristics of the video signal in an area composed of a group of unit pixels, and detect areas that contain a large amount of high frequency components and areas where moire occurs.

[0049] The moire remover 40 removes moire occurring in an area formed by a group of unit pixels based on the analysis result by the analyzer 30 .

[0050] For example, the moire remover 40 may remove high-frequency components from a video signal in an area containing many high-frequency components (for example, all binning data a1 of the unit pixel group) using a low-pass filter.

[0051] Furthermore, the moiré remover 40 may remove moiré based on video signals in areas where no moiré occurs near the area where moiré occurs (for example, all binning data a1 of other unit pixel groups). Here, an area where no moiré occurs near the area where moiré occurs is, for example, an area where no moiré occurs among areas adjacent to the area where moiré occurs (up, down, left, right, diagonal positions on an extension of a diagonal line) and surrounding the area where moiré occurs. In other words, the moiré remover 40 generates an image without moiré by interpolating the area where moiré occurs from video signals of other areas.

[0052] In addition, for areas (unit pixel groups) that are determined by the analysis unit 30 to include high-frequency components and require processing, regardless of the occurrence of moire, an image generation unit (not shown) that properly restores the high-frequency components may be provided instead of or in addition to the moire removal unit 40. Typically, the image generation unit may properly restore the high-frequency components in an area (unit pixel group) that is determined to require image processing by interpolating from the video signal of a neighboring area, or may further properly restore the high-frequency components using AI.

[0053] 4, the data acquisition unit 20 acquires the second partial binning data p2 by subtracting the first partial binning data p1 from the total binning data a1, and the analysis unit 30 calculates the cross-correlation between the first partial binning data and the second partial binning data p2, but is not limited to this. For example, the analysis unit 30 may analyze the frequency characteristics of the video signal of the region formed by the unit pixel group based on the cross-correlation between the first partial binning data p1 and the second partial binning data p2, based on the first partial binning data p1 and the total binning data a1.

[0054] The data acquisition unit 20 reads out the first partial binning data p1 and the entire binning data a1 from the image sensor 10, but is not limited to this. For example, the first partial binning data p1 and the second partial binning data p2 may be read out. In this case, the analysis unit 30 may analyze the frequency characteristics of the video signal of the area formed by the unit pixel group based on the cross-correlation between the first partial binning data p1 and the second partial binning data p2 read out from the image sensor 10.

[0055] Furthermore, based on the analysis results by the analysis unit 30, the moire removal unit 40 and / or the image generation unit typically generates an appropriate image based on the total binning data a1 for areas containing a large amount of high frequency components, including removing moire, but may also generate an image based on the first partial binning data p1 and the second partial binning data p2.

[0056] Furthermore, the moire remover 40 and / or the image generator may generate a suitable image after performing demosaic processing on the entire binning data a1, or the first partial binning data p1 and the second partial binning data p2.

[0057] [Circuit configuration of each pixel in the image sensor] A specific method of binning a unit pixel group of an image sensor will be described. Here, a more specific configuration and operation of the unit pixel group in the image sensor will be described in detail.

[0058] Fig. 5 is a diagram showing a schematic circuit configuration related to a signal flow for explaining an example of binning in 4 (2 x 2) pixels. As shown in Fig. 5, the 4 (2 x 2) pixels correspond to four photodiodes (PD1 to PD4), and are composed of floating diffusions (FD), source follower amplifiers (SF), reset transistors (RES), transfer transistors (TX1 to TX4), and selection transistors (SEL) connected to the photodiodes.

[0059] The four photodiodes (PD1 to PD4) are connected to a common floating diffusion (FD). The output of the source follower amplifier (SF) is connected to a common output line (corresponding to the signal line 3 in FIG. 1) in a column in which a plurality of pixel groups are arranged two-dimensionally via a selection transistor (SEL), and further connected to a constant current source (I) that serves as a load of the source follower amplifier (SF), a voltage gain conversion means (not shown), and an analog-to-digital converter (ADC).

[0060] Then, the digital signal (data) converted by the analog-to-digital converter (ADC) is held in the line memory 1 or the line memory 2.

[0061] FIG. 6 is a diagram for explaining the operation of each element in the circuit configuration of the four (2×2) pixels shown in FIG.

[0062] At time t1, the reset transistor (RES) and the transfer transistors (TX1 to TX4) are turned on to reset the photodiodes (PD1 to PD4).

[0063] After that, after a predetermined accumulation period for accumulating data has elapsed, the data read process is started from the pixels constituting the unit pixel group, but first, at time t2, the reset transistor (RES) is turned OFF and the selection transistor (SEL) is turned ON. Then, the value is converted from analog to digital with a predetermined voltage gain and stored in the line memory 1 (FD reset noise).

[0064] At time t3, for partial binning, for example, the transfer transistors (TX1 to TX2) of the transfer transistors (TX1 to TX4) are turned on to transmit signals from the photodiodes (PD1 to PD2) to the floating diffusion (FD). Then, the values ​​are analog-to-digital converted with a predetermined voltage gain and stored in the line memory 2 (partial binning data).

[0065] At time t4, the value held in line memory 1 is subtracted from the value held in line memory 2, and the result is output and transmitted to the downstream image signal processor (ISP) or frame memory. This makes it possible to obtain data from which the reset noise of the floating diffusion (FD) has been removed, a process called correlated double sampling (noise-removed partial binned data). This corresponds to the first partial binned data p1 in Figure 4.

[0066] At time t5, for all binning, the transfer transistors (TX1 to TX4) are turned on to transmit the signals from the photodiodes (PD1 to PD4) to the floating diffusion (FD). Then, the values ​​are converted from analog to digital with a predetermined voltage gain and stored in the line memory 2 (all binning data).

[0067] It is assumed here that the output of the partial binning data stored in line memory 2 is completed before the analog-to-digital conversion of all binning data is completed. However, if the output of the partial binning data is not completed, it is advisable to provide another line memory for storing all binning data.

[0068] In addition, since the data stored in line memory 1 can be used for the reset noise of the floating diffusion (FD) of all binning, at time t6, the value stored in line memory 1 is subtracted from the value stored in line memory 2, and the result is output. This makes it possible to obtain all binning data from which the reset noise of the floating diffusion (FD) has been removed (noise-removed · all binning data). This corresponds to the all binning data a1 in Figure 4.

[0069] In this manner, the first partial binning data p1 and the entire binning data a1 are extracted from each unit pixel group of the image sensor 10.

[0070] [Control method] Next, a control method for generating an image while removing moire using binning data will be described in detail.

[0071] 7 is a flowchart showing a process flow of a control method M100 executed by the imaging device 100 according to the first embodiment of the present disclosure. As shown in FIG 7, the control method M100 includes steps S10 to S50, and each step is executed by a processor included in the imaging device 100.

[0072] In step S10, the data acquisition unit 20 acquires first partial binning data based on a first pixel group among the unit pixel groups (data acquisition step). As a specific example, as shown in Fig. 3 and Fig. 4, the data acquisition unit 20 performs partial binning on two pixels indicated by the number "1" among a unit pixel group of four (2 x 2) pixels from the image sensor 10 and reads out data (first partial binning data p1).

[0073] In step S20, the analysis unit 30 analyzes the frequency characteristics of the video signal of the region composed of the unit pixel group based on the cross-correlation between the first partial binning data acquired in step S10 and the second partial binning data based on the second pixel group composed of pixels other than the first pixel group among the unit pixel group (analysis step). As a specific example, as shown in Fig. 3 and Fig. 4, the data acquisition unit 20 performs full binning on all pixels of the unit pixel group of 4 (2 x 2) pixels from the image sensor 10, reads out data (full binning data a1), and subtracts the first partial binning data p1 to acquire the second partial binning data p2. Then, the analysis unit 30 calculates the cross-correlation between the first partial binning data p1 and the second partial binning data p2, and analyzes the frequency characteristics of the video signal of the region composed of the unit pixel group.

[0074] In step S30, the analysis unit 30 judges whether or not the region constituted by the unit pixel group is a processing target region that contains many high frequency components and requires processing of the high frequency components. As a specific example, the analysis unit 30 judges whether or not the region constituted by the unit pixel group is a processing target region for high frequency components based on the cross-correlation between the first partial binning data p1 and the second partial binning data p2 calculated in step S20. If the cross-correlation is small, the region contains many high frequency components and is therefore judged to be a processing target region for high frequency components (Yes in step S30), and if the cross-correlation is large, the region is judged to be not a processing target region for high frequency components (No in step S30).

[0075] In step S40 (Yes in step S30), the moiré remover 40 generates an image while removing moiré that occurs in the region formed by the unit pixel group (moiré removal step). As a specific example, the moiré remover 40 generates an image while removing moiré from the region formed by the unit pixel group by removing high-frequency components using a low-pass filter or by interpolating from video signals of other regions.

[0076] In step S50 (No in step S30), the image generating unit generates an appropriate image for the region configured by the unit pixel group, based on all the binning data a1.

[0077] As described above, according to the imaging device 100 and control method M100 according to the first embodiment of the present disclosure, the data acquisition unit 20 acquires the first partial binning data p1 based on the first pixel group among the unit pixel groups, the analysis unit 30 analyzes the frequency characteristics of the video signal of the region made up of the unit pixel groups based on the cross-correlation between the first partial binning data p1 and the second partial binning data p2, and the moiré removal unit 40 removes the moiré occurring in the region made up of the unit pixel groups based on the analysis results. As a result, an image can be generated while appropriately removing the moiré.

[0078] [Other examples of grouping unit pixels (partial binning)] In this embodiment, as shown in Fig. 3, four (2 x 2) pixels are grouped to form a unit pixel group, and the left two pixels or the upper two pixels are subjected to partial binning to read out data as the first partial binning data, but partial binning is not limited to this. Other specific examples of partial binning will be described below.

[0079] (Another Example 1) Fig. 8 is a diagram showing another partial binning (another specific example 1) used in the image sensor 10 according to the first embodiment of the present disclosure. As shown in Fig. 8, Bayer units each consisting of green (G), red (R), blue (B), and green (G) are arranged in a matrix, similar to Fig. 3.

[0080] In the even-numbered row group, as indicated by the number "1," two pixels (first pixel group) at the top left and bottom right of the unit pixel group are subjected to partial binning and data is read out (first partial binned data).

[0081] In the odd-numbered row group, as indicated by the number "1," the two pixels (first pixel group) at the top right and bottom left of the unit pixel group are subjected to partial binning and data is read out (first partial binned data).

[0082] In this way, the pixels arranged on the diagonal line of the unit pixel group are subjected to partial binning. The rest of the process is the same as that described with reference to FIG.

[0083] (Another Example 2) Fig. 9 is a diagram showing another partial binning (another specific example 2) used in the image sensor 10 according to the first embodiment of the present disclosure. As shown in Fig. 9, Bayer units each consisting of green (G), red (R), blue (B), and green (G) are arranged in a matrix, similar to Fig. 3.

[0084] In the even row group and odd row group, as indicated by the number “1”, three pixels (first pixel group) at the top right, bottom right, and bottom left of the unit pixel group are subjected to partial binning and data is read out (first partial binned data).

[0085] In this way, three pixels out of the unit pixel group (four pixels) are subjected to partial binning. The rest of the process is the same as that described with reference to FIG.

[0086] In the example of Fig. 9, a plurality of pixels in a unit pixel group are grouped asymmetrically and subjected to partial binning. This allows the analysis unit 30 to more appropriately analyze the frequency of the video signal in the vertical and horizontal directions in the unit pixel group based on the cross-correlation between the first partial binning data (first pixel group indicated by the number "1") and the second partial binning data (second pixel group other than the first pixel group in the unit pixel group). In other words, the analysis unit 30 can more appropriately analyze whether a region formed by the unit pixel group contains many high-frequency components and whether moire occurs.

[0087] (Other Example 3) Fig. 10 is a diagram showing another partial binning (another specific example 3) used in the image sensor 10 according to the first embodiment of the present disclosure. As shown in Fig. 10, Bayer units each consisting of green (G), red (R), blue (B), and green (G) are arranged in a matrix, similar to Fig. 3.

[0088] In the even row group, as indicated by the number "1," the two pixels in the left half of the unit pixel group (first pixel group) are partially binned and data is read out (first partial binned data), and further, the upper right pixel is partially binned in addition to the first pixel group or alone, and data is read out (additional partial binned data).

[0089] In the odd-numbered row group, as indicated by the number "1," the top two pixels (first pixel group) of the unit pixel group are partially binned to read out data (first partial binning data), and further, the bottom left pixel is partially binned in addition to the first pixel group or alone to read out data (additional partial binning data).

[0090] In this way, among the unit pixel groups, the first pixel group is subjected to partial binning, and further, a different pixel group (the first pixel group+other pixels or other pixels alone) is subjected to partial binning. Then, the unit pixel group is subjected to full binning, and all binning data is read out.

[0091] 10, a plurality of partial binning data are obtained in an area composed of pixel groups with different centers of gravity, and thus a plurality of second partial binning data can be obtained by subtracting the partial binning data from the total binning data. Based on the first partial binning data and the second partial binning data obtained in various combinations in this way, the analysis unit 30 can more appropriately analyze whether a large number of high frequency components are included and whether moire is occurring in the area composed of the unit pixel group.

[0092] As illustrated here, there are various modes of partial binning, but the present invention is not limited thereto. The pixels to be partially binned in the unit pixel group may be set regularly or randomly. In order for the analysis unit 30 to appropriately analyze the unit pixel group (area) that contains many high-frequency components and where moire occurs, the pixels to be partially binned in the unit pixel group may be set according to, for example, the type and performance of the imaging device including the lens and image sensor, the subject and surrounding environment, and other imaging conditions.

[0093] As described above, one unit pixel group is not limited to being composed of 4 (2×2) pixels, and may be composed of, for example, 3 (3×1) pixels, 8 (4×2) pixels, 9 (3×3) pixels, 16 (4×4) pixels, etc., and one Bayer unit is not limited to being composed of 4 (2×2) unit pixel groups, and may be composed of, for example, 9 (3×3) unit pixel groups, 16 (4×4) unit pixel groups, etc. How to set the pixels to be partially binned among these may be determined appropriately, and may be determined using AI.

[0094] <Second embodiment> Next, a specific method of combining and operating dual conversion gain (DCG) and full-pixel image plane phase difference AF (autofocus) as an image sensor according to a second embodiment of the present disclosure will be described. The basic configuration of the image sensor according to this embodiment is the same as that of the image sensor 10 according to the first embodiment, and the same idea as in the first embodiment is used for pixel binning. Here, a specific configuration and operation of combining dual conversion gain and full-pixel image plane phase difference AF for pixels in the image sensor will be described in detail.

[0095] Fig. 11 is a diagram showing a schematic circuit configuration related to a signal flow for explaining an example of combining dual conversion gain and full-pixel imaging surface phase difference AF in 4 (2 x 2) pixels. As shown in Fig. 11, here, the four photodiodes (PD1 to PD4) shown in Fig. 5 are divided into two for full-pixel imaging surface phase difference AF, and each is a sub-photodiode (PD1L, PD1R to PD4L, PD4R), and the transfer transistors (TX1 to TX4) are transfer transistors (TX1L, TX1R to TX4L, TX4R) corresponding to the sub-photodiodes (L: left, R: right).

[0096] In this circuit, a floating diffusion (FD), a source follower amplifier (SF), a reset transistor (RES), a switching transistor (X), and a selection transistor (SEL) are arranged.

[0097] In addition, for dual conversion gain, an additional load capacitance that can be electrically switched via a switching transistor (X) is added to the pixel. By increasing the load capacitance of the floating diffusion (FD), the charge-voltage conversion gain can be set small, and by decreasing the load capacitance, the charge-voltage conversion gain can be set large.

[0098] Fig. 12 is a diagram for explaining the operation of each element in the circuit configuration of 4 (2 x 2) pixels shown in Fig. 11. Each of the 4 (2 x 2) pixels is composed of sub-pixels divided into two (L: left, R: right). Here, each pixel is composed of two sub-pixels, but this is not limited thereto, and each pixel may be composed of, for example, three or more sub-pixels.

[0099] At time t1, the reset transistor (RES), the switching transistor (X), and the transfer transistors (TX1L·TX1R to TX4L·TX4R) are turned ON to reset the sub-photodiodes (PD1L·PD1R to PD4L·PD4R).

[0100] Then, after a predetermined accumulation period for accumulating data has elapsed, the data readout process is started from the pixels constituting the unit pixel group, but first, at time t2, the reset transistor (RES) is turned OFF, and the switching transistor (X) and the selection transistor (SEL) are turned ON. Then, with the charge-voltage conversion gain of the floating diffusion (FD) reduced (LCG), the FD reset noise is analog-to-digital converted and stored in the line memory 1 (LCG FD reset noise).

[0101] At time t3, the switching transistor (X) is turned OFF, and with the charge-voltage conversion gain of the floating diffusion (FD) increased (HCG), the FD reset noise is AD converted and stored in line memory 2 (HCG·FD reset noise).

[0102] At time t4, the transfer transistor (TX1L) and the transfer transistor (TX2L) are turned ON, and the left portion binning data for image plane phase difference AF in the HCG state is acquired, AD converted, and stored in the line memory 3 (L portion binning data for HCG phase difference AF).

[0103] Then, by subtracting the HCG·FD reset noise stored in line memory 2 from the L partial binning data for HCG·phase difference AF stored in line memory 3, it is possible to obtain L partial binning data for phase difference AF in an HCG state with the reset noise removed (noise removed·HCG·phase difference AF L partial binning data).

[0104] At time t5, the transfer transistors (TX1L·TX1R) and the transfer transistors (TX2L·TX2R) are turned ON, the partial binning data in the HCG state is acquired, AD converted, and stored in the line memory 3 (HCG·partial binning data).

[0105] By subtracting the HCG·FD reset noise stored in line memory 2 from the HCG·partial binning data stored in line memory 3, partial binning data in the HCG state with the reset noise removed can be obtained (noise-removed·HCG·partial binning data).

[0106] Moreover, by subtracting the noise reduction·HCG·phase difference AF L partial binning data from the noise reduction·HCG·partial binning data, it is possible to obtain the noise reduction·HCG·phase difference AF R partial binning data.

[0107] At time t6, the switching transistor (X) is turned ON, and with the charge-voltage conversion gain of the floating diffusion (FD) reduced (LCG), the transfer transistors (TX1L, TX1R) and transfer transistors (TX2L, TX2R) are turned ON, and the partial binning data in the LCG state is acquired, AD converted, and stored in line memory 3 (LCG partial binning data).

[0108] Then, by subtracting the LCG·FD reset noise stored in line memory 1 from the LCG·partial binning data stored in line memory 3, partial binning data in the LCG state with the reset noise removed can be obtained (noise-removed·LCG·partial binning data).

[0109] At time t7, the transfer transistors (TX1L·TX1R to TX4L·TX4R) are turned ON, all binning data in the LCG state is acquired, AD converted, and stored in the line memory 3 (LCG·all binning data).

[0110] At time t8, by subtracting the LCG·FD reset noise stored in line memory 1 from the LCG·all binning data stored in line memory 3, it is possible to obtain all binning data in the LCG state with the reset noise removed (noise removed·LCG·all binning data).

[0111] Thus, in the HCG state, L partial binning data and partial binning data (corresponding to the first partial binning data p1) for phase difference AF are extracted from the image sensor 10, and in the LCG state, partial binning data (corresponding to the first partial binning data p1) and total binning data (corresponding to the total binning data a1) are extracted. Note that, as described above, in the HCG state, R partial binning data for phase difference AF can be obtained by calculation.

[0112] As described above, according to the imaging device and control method equipped with the image sensor according to the second embodiment of the present disclosure, in the LCG state, the partial binning data (corresponding to the first partial binning data p1) and the full binning data (corresponding to the full binning data a1) are extracted, so that an image can be generated while appropriately removing moire, as in the first embodiment of the present disclosure. By appropriately removing moire from the high SNR data in the LCG state, it is possible to prevent moire from occurring in a fine image.

[0113] Also, in this embodiment, in the HCG state, not all binning data is read out, but if the transfer transistors (TX1L, TX1R to TX4L, TX4R) are turned on in the HCG state and all binning data in the HCG state is acquired and AD converted, it is possible to acquire HCG and all binning data. Since transistor switching and AD conversion processing impose a load on the processor mounted in the imaging device 100, here, by reducing the number of transistor switching and AD conversion, it is possible to suppress an increase in the load on the processor mounted in the imaging device 100 and an increase in power consumption.

[0114] Furthermore, in this embodiment, partial binning data for phase-difference AF in the HCG state can be acquired. A high SNR is required for phase-difference AF data, and since data can be acquired in the HCG state that is resistant to noise, this is extremely effective.

[0115] In this embodiment, in the LCG state, it is not possible to obtain binning data for phase-difference AF. However, it is also possible to obtain a signal for phase-difference AF by setting dedicated pixels in the image sensor and masking part of the pixels or by using a (2×1) on-chip microlens structure.

[0116] Fig. 13 is a schematic diagram showing an image sensor in which a dedicated pixel is set to obtain a signal for phase difference AF. As shown in Fig. 13, a dedicated pixel is set among a plurality of pixels arranged in the image sensor, and the dedicated pixel is masked, for example, in the left half (L area) or the right half (R area). Note that here, the dedicated pixel is divided into two parts, left and right, but is not limited to this. For example, it is sufficient to divide the dedicated pixel into two or three or more parts, top and bottom, and mask it so that the phase signal used for phase difference AF can be appropriately obtained.

[0117] With the dedicated pixels, if a phase difference signal is optically obtained in an unmasked area in the LCG state, data for LCG and phase difference AF can be obtained.

[0118] The above-described embodiments are intended to facilitate understanding of the present disclosure and are not intended to limit the present disclosure. The elements of the embodiments, as well as their arrangements, materials, conditions, shapes, sizes, etc., are not limited to those illustrated and can be changed as appropriate. In addition, configurations shown in different embodiments can be partially substituted or combined with each other. [Explanation of symbols]

[0119] 1... control circuit, 2... pixel group, 3... signal line, 4... readout circuit, 5... digital signal processing unit (DSP), 10... image sensor, 20... data acquisition unit, 30... analysis unit, 40... moire removal unit, 100... imaging device, M100... control method, S10 to S50... each step of control method M100

Claims

1. a data acquisition unit that acquires first partial binning data based on a first pixel group, the first pixel group being composed of at least one pixel by binning, in a unit pixel group composed of a plurality of grouped adjacent pixels of the same color; an analysis unit that analyzes a frequency characteristic of a video signal of an area configured by the unit pixel group based on a cross-correlation between the first partial binning data and second partial binning data based on a second pixel group including pixels other than the first pixel group among the unit pixel groups; a moire remover that removes moire occurring in an image generated based on a video signal of an area configured by the unit pixel group based on the analysis result. Imaging device.

2. the moire removal unit removes the moire by removing high frequency components of the video signal using a low pass filter. The imaging device according to claim 1 .

3. the moire removal unit removes the moire based on a video signal of an area where no moire occurs, the area being configured of unit pixel groups adjacent to the unit pixel groups. The imaging device according to claim 1 .

4. the data acquisition unit acquires all binning data based on all pixels constituting the unit pixel group; the second partial binning data is obtained by subtracting the first partial binning data from the total binning data; The imaging device according to claim 1 .

5. Each of the photodiodes corresponding to the plurality of pixels is connected to a common floating diffusion. The imaging device according to claim 1 .

6. the floating diffusion is capable of switching a plurality of charge-to-voltage conversion gains; the data acquisition unit acquires the first partial binning data at a low conversion gain among the plurality of charge-voltage conversion gains, the analysis unit analyzes a frequency characteristic of a video signal of an area constituted by the unit pixel group based on a cross-correlation between the first partial binning data and the second partial binning data at the low conversion gain; the moiré removal unit removes moiré occurring in an area constituted by the unit pixel group based on the analysis result at the low conversion gain. The imaging device according to claim 5 .

7. Each of the plurality of pixels is further composed of two or more sub-pixels, the data acquisition unit acquires, from the two or more sub-pixels, first sub-partial binning data based on a first sub-pixel group constituted by at least one sub-pixel by binning, and second sub-partial binning data based on a second sub-pixel group constituted by sub-pixels other than the first sub-pixel group; the first sub-partial binning data and the second sub-partial binning data are used for phase-difference autofocus; The imaging device according to claim 1 .

8. Each of the plurality of pixels is further composed of two or more sub-pixels, Any one of the plurality of pixels includes a mask pixel that masks at least one or more sub-pixels among the two or more sub-pixels, the data acquisition unit acquires sub-partial binning data based on sub-pixels other than the mask pixel in a pixel including the mask pixel; The sub-portion binning data is used for phase-detection autofocus. The imaging device according to claim 1 .

9. a data acquisition unit that acquires first partial binning data based on a first pixel group, the first pixel group being composed of at least one pixel by binning, in a unit pixel group composed of a plurality of grouped adjacent pixels of the same color; an analysis unit that analyzes a frequency characteristic of a video signal of an area configured by the unit pixel group based on a cross-correlation between the first partial binning data and second partial binning data based on a second pixel group including pixels other than the first pixel group among the unit pixel groups; and an image generating unit configured to generate an image while restoring high frequency components in the area configured by the unit pixel group based on the analysis result. Imaging device.

10. The number of pixels of the image is greater than the sum of the first partial binning data and the second partial binning data. The imaging device according to claim 9.

11. A control method executed by a processor included in an imaging device, comprising: a data acquisition step of acquiring first partial binning data based on a first pixel group constituted by at least one pixel by binning in a unit pixel group constituted by a plurality of grouped adjacent pixels of the same color; an analyzing step of analyzing a frequency characteristic of a video signal of an area constituted by the unit pixel group based on a cross-correlation between the first partial binning data and second partial binning data based on a second pixel group including pixels other than the first pixel group among the unit pixel group; and a moiré removing step of removing moiré occurring in an area constituted by the unit pixel group based on the analysis result. Control methods.

12. The imaging device according to any one of claims 1 to 10 is mounted. Terminal.

Images