Imaging apparatus and control method thereof

Abstract

An imaging device (100) is provided with: a data acquisition unit (20) that acquires first partial pixel merge data (p1) based on a first pixel group comprising at least one pixel in a unit pixel group comprising a plurality of grouped pixels by pixel merge; an analysis unit (30) that, on the basis of the correlation between the first partial pixel merge data (p2) and second partial pixel merge data (p2) based on a second pixel group comprising pixels other than the first pixel group in the unit pixel group, analyzes the frequency characteristics of the video signal in a region comprising the unit pixel group; and a moire fringe removal unit (40) that removes moire fringes generated in a region constituted by the unit pixel group on the basis of the analysis result. Wherein one microlens is disposed such that two or more pixels among the plurality of pixels in the unit pixel group share one microlens.

Description

Shooting device and control method thereof Technical Field

[0001] The present disclosure relates to a photographing device and a control method thereof. Background Art

[0002] Generally speaking, in imaging devices such as cameras, there is a demand for improved performance such as higher image quality and higher functionality, and various studies have been conducted on image sensors such as CMOS sensors mounted in these imaging devices.

[0003] For example, a technology is disclosed in which a high dynamic range (HDR) image is realized by processing a plurality of pixels of an image sensor in groups (for example, see Patent Document 1).

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: U.S. Patent Application Publication No. 2021 / 0385389

[0007] Summary of the Invention

[0008] Problems to be solved by the invention

[0009] However, the image sensor described in Patent Document 1 does not take moire fringes generated in a captured image into consideration. As a result, there is a possibility that moire fringes may be generated in the captured image.

[0010] Therefore, an object of the present disclosure is to provide a photographing device and a control method thereof that can appropriately remove moire fringes.

[0011] Solutions for solving problems

[0012] According to one aspect of the present invention, an imaging device includes: a data acquisition unit that acquires first partial pixel-binning data based on a first pixel group formed by pixel-binning at least one pixel in a unit pixel group composed of a plurality of grouped pixels, and second partial pixel-binning data based on a second pixel group; the first pixel group being formed by pixel-binning from at least one pixel in the unit pixel group, and the second pixel group being formed by pixels other than the first pixel group in the unit pixel group; an analysis unit that analyzes the frequency characteristics of an image signal in a region formed by the unit pixel group based on a correlation between the first partial pixel-binning data and the second partial pixel-binning data; and a moiré removal unit that removes moiré generated in the region formed by the unit pixel group based on the analysis result. The microlens is arranged such that two or more of the plurality of pixels in the unit pixel group share the same microlens.

[0013] In the above aspect, the moire fringe removal unit may remove the moire fringe by removing a high-frequency component of the image signal using a low-pass filter.

[0014] In the above aspect, the moiré removal unit may remove the moiré based on an image signal of a region consisting of a unit pixel group near a unit pixel group.

[0015] In the above solution, the data acquisition unit may acquire full pixel combination data based on all pixels constituting the unit pixel group, and the second partial pixel combination data may be acquired by subtracting the first partial pixel combination data from the full pixel combination data.

[0016] In the above scheme, each photodiode formed corresponding to a plurality of pixels may be connected to a common floating diffusion region.

[0017] According to another embodiment of the present invention, an imaging device includes: a data acquisition unit that acquires first partial pixel-binning data based on a first pixel group formed by pixel-binning of at least one pixel in a unit pixel group formed by grouping a plurality of pixels, and second partial pixel-binning data based on a second pixel group, wherein the first pixel group is formed by pixel-binning of at least one pixel in the unit pixel group, and the second pixel group is formed by pixels in the unit pixel group other than the first pixel group; an analysis unit that analyzes the frequency characteristics of an image signal in a region formed by the unit pixel group based on a correlation between the first partial pixel-binning data and the second partial pixel-binning data; and an image generation unit that generates an image while restoring high-frequency components in the region formed by the unit pixel group based on the analysis results. A microlens is arranged such that two or more of the plurality of pixels in the unit pixel group share the same microlens.

[0018] A control method according to one embodiment of the present disclosure is executed by a processor included in a camera device, the control method comprising: a data acquisition step of acquiring first partial pixel-binning data based on a first pixel group and second partial pixel-binning data based on a second pixel group, wherein the first pixel group is formed by pixel-binning at least one pixel in a unit pixel group formed by a plurality of grouped pixels, and the second pixel group is formed by pixels in the unit pixel group other than the first pixel group; an analysis step of analyzing the frequency characteristics of an image signal in an area formed by the unit pixel group based on a correlation between the first partial pixel-binning data and the second partial pixel-binning data; and a moiré fringe removal step of removing moiré fringes generated in the area formed by the unit pixel group based on the analysis results. A microlens is configured such that two or more of the plurality of pixels in the unit pixel group share the same microlens.

[0019] Effects of the Invention

[0020] According to the present disclosure, it is possible to provide an imaging device that appropriately removes moire fringes and a method for controlling the same. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] FIG. 1 is a schematic diagram for explaining the structure of an image sensor 10 according to a first embodiment of the present disclosure.

[0022] FIG. 2 is a diagram for explaining pixel binning used in the image sensor 10 according to the first embodiment of the present disclosure.

[0023] FIG. 3 is a diagram for explaining partial pixel binning used in the image sensor 10 according to the first embodiment of the present disclosure.

[0024] FIG. 4 is a block diagram for explaining functions and data flows of the imaging device 100 according to the first embodiment of the present disclosure.

[0025] FIG. 5 is a diagram schematically showing a circuit configuration related to a signal flow for explaining an example of pixel binning in 4 (2×2) pixels.

[0026] FIG. 6 is a diagram for explaining the operation of each element in the circuit configuration of 4 (2×2) pixels shown in FIG. 5 .

[0027] FIG. 7 is a flowchart showing the processing flow of a control method M100 executed by the imaging device 100 according to the first embodiment of the present invention.

[0028] FIG. 8 is a diagram showing another partial pixel binning (another specific example 1) used in the image sensor 10 according to the first embodiment of the present disclosure.

[0029] FIG. 9 is a diagram showing another partial pixel binning (another specific example 2) used in the image sensor 10 according to the first embodiment of the present disclosure.

[0030] FIG. 10 is a diagram showing another partial pixel binning (another specific example 3) used in the image sensor 10 according to the first embodiment of the present disclosure.

[0031] FIG. 11 is an explanatory diagram showing the arrangement relationship between the microlens and the unit pixel group of the present disclosure.

[0032] FIG. 12 is an explanatory diagram showing the arrangement relationship between the microlens and the unit pixel group of the present disclosure.

[0033] FIG. 13 is an explanatory diagram showing the arrangement relationship between the microlens and the unit pixel group of the present disclosure.

[0034] FIG. 14 is an explanatory diagram showing the arrangement relationship between the microlens and the unit pixel group of the present disclosure.

[0035] FIG. 15 is an explanatory diagram showing the arrangement relationship between the microlens and the unit pixel group of the present disclosure.

[0036] FIG. 16 is an explanatory diagram showing the arrangement relationship between the microlens and the unit pixel group of the present disclosure. DETAILED DESCRIPTION

[0037] The following describes preferred embodiments of the present disclosure in detail with reference to the accompanying drawings. Each embodiment described below is merely a specific example for implementing the present disclosure and is not intended to limit the present disclosure. Furthermore, to facilitate understanding, identical components are denoted by the same reference numerals in the accompanying drawings whenever possible, and duplicate descriptions may be omitted.

[0038] <First embodiment>

[0039] [About image sensors]

[0040] FIG1 is a schematic diagram illustrating the structure of an image sensor 10 according to a first embodiment of the present disclosure. As shown in FIG1 , image sensor 10 is typically a CMOS image sensor or the like, and includes a control circuit 1, a plurality of pixel groups 2 arranged two-dimensionally, signal lines 3, a readout circuit 4, and a digital signal processing unit (DSP) 5.

[0041] In addition, here, pixel group 2 is set as a pixel group (unit pixel group) by grouping 4 (2×2) pixels, but it is not limited to this. For example, 3 (3×1) pixels, 8 (4×2) pixels, 9 (3×3) pixels and 16 (4×4) pixels can also be set as unit pixel groups.

[0042] The control circuit 1 controls to drive the plurality of pixel groups 2 of the image sensor 10 , reads out data based on light signals accumulated in the plurality of pixel groups 2 , and outputs the data to the outside of the image sensor 10 .

[0043] A plurality of pixel groups 2 are arranged two-dimensionally, and accumulate light signals directed to the image sensor 10 based on control signals from the control circuit 1 and control signals generated by the pixel groups 2 themselves, and read out as data (electrical signals) based on the light signals.

[0044] The electric signals read out from the plurality of pixel groups 2 are transmitted to the readout circuit 4 via the signal lines 3 (typically, column signal lines parallel to the column direction), and the electric signals are subjected to analog-to-digital conversion.

[0045] A digital signal processing unit (DSP) 5 processes the digital signal obtained by analog-to-digital conversion by the readout circuit 4. The processed digital signal is then transmitted to a processor or memory included in the imaging device via a data bus.

[0046] Furthermore, the DSP 5 is not limited to this configuration. For example, the image sensor 10 may not include the DSP 5, but a subsequent processor may include a DSP. Furthermore, a configuration may be employed in which the DSP 5 of the image sensor 10 and a DSP included in a subsequent processor each handle a portion of the digital signal processing involved in image processing. In other words, the location of the DSP in this disclosure is not limited to a specific location.

[0047] [About pixel binning]

[0048] Fig. 2 is a diagram for explaining pixel binning used in the image sensor 10 according to the first embodiment of the present disclosure. In Fig. 2 , as an example, in a single-plate Bayer arrangement of color pixels, each color is configured with 4 (2×2) pixels.

[0049] By reading data from each pixel as an independent pixel, a high-resolution image can be acquired at a high sampling frequency. On the other hand, as shown in Figure 2, by binning four pixels as a single pixel group (unit pixel group) and reading data from those four pixels, a high SNR based on a high signal electron count, high sensitivity based on a wide pixel size, a high frame rate based on a small number of pixels, and low power consumption based on a low readout rate can be achieved.

[0050] That is, pixel binning trades off resolution against other properties. Specifically, when data from all pixels is read out as an independent pixel, the sampling frequency during this readout is set to fs (full readout mode). In contrast, when pixel binning is used to read data from four pixels as a single pixel group (unit pixel group), the sampling frequency during this readout is reduced to fs / 2 (binning mode).

[0051] FIG3 is a diagram illustrating partial pixel binning used in the image sensor 10 according to the first embodiment of the present disclosure. FIG3 illustrates, as an example, a Bayer arrangement consisting of green (G), red (R), blue (B), and green (G) as one Bayer unit ("one Bayer unit"), which are arranged in a matrix.

[0052] In addition, here, one Bayer unit ("one Bayer unit") is composed of four (2×2) unit pixel groups of G, R, B, and G, but is not limited to this. For example, it can also be composed of nine (3×3) unit pixel groups and 16 (4×4) unit pixel groups.

[0053] In an even-numbered row group, as indicated by the number "1," for example, partial pixel binning is performed on the two pixels in the left half of a unit pixel group of 4 (2×2) pixels composed of G (the first pixel group) to read out data (first partial pixel binning data). Next, full pixel binning is performed on the same 4 (2×2) pixels composed of G to read out data (full pixel binning data).

[0054] In an odd-numbered row group, as indicated by the number "1," for example, partial pixel binning is performed on the two pixels in the upper half of a unit pixel group of 4 (2×2) pixels composed of G (a first pixel group) to read out data (first partial pixel binning data). Next, full pixel binning is performed on the same 4 (2×2) pixels composed of G to read out data (full pixel binning data).

[0055] In addition, it is possible to generate data for the right half and lower half of a unit pixel group of 4 (2×2) pixels composed of G (a second pixel group, second partial pixel merged data) based on the difference between the full pixel merged data read out by full pixel merged and the partial pixel merged data read out by partial pixel merged.

[0056] In addition, here, a part of the unit pixel group of 4 (2×2) pixels composed of G has been specifically described by way of example, but the same processing is also performed on the unit pixel group of 4 (2×2) pixels composed of G, the unit pixel group of 4 (2×2) pixels composed of R, and the unit pixel group of 4 (2×2) pixels composed of B.

[0057] [About Moire Removal]

[0058] The following describes a process for removing moiré (aliasing) using partial pixel binning data and full pixel binning data output from the image sensor 10. While moiré is a type of noise generated in an image, the removal of moiré in this specification also encompasses the removal of similar aliasing, and such removal is naturally possible.

[0059] Figure 4 is a block diagram illustrating the functions and data flow of a camera 100 according to the first embodiment of the present disclosure. As shown in Figure 4 , the camera 100 includes an image sensor 10, a data acquisition unit 20, an analysis unit 30, a moiré removal unit 40, an autofocus calculation unit 60, and a focus adjustment unit 70. While the optical system and memory are not shown here, and detailed descriptions thereof will be omitted, the camera 100 also includes the functions and components typically found in camera devices.

[0060] The image sensor 10 is the image sensor described above with reference to FIG. 1 to FIG. 3 . As shown in FIG. 4 , the first partial pixel-binning data p1 and the full pixel-binning data a1 are read out from the image sensor 10 .

[0061] The full pixel-merged data a1 , the first partial pixel-merged data p1 , and the second partial pixel-merged data p2 can be read out and generated, for example, through the steps described with reference to FIG. 3 .

[0062] Specifically, the first partial pixel-binning data p1 is based on data from a first pixel group formed by pixel-binning at least one pixel in a unit pixel group formed by grouping a plurality of pixels. In FIG3 , the first partial pixel-binning data p1 corresponds to data read from two pixels represented by the number "1" in the unit pixel group.

[0063] The full pixel combination data a1 is data based on all pixels in a unit pixel group composed of a plurality of grouped pixels. In FIG3 , the full pixel combination data a1 corresponds to data read out from 4 (2×2) pixels in the unit pixel group.

[0064] Then, by subtracting the first partial pixel-binning data p1 from the full pixel-binning data a1 , the second partial pixel-binning data p2 is generated based on the difference.

[0065] The analyzing unit 30 analyzes the frequency characteristics of the image signal of the region consisting of the unit pixel group based on the mutual correlation between the first partial pixel-binning data p1 and the second partial pixel-binning data p2 .

[0066] For example, the analysis unit 30 calculates the mutual correlation between the first partial pixel-binning data p1 and the second partial pixel-binning data p2, and when the mutual correlation is small (below a predetermined threshold), determines that the area formed by the unit pixel group includes more high-frequency components.

[0067] Furthermore, moire fringes are likely to occur periodically, and considering this characteristic, the analysis unit 30 can also estimate in which region of the image sensor 10 a group of unit pixels includes the moire fringes.

[0068] For example, in the example shown in FIG3 , in even-numbered row groups, the analysis unit 30 calculates the correlation between the first partial pixel-binning data based on the two pixels in the left half of the unit pixel group and the second partial pixel-binning data based on the two pixels in the right half. Furthermore, in odd-numbered row groups, the analysis unit 30 calculates the correlation between the first partial pixel-binning data based on the two pixels in the upper half of the unit pixel group and the second partial pixel-binning data based on the two pixels in the lower half. Specifically, the analysis unit 30 analyzes the frequency characteristics of the image signal in both the vertical and horizontal directions within the unit pixel group, but sets the pixel-binning groups alternately on a vertical and horizontal basis. Therefore, depending on the occurrence of moiré fringes, it may be impossible to determine all regions (unit pixel groups) where moiré fringes occur. However, as described above, by assuming that moiré fringes occur periodically (with a predetermined length and period) in a fringe pattern, it is possible to estimate which regions (unit pixel groups) have generated moiré fringes.

[0069] Furthermore, the threshold used to calculate the correlation between the first partial pixel-binned data p1 and the second partial pixel-binned data p2 and determine whether a high-frequency component is present can be pre-set or modified, for example, based on the type and performance of the camera device, including the lens and image sensor, the subject and surrounding environment, and other shooting conditions. Furthermore, appropriate thresholds can be set through learning using AI (artificial intelligence). Furthermore, whether moiré fringes are present can be determined using AI, for example, using the first partial pixel-binned data p1, the second pixel-binned data p2, and the full pixel-binned data a1 as training data.

[0070] As described above, the analysis unit 30 is not particularly limited to any specific analysis method, and may use various analysis methods to analyze the frequency characteristics of the image signal in the region formed by the unit pixel group and detect regions containing a large number of high-frequency components and regions where moire fringes occur.

[0071] The moire removal unit 40 removes moire fringes generated in a region consisting of a unit pixel group based on the analysis result of the analysis unit 30 .

[0072] For example, the moire removal unit 40 may remove high-frequency components of the image signal in a region containing relatively more high-frequency components (eg, the full-pixel combined data a1 of the unit pixel group) using a low-pass filter.

[0073] Furthermore, the moiré removal unit 40 can also remove moiré near the moiré-producing region based on image signals from regions where moiré is not present (e.g., full-pixel binning data a1 from another unit pixel group). Specifically, the moiré removal unit 40 interpolates the moiré-producing region with image signals from other regions to generate a moiré-free image.

[0074] Furthermore, the present invention is not limited to the generation of moiré fringes. For regions (unit pixel groups) determined by the analysis unit 30 to contain high-frequency components and to require processing of the high-frequency components, an image generation unit (not shown) may be provided in place of or in addition to the moiré fringing removal unit 40 to appropriately restore the high-frequency components. Typically, the image generation unit appropriately restores the high-frequency components by interpolating the image signals of the regions nearby within the region (unit pixel group) determined to require image processing. Furthermore, AI may be used to appropriately restore the high-frequency components.

[0075] The autofocus calculation unit 60 calculates the defocus amount by analyzing the spatial phase difference between the first partial pixel-binning data p1 and the second partial pixel-binning data p2 . The focus adjustment unit 70 adjusts the focus of the imaging device 100 according to the defocus amount.

[0076] Furthermore, in the example shown in FIG4 , the data acquisition unit 20 obtains the second partial pixel-binning data p2 by subtracting the first partial pixel-binning data p1 from the full pixel-binning data a1, and the analysis unit 30 calculates the correlation between the first partial pixel-binning data p2 and the second partial pixel-binning data p2. However, the present invention is not limited to this embodiment. For example, the analysis unit 30 may also analyze the frequency characteristics of the image signal of the region consisting of the unit pixel group based on the first partial pixel-binning data p1 and the full pixel-binning data a1, and based on the correlation between the first partial pixel-binning data p1 and the second partial pixel-binning data p2.

[0077] The data acquisition unit 20 reads the first partial pixel-binning data p1 and the full pixel-binning data a1 from the image sensor 10, but the present invention is not limited thereto. For example, the first partial pixel-binning data p1 and the second partial pixel-binning data p2 may also be read. In this case, the analysis unit 30 can analyze the frequency characteristics of the image signal of the region consisting of the unit pixel group based on the correlation between the first partial pixel-binning data p1 and the second partial pixel-binning data p2 read from the image sensor 10.

[0078] In addition, the moiré fringe removal unit 40 and / or the image generation unit generates an appropriate image based on the analysis results of the analysis unit 30, including removing moiré fringes, for an area containing more high-frequency components, typically based on the full pixel merged data a1, but can also generate an image based on the first part pixel merged data p1 and the second part pixel merged data p2.

[0079] Furthermore, the moire removal unit 40 and / or the image generation unit may generate an appropriate image after performing demosaicing on the full pixel-binning data a1 or the first partial pixel-binning data p1 and the second partial pixel-binning data p2.

[0080] [Circuit Structure of Each Pixel in an Image Sensor]

[0081] A specific method of pixel binning of a unit pixel group as an image sensor is described below. Here, a more specific configuration and operation of a unit pixel group in an image sensor are described in detail.

[0082] Figure 5 schematically illustrates a circuit configuration related to signal flow for explaining an example of pixel binning in a 4 (2×2) pixel array. As shown in Figure 5, the 4 (2×2) pixels correspond to four photodiodes (PD1 to PD4) and are composed of a floating diffusion (FD), a source follower amplifier (SF), a reset transistor (RES), transfer transistors (TX1 to TX4), and a select transistor (SEL) connected to these diodes.

[0083] The four photodiodes (PD1-PD4) are connected to a shared floating diffusion (FD). The output of the source follower amplifier (SF) is connected via a select transistor (SEL) to a shared output line (equivalent to signal line 3 in Figure 1) in a column of two-dimensionally arranged pixel groups. Furthermore, the output is connected to a constant current source (I), a voltage gain conversion unit (not shown), and an analog-to-digital converter (ADC), which serves as the load for the source follower amplifier (SF).

[0084] Then, the digital signal (data) converted by the analog-to-digital converter (ADC) is held in the line memory 1 or the line memory 2 .

[0085] FIG. 6 is a diagram for explaining the operation of each element in the circuit configuration of 4 (2×2) pixels shown in FIG. 5 .

[0086] At time t1 , the reset transistor (RES) and the transfer transistors ( TX1 ˜ TX4 ) are turned on, and the photodiodes ( PD1 ˜ PD4 ) are reset.

[0087] Then, after a predetermined accumulation period for accumulating data has elapsed, the process of reading data from the pixels constituting the unit pixel group begins. However, first, at time t2, the reset transistor (RES) is turned off and the select transistor (SEL) is turned on. This value is then analog-to-digital converted with a predetermined voltage gain and stored in line memory 1 (FD reset noise).

[0088] At time t3, for partial pixel binning, one of the transfer transistors (TX1-TX4), for example, transfer transistors (TX1-TX2), is turned on, thereby transferring the signal from the photodiodes (PD1-PD2) to the floating diffusion (FD). This value is then converted from analog to digital using a predetermined voltage gain and stored in line memory 2 (partial pixel binning data).

[0089] At time t4, the value held in line memory 1 is subtracted from the value held in line memory 2, and the result is output and transferred to a subsequent image signal processor (ISP) or frame memory. This generates data (noise-removed, partially binned data) from which reset noise in the floating diffusion (FD) has been removed, a technique known as correlated double sampling. This corresponds to the first partially binned data p1 in Figure 4.

[0090] At time t5, to perform full-pixel binning, transfer transistors (TX1-TX4) are turned on, transferring the signal from the photodiodes (PD1-PD4) to the floating diffusion (FD). This value is then converted to digital using a predetermined voltage gain and stored in line memory 2 (full-pixel binning data).

[0091] In addition, it is assumed here that the output of the partial pixel merging data maintained in the line memory 2 is completed before the analog-to-digital conversion of the full pixel merging data is completed, but there may also be other line memories for maintaining the full pixel merging data assuming that the output of the partial pixel merging data is not completed.

[0092] Furthermore, the reset noise of the floating diffusion (FD) in full-pixel binning can be calculated using the data stored in line memory 1. Therefore, at time t6, the value stored in line memory 1 is subtracted from the value stored in line memory 2, and the result is output. This results in full-pixel binning data (noise-removed full-pixel binning data) from which the reset noise of the floating diffusion (FD) has been removed. This corresponds to the full-pixel binning data a1 in Figure 4 .

[0093] As described above, the first partial pixel-binning data p1 and the full pixel-binning data a1 are extracted from each unit pixel group of the image sensor 10 .

[0094] Furthermore, at this timing, when using a pixel binning mode as shown in Figure 3, only the signals from pixels located to the left or above the microlens shared by multiple binned pixels are collected in p1. The data obtained by subtracting p1 from a1 only collects signals from pixels located to the right or below the microlens. Based on this information, the phase difference in the image plane can be calculated for autofocus. By appropriately selecting the pixel binning mode in this way, high-frequency image processing can be performed by controlling the readout of the same data as image plane phase difference AF, while simultaneously achieving faster sensor operation and lower power consumption.

[0095] [About control methods]

[0096] Next, a control method for generating an image while removing moiré fringes using pixel binning data will be described in detail.

[0097] 7 is a flowchart illustrating a process flow of a control method M100 executed by the photographing device 100 according to the first embodiment of the present disclosure. As shown in FIG7 , the control method M100 includes steps S10 to S70 , each of which is executed by a processor included in the photographing device 100 .

[0098] In step S10, the data acquisition unit 20 acquires first partial pixel-binning data based on the first pixel group in the unit pixel group (data acquisition step). As a specific example, as shown in Figures 3 and 4, the data acquisition unit 20 performs partial pixel binning on two pixels represented by the number "1" in the unit pixel group of 4 (2×2) pixels from the image sensor 10 to read out data (first partial pixel-binning data p1).

[0099] In step S20, the analysis unit 30 analyzes the frequency characteristics of the image signal of the area composed of the unit pixel group based on the mutual correlation between the first partial pixel-binning data acquired in step S10 and the second partial pixel-binning data based on the second pixel group, wherein the second pixel group is composed of pixels in the unit pixel group other than the first pixel group (analysis step). As a specific example, as shown in Figures 3 and 4, the data acquisition unit 20 performs full pixel binning on all pixels of the unit pixel group of 4 (2×2) pixels from the image sensor 10 to read out data (full pixel binning data a1), and subtracts the first partial pixel binning data p1 from it to obtain the second partial pixel binning data p2. Then, the analysis unit 30 calculates the mutual correlation between the first partial pixel binning data p1 and the second partial pixel binning data p2, and analyzes the frequency characteristics of the image signal of the area composed of the unit pixel group.

[0100] In step S30, the analysis unit 30 determines whether the region formed by the unit pixel group is a target region for processing that contains a large number of high-frequency components and requires high-frequency component processing. Specifically, the analysis unit 30 determines whether the region formed by the unit pixel group is a target region for processing high-frequency components based on the correlation between the first partial pixel-binning data p1 and the second partial pixel-binning data p2 calculated in step S20. If the correlation is small, the region contains a large number of high-frequency components, and thus the region is determined to be a target region for processing high-frequency components ("Yes" in step S30). If the correlation is large, the region is determined not to be a target region for processing high-frequency components ("No" in step S30).

[0101] In step S40 ("Yes" in step S30), the moiré removal unit 40 generates an image while removing moiré fringes generated in the region formed by the unit pixel group (moiré removal step). Specifically, the moiré removal unit 40 generates an image while removing moiré fringes by using a low-pass filter to remove high-frequency components in the region formed by the unit pixel group, or by interpolating the high-frequency components based on image signals from other regions.

[0102] In step S50 (No in step S30 ), the image generating unit generates an appropriate image based on the full pixel combination data a1 for the region formed by the unit pixel group.

[0103] In step S60 , the automatic focus calculation unit 60 calculates the defocus amount by analyzing the spatial phase difference between the first partial pixel-binning data p1 and the second partial pixel-binning data p2 .

[0104] In step S70 , the focus adjustment unit 70 adjusts the focus of the imaging device 100 based on the defocus amount calculated in step S60 .

[0105] As described above, according to the imaging device 100 and control method M100 of the first embodiment of the present disclosure, the data acquisition unit 20 acquires first partial pixel-binning data p1 based on a first pixel group in a unit pixel group. The analysis unit 30 analyzes the frequency characteristics of the image signal in the region formed by the unit pixel group based on the correlation between the first partial pixel-binning data p1 and the second partial pixel-binning data p2. The moiré fringe removal unit 40 removes moiré fringes generated in the region formed by the unit pixel group based on the analysis results. As a result, an image can be generated while appropriately removing moiré fringes.

[0106] [Other Specific Examples of Grouping Unit Pixel Groups (Partial Pixel Binning)]

[0107] In this embodiment, as shown in FIG3 , four (2×2) pixels are grouped as a unit pixel group, and partial pixel binning is performed on two pixels in the left half or two pixels in the upper half to read out data as first partial pixel binning data. However, partial pixel binning is not limited to this. Other specific examples of partial pixel binning are described below.

[0108] (Other specific example 1)

[0109] FIG8 is a diagram illustrating another partial pixel binning method (another specific example 1) used in the image sensor 10 according to the first embodiment of the present disclosure. As shown in FIG8 , similarly to FIG3 , Bayer cells composed of green (G), red (R), blue (B), and green (G) are arranged in a matrix.

[0110] In the even-numbered row group, as indicated by numeral “1”, partial pixel binning is performed on the upper left and lower right two pixels in the unit pixel group (first pixel group) to read out data (first partial pixel binning data).

[0111] In the odd-numbered row group, as indicated by numeral “1”, partial pixel binning is performed on the two upper right and lower left pixels in the unit pixel group (first pixel group) to read out data (first partial pixel binning data).

[0112] In this way, partial pixel binning is performed on the pixels arranged on the diagonal line in the unit pixel group. The other processing is the same as the processing described using FIG.

[0113] (Other specific example 2)

[0114] FIG9 is a diagram illustrating another partial pixel binning method (another specific example 2) used in the image sensor 10 according to the first embodiment of the present disclosure. As shown in FIG9 , similarly to FIG3 , Bayer cells composed of green (G), red (R), blue (B), and green (G) are arranged in a matrix.

[0115] In the even row group and the odd row group, as indicated by number "1", partial pixel binning is performed on the three pixels at the upper right, lower right and lower left in the unit pixel group (first pixel group) to read out data (first partial pixel binning data).

[0116] In this way, partial pixel binning is performed on three pixels in the unit pixel group (four pixels). The other processing is the same as the processing explained using FIG.

[0117] In the example of FIG9 , multiple pixels are grouped asymmetrically within a unit pixel group, and partial pixel binning is performed. This allows analysis unit 30 to more appropriately analyze the frequency of the image signal in the vertical and horizontal directions within the unit pixel group based on the correlation between the first partial pixel binning data (the first pixel group represented by the number "1") and the second partial pixel binning data (the second pixel group other than the first pixel group within the unit pixel group). Specifically, analysis unit 30 can perform more appropriate analysis in relation to the presence of a high frequency component within the region formed by the unit pixel group and the occurrence of moiré fringes.

[0118] (Other specific examples 3)

[0119] FIG10 is a diagram illustrating another partial pixel binning method (another specific example 3) used in the image sensor 10 according to the first embodiment of the present disclosure. As shown in FIG10 , similarly to FIG3 , Bayer cells composed of green (G), red (R), blue (B), and green (G) are arranged in a matrix.

[0120] In the even row group, as indicated by the number "1", partial pixel merging is performed on the two pixels in the left half of the unit pixel group (first pixel group) to read out data (first partial pixel merging data), and the upper right pixel is added to the first pixel group or partially pixel merged is performed separately to read out data (additional partial pixel merging data).

[0121] In the odd row group, as indicated by the number "1", partial pixel merging is performed on the two pixels in the upper half of the unit pixel group (first pixel group) to read out data (first partial pixel merging data), and the lower left pixel is added to the first pixel group or partially pixel merged is performed separately to read out data (additional partial pixel merging data).

[0122] In this way, partial pixel merging is performed on the first pixel group in the unit pixel group, and partial pixel merging is performed on a different pixel group (the first pixel group + a single pixel or a single pixel). Then, full pixel merging is performed on the unit pixel group to read out full pixel merging data.

[0123] In the example shown in FIG10 , in order to obtain a plurality of partial pixel-binning data for an area consisting of pixel groups with different centers of gravity, it is also possible to obtain a plurality of second partial pixel-binning data by subtracting these partial pixel-binning data from the full pixel-binning data. Based on the various combinations of the first partial pixel-binning data and the second partial pixel-binning data thus obtained, the analysis unit 30 can perform more appropriate analysis with respect to whether a region consisting of unit pixel groups contains a large number of high-frequency components and whether moiré fringes are generated.

[0124] As illustrated herein, there are various ways to perform partial pixel merging, but the methods are not limited thereto. The pixels in the unit pixel group that are to be partially pixel-merged can be set regularly or randomly. In order to appropriately analyze the unit pixel group (area) containing a large number of high-frequency components and generating moiré fringes, the analysis unit 30 can set the pixels in the unit pixel group that are to be partially pixel-merged based on, for example, the type and performance of the shooting device including the lens, image sensor, etc., the subject, the surrounding environment, and other shooting conditions.

[0125] Furthermore, as described above, a unit pixel group is not limited to being composed of 4 (2×2) pixels, but may also be composed of, for example, 3 (3×1) pixels, 8 (4×2) pixels, 9 (3×3) pixels, or 16 (4×4) pixels. A Bayer unit is also not limited to being composed of 4 (2×2) unit pixel groups, but may be composed of, for example, 9 (3×3) unit pixel groups or 16 (4×4) unit pixel groups. The selection of pixels for partial pixel binning can be determined appropriately, and AI may also be used.

[0126] [Microlens configuration]

[0127] 11 to 16 are explanatory diagrams showing the arrangement relationship between microlenses and unit pixel groups.

[0128] The unit pixel groups shown in FIG. 11 to FIG. 14 correspond to the unit pixel groups shown in FIG. 3 and FIG. 8 to FIG. 10 , respectively. One microlens 50 may be arranged so that two or more pixels among the plurality of pixels in the unit pixel group share one microlens 50 .

[0129] The operating method of the imaging device having the unit pixels shown in Figures 11 to 14 is the same as the operating method of the imaging device having the unit pixels shown in Figures 3 and 8 to 10 (for example, the above-mentioned various processing performed by the data acquisition unit, analysis unit, moiré fringe removal unit, analysis unit or image generation unit).

[0130] In the examples shown in Figures 11 to 14 , all pixels within a unit pixel group share a single microlens 50. However, as shown in Figure 15 , microlenses 51 and 52 may be arranged such that two or more of the plurality of pixels within the unit pixel group share microlens 51, and the remaining two or more pixels share microlens 52. Furthermore, as shown in Figure 15 , microlens 51 may be arranged such that at least one pixel within a first pixel group and at least one pixel within a second pixel group share microlens 51, and microlens 52 may be arranged such that the remaining at least one pixel within the first pixel group and the remaining at least one pixel within the second pixel group share microlens 52.

[0131] In the example shown in FIG15 , two microlenses are arranged in a unit pixel group. However, the number of microlenses arranged in a unit pixel group may be three or more. In this case, each microlens in a unit pixel group may be shared by two or more pixels in the unit pixel group.

[0132] As shown in FIG11 , by configuring a large microlens 50 so as to cover a unit pixel group, phase difference autofocus (PDAF) can be performed based on a spatial correlation operation between the second partial pixel-binned data p2 and the first partial pixel-binned data p1, where the second partial pixel-binned data p2 is obtained by subtracting the first partial pixel-binned data p1 from the full pixel-binned data a1. PDAF data acquisition and partial pixel-binned data acquisition for high-frequency recovery are common, requiring no additional operations, enabling multifunctionality without increasing readout time and also enabling high speed. Thus, according to this embodiment, high-frequency recovery and PDAF can be performed simultaneously through an operation derived from a single sensor readout operation, enabling multifunctionality and high speed.

[0133] In the example shown in FIG15 , unlike the example shown in FIG11 , two microlenses 51 and 52 are arranged per unit pixel in the horizontal or vertical length in a manner spanning the first pixel group and the second pixel group. The high-frequency recovery and PDAF in the example shown in FIG15 are performed in the same manner as the high-frequency recovery and PDAF in the example shown in FIG11 . In the example shown in FIG15 , it is possible to achieve high resolution when “full pixel readout” is performed without pixel merging. This is because, in the example shown in FIG15 , since the number of pixels covered by each microlens is small, the aperture MTF (Modulation Transfer Function) of the microlens is higher than the aperture MTF of the microlens shown in FIG11 , and therefore the comprehensive MTF of the shooting device 100 can be improved.

[0134] As shown in FIG. 16 , each unit pixel group may include two or more micro lenses, and the micro lenses arranged in the unit pixel group may be arranged in two or more different types.

[0135] In the example shown in FIG16 , reference numerals G1 and G2 denote unit pixel groups arranged corresponding to green color filters, reference numeral R denotes a unit pixel group arranged corresponding to red color filters, and reference numeral B denotes a unit pixel group arranged corresponding to blue color filters. Furthermore, reference numerals X and Y denote two orthogonal directions, and the unit pixel groups are arranged along the X direction (row direction) and the Y direction (column direction).

[0136] Unit pixel group G1 is composed of four (2×2) pixels, denoted by the reference numerals G11, G12, G13, and G14. Two microlenses M1 and M2 are arranged within unit pixel group G1. Pixels G11 and G13 share microlens M1, while pixels G12 and G14 share microlens M2. For example, pixels G11, G12, and G13 may constitute a first pixel group, while pixel G14 may constitute a second pixel group.

[0137] The unit pixel group R is composed of four (2×2) pixels, denoted by reference numerals R1, R2, R3, and R4. Two microlenses M3 and M4 are arranged within the unit pixel group R. Pixels R1 and R3 share microlens M3, while pixels R2 and R4 share microlens M4. For example, pixels R1, R2, and R3 may constitute a first pixel group, while pixel R4 may constitute a second pixel group.

[0138] Unit pixel group B is composed of four (2×2) pixels, denoted by reference numerals B1, B2, B3, and B4. Two microlenses M5 and M6 are arranged within unit pixel group B. Pixels B1 and B2 share microlens M5, while pixels B3 and B4 share microlens M6. For example, pixels B1, B2, and B3 may constitute a first pixel group, while pixel B4 may constitute a second pixel group.

[0139] Unit pixel group G2 consists of four (2×2) pixels, denoted by G21, G22, G23, and G24. Two microlenses, M7 and M8, are located within unit pixel group G2. Pixels G21 and G22 share microlens M7, while pixels G23 and G24 share microlens M8. For example, pixels G21, G22, and G23 may constitute a first pixel group, while pixel G24 may constitute a second pixel group.

[0140] In the example shown in FIG16 , microlenses M1 to M8 have a shape that approximates an ellipse having a major axis and a minor axis. Microlenses M1 to M4 are configured with their major axes oriented in the X direction and their minor axes oriented in the Y direction. Microlenses M5 to M8 are configured with their major axes oriented in the Y direction and their minor axes oriented in the X direction. In this manner, by configuring microlenses M1 to M4 differently from microlenses M5 to M8, it is possible to calculate the correlation between first partial pixel binning data p1 and second partial pixel binning data p2 having different spatial frequency filtering characteristics based on pixels with different microlens configurations, and analyze the frequency characteristics of the image signal in the region consisting of the unit pixel group.

[0141] In addition, in order to set the configuration types of microlenses arranged in a unit pixel group to two or more, for example, the arrangement directions of microlenses of the same shape can be set to two or more, or multiple microlenses of different sizes or shapes can be arranged in a unit pixel group. In addition, in the example shown in Figure 16, the shape of the microlens is shown to be close to an ellipse, but the shape of the microlens can be any shape. The shape of the microlens can be symmetrical (for example, circular) or asymmetrical. In addition, the number of pixels shared by one microlens does not need to be the same as the number of pixels shared by another microlens, and the two can also be different.

[0142] The embodiments described above are intended to facilitate understanding of the present disclosure and are not intended to limit the interpretation of the present disclosure. The various elements and their configurations, materials, conditions, shapes, and sizes of the embodiments are not limited to those illustrated and can be appropriately modified. In addition, the structures shown in different embodiments can be partially replaced or combined with each other.

[0143] Description of reference numerals:

[0144] 1…control circuit; 2…pixel group; 3…signal line; 4…readout circuit; 5…digital signal processing unit (DSP); 10…image sensor; 20…data acquisition unit; 30…analysis unit; 40…moire fringe removal unit; 100…shooting device; M100…control method; S10 to S50…steps of control method M100.

Claims

1. A camera device comprising a unit pixel group consisting of a plurality of grouped pixels, wherein: The one microlens is arranged so that two or more pixels among the plurality of pixels in the unit pixel group share one microlens. The photographing device comprises: a data acquisition unit configured to acquire first partial pixel-binning data based on a first pixel group formed by pixel binning of at least one pixel in the unit pixel group and second partial pixel-binning data based on a second pixel group, wherein the first pixel group is formed by pixel binning of at least one pixel in the unit pixel group and the second pixel group is formed by pixels in the unit pixel group other than the first pixel group; an analyzing unit configured to analyze a frequency characteristic of an image signal of a region formed by the unit pixel group based on a correlation between the first partial pixel-binning data and the second partial pixel-binning data; as well as A moiré removal unit removes moiré fringes generated in a region formed by the unit pixel group based on the analysis result.

2. The photographing device according to claim 1, wherein: The one microlens is configured such that at least one pixel in the first pixel group and at least one pixel in the second pixel group share one microlens.

3. The photographing device according to claim 1, wherein: The moire fringe removal unit removes the moire fringe by removing a high-frequency component of the image signal using a low-pass filter.

4. The photographing device according to claim 1, wherein: The moiré removal unit removes the moiré based on an image signal of a region consisting of a unit pixel group near the unit pixel group.

5. The photographing device according to claim 1, wherein: The data acquisition unit acquires full pixel binning data based on all pixels constituting the unit pixel group. The second portion of pixel-binned data is obtained by subtracting the first portion of pixel-binned data from the full pixel-binned data.

6. The photographing device according to claim 1, wherein: The photodiodes formed corresponding to the plurality of pixels are connected to a common floating diffusion region.

7. A camera device comprising a unit pixel group consisting of a plurality of grouped pixels, wherein: The one microlens is arranged so that two or more pixels among the plurality of pixels in the unit pixel group share one microlens. The photographing device comprises: a data acquisition unit configured to acquire first partial pixel-binning data based on a first pixel group formed by pixel binning of at least one pixel in the unit pixel group and second partial pixel-binning data based on a second pixel group, wherein the first pixel group is formed by pixel binning of at least one pixel in the unit pixel group and the second pixel group is formed by pixels in the unit pixel group other than the first pixel group; an analyzing unit configured to analyze a frequency characteristic of an image signal of a region formed by the unit pixel group based on a correlation between the first partial pixel-binning data and the second partial pixel-binning data; as well as An image generating unit generates an image while restoring high-frequency components in a region constituted by the unit pixel group based on the analysis result.

8. The photographing device according to claim 7, wherein: The one microlens is configured such that at least one pixel in the first pixel group and at least one pixel in the second pixel group share one microlens.

9. A control method executed by a processor included in a camera device, the camera device including a unit pixel group consisting of a plurality of grouped pixels, wherein: The one microlens is arranged so that two or more pixels among the plurality of pixels in the unit pixel group share one microlens. The control method includes: a data acquisition step of acquiring first partial pixel-binning data based on a first pixel group and second partial pixel-binning data based on a second pixel group, wherein the first pixel group is formed by pixel merging of at least one pixel in the unit pixel group, and the second pixel group is formed by pixels in the unit pixel group other than the first pixel group; an analyzing step of analyzing a frequency characteristic of an image signal of a region formed by the unit pixel group based on a correlation between the first portion of pixel-binned data and the second portion of pixel-binned data; and The moiré fringe removal step removes the moiré fringe generated in the area constituted by the unit pixel group based on the analysis result.

10. The control method according to claim 9, wherein: The one microlens is configured such that at least one pixel in the first pixel group and at least one pixel in the second pixel group share one microlens.

Images