Imaging device and control method thereof

Abstract

A photographing apparatus (100) includes: a data acquisition unit (20) that acquires first partial binning data (p1) based on a first pixel group that is constituted by at least one pixel by binning in a unit pixel group constituted by a plurality of pixels that are grouped; an analysis unit (30) that analyzes a frequency characteristic of an image signal of a region constituted by the unit pixel group based on cross-correlation between the first partial binning data (p1) and second partial binning data (p2) based on a second pixel group constituted by pixels other than the first pixel group in the unit pixel group; and a moire removal unit (40) that removes a moire generated in the region constituted by the unit pixel group based on an analysis result.

Description

Technical Field

[0001] This disclosure relates to a camera device and its control method. Background Technology

[0002] Generally speaking, in shooting devices such as cameras, in order to improve performance such as high image quality and high functionality, various methods have been used to design image sensors such as CMOS sensors to be installed in the shooting devices.

[0003] For example, techniques for realizing high dynamic range (HDR) images by processing multiple pixels of an image sensor in groups have been disclosed (e.g., see Patent Document 1).

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: U.S. Patent Application Publication No. 2021 / 0385389. Summary of the Invention

[0007] The problem the invention aims to solve

[0008] However, the image sensor described in Patent Document 1 does not take into account the moiré fringes generated in the captured image, which may result in moiré fringes in the captured image.

[0009] Therefore, the purpose of this disclosure is to provide an imaging device and control method for appropriately removing moiré fringes.

[0010] Solution for solving the problem

[0011] One aspect of the imaging apparatus disclosed herein includes: a data acquisition unit that acquires first partial merged data based on a first pixel group, wherein the first pixel group is formed by merging at least one pixel in a unit pixel group composed of a plurality of grouped pixels; an analysis unit that analyzes the frequency characteristics of an image signal of a region composed of the unit pixel group based on the first partial merged data and second partial merged data based on the cross-correlation between the first partial merged data and a second pixel group composed of pixels other than the first pixel group in the unit pixel group; and a moiré fringe removal unit that removes moiré fringes generated in the region composed of the unit pixel group based on the analysis results.

[0012] In the above aspects, the moiré fringe removal unit can also remove the high-frequency components of the image signal by using a low-pass filter, thereby removing moiré fringes.

[0013] In the above aspects, the moiré fringe removal unit can also remove moiré fringes based on the image signal of the region composed of unit pixel groups near the unit pixel groups.

[0014] In the above aspects, the data acquisition unit may also acquire all merged data based on all pixels constituting a unit pixel group, and subtract the first part of the merged data from the all merged data to obtain the second part of the merged data.

[0015] In the above aspects, each photodiode, which corresponds to multiple pixels, can also be connected to a common floating diffusion region.

[0016] In the above aspects, the floating diffusion region can switch between multiple charge-voltage conversion gains, the data acquisition unit acquires the first part of the merged data under the low conversion gain among the multiple charge-voltage conversion gains, the analysis unit analyzes the frequency characteristics of the image signal of the region composed of unit pixel groups based on the cross-correlation between the first part of the merged data and the second part of the merged data under the low conversion gain, and the moiré fringe removal unit removes the moiré fringes generated in the region composed of unit pixel groups based on the analysis results under the low conversion gain.

[0017] In the above aspects, each of the multiple pixels may further consist of two or more sub-pixels, and the data acquisition unit acquires, based on first sub-part merged data of a first sub-pixel group consisting of at least one or more sub-pixels and second sub-part merged data of a second sub-pixel group consisting of sub-pixels other than the first sub-pixel group, the first sub-part merged data and the second sub-part merged data are used for phase difference autofocus.

[0018] In the above aspects, each of the plurality of pixels may further consist of two or more sub-pixels, any one of the plurality of pixels may contain a masking pixel that masks at least one of the two or more sub-pixels, and the data acquisition unit may acquire sub-partial merged data based on sub-pixels other than the masking pixel in the pixel containing the masking pixel, and the sub-partial merged data may be used for phase difference autofocus.

[0019] Another aspect of the imaging apparatus disclosed herein includes: a data acquisition unit that acquires first partial merged data based on a first pixel group, wherein the first pixel group is formed by merging at least one pixel in a unit pixel group composed of multiple grouped pixels; an analysis unit that analyzes the frequency characteristics of an image signal of a region composed of the unit pixel group based on the cross-correlation between the first partial merged data and second partial merged data based on a second pixel group composed of pixels other than the first pixel group in the unit pixel group; and an image generation unit that recovers the high-frequency components in the region composed of the unit pixel group and generates an image based on the analysis results.

[0020] The control method of one aspect of this disclosure is executed by a processor included in an imaging device, comprising: a data acquisition step of acquiring first partial merged data based on a first pixel group, the first pixel group being formed by merging at least one pixel in a unit pixel group composed of multiple grouped pixels; an analysis step of analyzing the frequency characteristics of an image signal in a region composed of the unit pixel group based on the cross-correlation between the first partial merged data and second partial merged data based on a second pixel group composed of pixels other than the first pixel group in the unit pixel group; and a moiré fringe removal step of removing moiré fringes generated in the region composed of the unit pixel group based on the analysis results.

[0021] Invention Effects

[0022] According to this disclosure, a suitable imaging device and control method for removing moiré fringes can be provided. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating the configuration of the image sensor 10 according to the first embodiment of this disclosure.

[0024] Figure 2 This is a diagram illustrating the merging used in the image sensor 10 of the first embodiment of this disclosure.

[0025] Figure 3 This is a diagram illustrating the partial merging used in the image sensor 10 of the first embodiment of this disclosure.

[0026] Figure 4 This is a block diagram illustrating the functions and data flow of the imaging device 100 according to the first embodiment of this disclosure.

[0027] Figure 5 It is a schematic diagram showing the circuit configuration of the signal flow used to illustrate an example of merging in 4 (2×2) pixels.

[0028] Figure 6 It is used for explanation Figure 5 The diagram shows the operation of each element of the circuit consisting of 4 (2×2) pixels.

[0029] Figure 7 This is a flowchart illustrating the processing flow of the control method M100 executed by the imaging device 100 according to the first embodiment of the present disclosure.

[0030] Figure 8 This is a diagram illustrating another part of the image sensor 10 used in the first embodiment of this disclosure (another specific example 1).

[0031] Figure 9This is a diagram illustrating another part of the image sensor 10 used in the first embodiment of this disclosure (another specific example 2).

[0032] Figure 10 This is a diagram illustrating another part of the image sensor 10 used in the first embodiment of this disclosure (another specific example 3).

[0033] Figure 11 This is a schematic diagram illustrating the circuit configuration of the signal flow, which is used to illustrate an example of combining dual conversion gain and phase difference AF of all pixel imaging surfaces in 4 (2×2) pixels.

[0034] Figure 12 It is used for explanation Figure 11 The diagram shows the operation of each element of the circuit consisting of 4 (2×2) pixels.

[0035] Figure 13 This is a schematic diagram of an image sensor that uses dedicated pixels to acquire signals for phase difference autofocus (AF). Detailed Implementation

[0036] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Furthermore, the embodiments described below are merely specific examples for implementing the present disclosure and are not intended to limit the interpretation of the disclosure. Additionally, for ease of understanding, the same reference numerals will be used as much as possible for the same constituent elements in the drawings, and sometimes repeated descriptions will be omitted.

[0037] <First Implementation>

[0038] [About image sensors]

[0039] Figure 1 This is a schematic diagram illustrating the configuration of the image sensor 10 according to the first embodiment of this disclosure. Figure 1 As shown, the image sensor 10 is typically a CMOS image sensor, etc., and includes a control circuit 1, multiple pixel groups arranged in two dimensions 2, signal lines 3, readout circuit 4, and a digital signal processing unit (DSP) 5.

[0040] Furthermore, here, pixel group 2 is made into a pixel group (unit pixel group) by grouping 4 (2×2) pixels, but it is not limited to this. For example, 3 (3×1) pixels, 8 (4×2) pixels, 9 (3×3) pixels, and 16 (4×4) pixels can also be made into a unit pixel group.

[0041] The control circuit 1 drives multiple pixel groups 2 of the image sensor 10, controls the reading of data based on the light signals accumulated in the multiple pixel groups 2, and outputs it to the outside of the image sensor 10.

[0042] Multiple pixel groups 2 are arranged in two dimensions. Based on the control signal from the control circuit 1 and the control signal generated by the pixel group 2 itself, the light signal is accumulated and brought to the image sensor 10, and is read as data (electrical signal) based on the light signal.

[0043] The electrical signals read from multiple pixel groups 2 are transmitted to the readout circuit 4 via signal lines 3 (typically column signal lines parallel to the column direction), where they are converted from analog to digital.

[0044] The digital signal processing unit (DSP) 5 processes the digital signal converted from analog to digital by the readout circuit 4. Then, the processed digital signal is transmitted via a data bus to the processor or memory of the imaging device.

[0045] Furthermore, the DSP5 is not limited to this configuration. For example, it can also be configured such that the image sensor 10 does not include the DSP5, but the subsequent processor has the DSP. Moreover, it can also be configured such that the digital signal processing in image processing is handled by both the DSP5 of the image sensor 10 and the DSP included in the subsequent processor. In other words, the location of the DSP in this disclosure is not limited to a specified location.

[0046] [Regarding the merger]

[0047] Figure 2 This is a diagram illustrating the merging used in the image sensor 10 of the first embodiment of this disclosure. Figure 2 As an example, in a single-board Bayer array color pixel configuration, each color consists of 4 (2×2) pixels.

[0048] If each pixel is treated as an independent pixel and data from each pixel is read in, high-resolution images based on a high sampling frequency can be obtained. On the other hand, such as Figure 2 As shown, by merging four pixels into a pixel group (unit pixel group) and reading data from these four pixels, it is possible to achieve high SNR based on high signal electron count, high sensitivity based on wide pixel size, high frame rate based on a small number of pixels, and low power consumption based on low readout.

[0049] In other words, resolution and other properties are compromised depending on the merging process. Specifically, when each pixel is treated as an independent pixel and data from all pixels is read, the sampling frequency for this readout is set to fs (full readout mode). In contrast, when four pixels are merged into a single pixel group (unit pixel group) and data from these four pixels is read, the sampling frequency for this readout decreases to fs / 2 (binning mode).

[0050] Figure 3 This is a diagram illustrating the partial merging used in the image sensor 10 of the first embodiment of this disclosure. Figure 3 In this example, a Bayer arrangement consisting of green (G), red (R), blue (B) and green (G) is considered as a Bayer unit, which is configured as a matrix.

[0051] Furthermore, here, a Bayer unit consists of four (2×2) unit pixel groups of G, R, B, G, but is not limited to this. For example, it can also consist of nine (3×3) unit pixel groups or sixteen (4×4) unit pixel groups.

[0052] In even-numbered row groups, as shown by the number "1", for example, in a unit pixel group of 4 (2×2) pixels consisting of G, the two pixels in the left half (first pixel group) are partially merged, and the data is read (first part merged data). Next, all 4 (2×2) pixels consisting of the above G are merged, and the data is read (all merged data).

[0053] In odd-numbered rows, as shown by the number "1", for example, in a unit pixel group of 4 (2×2) pixels consisting of G, the two pixels in the upper half (first pixel group) are merged, and the data is read (first part merged data). Next, all 4 (2×2) pixels consisting of the above G are merged, and the data is read (all merged data).

[0054] In addition, in a unit pixel group of 4 (2×2) pixels consisting of G, the data of the right half and the lower half can be generated based on the difference between the full merged data read in the full merge and the partial merged data read in the partial merge (second pixel group, second part merged data).

[0055] Furthermore, this example specifically illustrates a portion of a unit pixel group consisting of 4 (2×2) pixels composed of G, but the same processing is applied to other unit pixel groups consisting of 4 (2×2) pixels composed of G, 4 (2×2) pixels composed of R, and 4 (2×2) pixels composed of B.

[0056] [Regarding the removal of moiré stripes]

[0057] The following describes the process of removing moiré fringes (aliasing) using partially merged data and fully merged data output from image sensor 10. Furthermore, while moiré fringes are a type of noise generated in an image, the moiré fringing removal described in this specification also includes the removal of similar aliasing, which can naturally also be processed.

[0058] Figure 4 This is a block diagram illustrating the functions and data flow of the imaging apparatus 100 according to the first embodiment of this disclosure. For example... Figure 4 As shown, the imaging device 100 includes an image sensor 10, a data acquisition unit 20, an analysis unit 30, and a moiré fringe removal unit 40. Furthermore, although optical systems, memory, etc., are not shown here and detailed descriptions are omitted, the functions or components generally found in imaging devices are also present in the imaging device 100. In addition, the imaging device disclosed herein is applicable to digital cameras and terminals such as smartphones, tablets, and laptops equipped with camera functions.

[0059] Image sensor 10 uses the above-mentioned Figures 1-3 The image sensor described, such as Figure 4 As shown, the first part of the merged data p1 and the entire merged data a1 are read from the image sensor 10.

[0060] Here, the entire merged data a1, the first part of the merged data p1, and the second part of the merged data p2 can be used, for example, as described above. Figure 3 The process described is for reading and generating.

[0061] Specifically, the first part of the merged data p1 is based on data from a unit pixel group consisting of multiple grouped pixels, obtained by merging data from a first pixel group consisting of at least one pixel. Figure 3 In the first part, the merged data p1 is equivalent to the data read from the two pixels represented by the number "1" in the unit pixel group.

[0062] The merged data a1 is based on the data of all pixels in a unit pixel group consisting of multiple grouped pixels. Figure 3 In this context, the merged data a1 is equivalent to the data read from 4 (2×2) pixels in the unit pixel group.

[0063] Then, the first part of the merged data p1 is subtracted from the total merged data a1, thereby generating the second part of the merged data p2 based on the difference.

[0064] The analysis unit 30 analyzes the frequency characteristics of the image signal of the region composed of the unit pixel group based on the cross-correlation between the first part of the merged data p1 and the second part of the merged data p2.

[0065] For example, the analysis unit 30 calculates the cross-correlation between the first part of merged data p1 and the second part of merged data p2. If the cross-correlation is small (below a specified threshold), it determines that the region composed of the unit pixel group contains more high-frequency components.

[0066] Moreover, since moiré fringes are highly likely to be generated periodically, the analysis unit 30 can estimate in the image sensor 10 which region is composed of which group of unit pixels has generated moiré fringes.

[0067] For example, in Figure 3 In the example shown, in even-numbered row groups, the analysis unit 30 calculates the cross-correlation between the first portion of merged data based on two pixels in the left half of the unit pixel group and the second portion of merged data based on two pixels in the right half. Similarly, in odd-numbered row groups, the analysis unit 30 calculates the cross-correlation between the first portion of merged data based on two pixels in the upper half of the unit pixel group and the second portion of merged data based on two pixels in the lower half. That is, while the analysis unit 30 analyzes the frequency characteristics of the image signal in both the vertical and horizontal directions within the unit pixel group, it alternately sets merge groups for each row in both directions. Therefore, considering the generation of moiré fringes, it is also possible to determine which region (unit pixel group) generates moiré fringes, but as described above, by assuming that the moiré fringes are generated periodically (with a specified length and period) in a fringe pattern, it is possible to estimate which region (unit pixel group) generates moiré fringes.

[0068] Furthermore, the threshold used to calculate the cross-correlation between the first part of the merged data p1 and the second part of the merged data p2, and to determine the threshold containing more high-frequency components, can be preset or changed based on the type and performance of the imaging device including lenses or image sensors, the subject or surrounding environment, and other shooting conditions. Alternatively, an appropriate threshold can be set using AI (Artificial Intelligence) learning. Moreover, for example, the first part of the merged data p1, the second part of the merged data p2, and all the merged data a1 can be used as supervisory data, and AI can be used to determine whether moiré fringes have been generated.

[0069] Thus, the analysis method of the analysis unit 30 is not particularly limited, and various analysis methods can be used to analyze the frequency characteristics of the image signal of the region composed of unit pixel groups, detect regions containing more high-frequency components and regions that produce moiré fringes, etc.

[0070] Based on the analysis results of the analysis unit 30, the moiré stripe removal unit 40 removes the moiré stripes generated in the region composed of unit pixel groups.

[0071] For example, the moiré stripe removal unit 40 can remove high-frequency components of the image signal (e.g., all merged data a1 of the unit pixel group) in areas containing more high-frequency components using a low-pass filter.

[0072] Alternatively, the moiré fringe removal unit 40 can also remove moiré fringes based on image signals from areas near the moiré fringe-generating region that do not produce moiré fringes (e.g., all merged data a1 of another unit pixel group). Here, the area near the moiré fringe-generating region that does not produce moiré fringes refers to, for example, an area located adjacent to (up, down, left, right, or diagonally) the moiré fringe-generating region and surrounding it, that does not produce moiré fringes. That is, the moiré fringe removal unit 40 interpolates the moiré fringe-generating region based on the image signal of another region, thereby generating an image without moiré fringes.

[0073] Furthermore, instead of the moiré fringe removal unit 40, an image generation unit (not shown) that appropriately recovers high-frequency components may be provided, or additionally provided, for regions (unit pixel groups) determined by the analysis unit 30 to contain high-frequency components and require processing of these high-frequency components. Typically, the image generation unit can interpolate the regions (unit pixel groups) determined to require image processing based on image signals from the surrounding areas, thereby appropriately recovering the high-frequency components. Moreover, AI can be used to appropriately recover the high-frequency components.

[0074] In addition, Figure 4 In the example shown, the data acquisition unit 20 obtains the second merged data p2 by subtracting the first merged data p1 from the total merged data a1, and the analysis unit 30 calculates the cross-correlation between the first merged data and the second merged data p2, but is not limited thereto. For example, the analysis unit 30 may also analyze the frequency characteristics of the image signal of the region composed of unit pixel groups based on the first merged data p1 and the total merged data a1, and based on the cross-correlation between the first merged data p1 and the second merged data p2.

[0075] The data acquisition unit 20 reads first partial merged data p1 and all merged data a1 from the image sensor 10, but is not limited thereto. For example, it may also read first partial merged data p1 and second partial merged data p2. In this case, the analysis unit 30 can analyze the frequency characteristics of the image signal of the region composed of the unit pixel group based on the cross-correlation between the first partial merged data p1 and the second partial merged data p2 read from the image sensor 10.

[0076] In addition, based on the analysis results of the analysis unit 30, the moiré fringe removal unit 40 and / or the image generation unit typically generate an appropriate image based on all merged data a1 for regions containing more high-frequency components, including removing moiré fringes, but may also generate an image based on the first part of merged data p1 and the second part of merged data p2.

[0077] Furthermore, the moiré stripe removal unit 40 and / or the image generation unit can generate an appropriate image after performing demosaic processing on all merged data a1 or the first part of merged data p1 and the second part of merged data p2.

[0078] [The circuit structure of each pixel in an image sensor]

[0079] As an image sensor, the specific method for merging unit pixel groups will be explained. Here, the specific structure and operation of unit pixel groups in an image sensor will be explained in further detail.

[0080] Figure 5 This is a schematic diagram illustrating the circuit configuration for the signal flow used to illustrate an example of merging 4 (2×2) pixels. (See diagram for example.) Figure 5 As shown, the 4 (2×2) pixels correspond to 4 photodiodes (PD1~PD4), which are composed of a floating diffusion region (FD), a source follower amplifier (SF), a reset transistor (RES), a transfer transistor (TX1~TX4), and a select transistor (SEL) connected to them.

[0081] Four photodiodes (PD1–PD4) are connected to a common floating diffusion region (FD). The output of the source follower amplifier (SF) is connected to a common output line (equivalent to) on a column with multiple pixel groups arranged in a two-dimensional configuration via a select transistor (SEL). Figure 1 The signal line 3), and also the constant current source (I) connected as the load of the source follower amplifier (SF), the voltage gain conversion unit (not shown), and the analog-to-digital converter (ADC).

[0082] Furthermore, the digital signal (data) converted by the analog-to-digital converter (ADC) is stored in row memory 1 or row memory 2.

[0083] Figure 6 It is used for explanation Figure 5 The diagram shows the operation of each element of the circuit consisting of 4 (2×2) pixels.

[0084] At time t1, the reset transistor (RES) and the transfer transistors (TX1 to TX4) are turned on, and the photodiodes (PD1 to PD4) are reset.

[0085] After a specified accumulation period for data accumulation, the process of reading data from the pixels constituting the unit pixel group begins. First, at time t2, the reset transistor (RES) is turned off, and the selection transistor (SEL) is turned on. Then, the value is converted from analog to digital with a specified voltage gain and stored in row memory 1 (FD reset noise).

[0086] At time t3, for partial merging, in the transmission transistors (TX1-TX4), for example, the transmission transistors (TX1-TX2) are turned on, thereby transmitting the signal from the photodiodes (PD1-PD2) to the floating diffusion region (FD). Then, its value is converted from analog to digital with a specified voltage gain and stored in the row memory 2 (partially merged data).

[0087] At time t4, the value held in line memory 1 is subtracted from the value held in line memory 2, the result is output, and transmitted to the subsequent image signal processor (ISP) or frame memory. This allows the acquisition of reset noise-removed data (noise removal / partial merging data), known as correlated double sampling, after removing the floating diffusion region (FD). This is equivalent to... Figure 4 The first part is the merged data p1.

[0088] At time t5, in order to merge all the data, the signals from photodiodes (PD1 to PD4) are transmitted to the floating diffusion region (FD) by turning on the transmission transistors (TX1 to TX4). Then, the values ​​are converted from analog to digital with a specified voltage gain and stored in row memory 2 (all merged data).

[0089] Furthermore, here, it is assumed that the output of the partial merged data held in the row memory 2 is completed before the analog-to-digital conversion of all merged data is finished. However, if the output of the partial merged data has not yet been completed, it is preferable to have another row memory for holding all merged data.

[0090] Furthermore, since the reset noise of the fully merged floating diffusion region (FD) can be obtained using the data held in row memory 1, at time t6, the value held in row memory 1 is subtracted from the value held in row memory 2, and the result is output. Thus, it is possible to obtain all merged data with the reset noise of the floating diffusion region (FD) removed (noise removal • fully merged data). This is equivalent to... Figure 4 All merged data a1.

[0091] In this way, the first part of the merged data p1 and the entire merged data a1 are extracted from each unit pixel group of the image sensor 10.

[0092] [Regarding control methods]

[0093] Next, a detailed explanation of the control method for removing moiré fringes and generating an image using merged data is provided.

[0094] Figure 7 This is a flowchart illustrating the processing flow of the control method M100 executed by the imaging device 100 according to the first embodiment of this disclosure. Figure 7 As shown, the control method M100 includes steps S10 to S50, each step being executed by a processor included in the imaging device 100.

[0095] In step S10, the data acquisition unit 20 acquires a first portion of merged data based on the first pixel group in the unit pixel group (data acquisition step). For example, as shown... Figure 3 and Figure 4 As shown, the data acquisition unit 20 merges two pixel portions represented by the number "1" in a unit pixel group of 4 (2×2) pixels from the image sensor 10 and reads the data (first part merged data p1).

[0096] In step S20, the analysis unit 30 analyzes the frequency characteristics of the image signal of the region composed of the unit pixel group based on the cross-correlation between the first portion of merged data obtained in step S10 and the second portion of merged data based on the second pixel group composed of pixels other than the first pixel group in the unit pixel group (analysis step). For example, such as... Figure 3 as well as Figure 4 As shown, the data acquisition unit 20 merges all pixels of a unit pixel group of 4 (2×2) pixels from the image sensor 10 and reads the data (complete merged data a1). By subtracting the first part of the merged data p1, the second part of the merged data p2 is obtained. Next, the analysis unit 30 calculates the cross-correlation between the first part of the merged data p1 and the second part of the merged data p2, and analyzes the frequency characteristics of the image signal of the region composed of the unit pixel group.

[0097] In step S30, the analysis unit 30 determines whether the region composed of unit pixel groups is a processing target region containing a large number of high-frequency components and requiring processing of these high-frequency components. Specifically, the analysis unit 30 determines whether the region composed of the unit pixel groups is a processing target region for high-frequency components based on the cross-correlation between the first part of the merged data p1 and the second part of the merged data p2 calculated in step S20. If the cross-correlation is small, the region is determined to be a processing target region for high-frequency components because it contains a large number of high-frequency components (Yes in step S30); if the cross-correlation is large, the region is determined not to be a processing target region for high-frequency components (No in step S30).

[0098] In step S40 (which is the case in step S30), the moiré fringe removal unit 40 removes the moiré fringes generated in the region composed of unit pixel groups and generates an image simultaneously (moiré fringe removal step). Specifically, the moiré fringe removal unit 40 removes high-frequency components from the region composed of the unit pixel groups by using a low-pass filter or by interpolating based on an image signal from another region, thereby removing the moiré fringes and generating an image simultaneously.

[0099] In step S50 (the opposite of step S30), the image generation unit generates an appropriate image for the region composed of the unit pixel group based on all merged data a1.

[0100] As described above, according to the imaging apparatus 100 and control method M100 of the first embodiment of this disclosure, the data acquisition unit 20 acquires first partial merged data p1 based on the first pixel group in the unit pixel group, the analysis unit 30 analyzes the frequency characteristics of the image signal of the region composed of the unit pixel group based on the cross-correlation between the first partial merged data p1 and the second partial merged data p2, and the moiré fringe removal unit 40 removes the moiré fringes generated in the region composed of the unit pixel group based on the analysis results. As a result, an image can be generated while appropriately removing moiré fringes.

[0101] [Another specific example of grouping (partial merging) unit pixel groups]

[0102] In this embodiment, such as Figure 3 As shown, 4 (2×2) pixels are grouped into unit pixel groups. Two pixels from the left half or two pixels from the top half are partially merged, and the data is read as the first part of the merged data. However, partial merging is not limited to this. The following describes another specific example of partial merging.

[0103] (Another specific example 1)

[0104] Figure 8 This is a diagram illustrating another portion of the image sensor 10 used in the first embodiment of this disclosure (another specific example 1). As shown... Figure 8 As shown, with Figure 3 Similarly, the Bayer units, consisting of green (G), red (R), blue (B) and green (G), are configured in a matrix.

[0105] In the even-numbered row group, as shown by the number "1", in the unit pixel group, the two pixels at the top left and bottom right (the first pixel group) are partially merged, and the data is read (the first part of the merged data).

[0106] In the odd-numbered row group, as shown by the number "1", in the unit pixel group, the two pixels in the upper right and lower left (the first pixel group) are partially merged and the data is read (the first part of the merged data).

[0107] In this way, within a unit pixel group, pixels positioned diagonally are partially merged. Other aspects related to usage... Figure 3 The processing described is the same.

[0108] (Another specific example 2)

[0109] Figure 9 This is a diagram illustrating another portion of the image sensor 10 used in the first embodiment of this disclosure (another specific example 2). As shown... Figure 9 As shown, with Figure 3 Similarly, the Bayer units, consisting of green (G), red (R), blue (B) and green (G), are configured in a matrix.

[0110] In even-numbered and odd-numbered row groups, as shown by the number "1", the top right, bottom right, and bottom left 3-pixel portions (first pixel group) in the unit pixel group are merged and the data is read (first part merged data).

[0111] In this way, within a unit pixel group (4 pixels), three pixels are partially merged. Other uses... Figure 3 The processing described is the same.

[0112] exist Figure 9 In the example, multiple pixels are grouped asymmetrically within a unit pixel group and partially merged. Therefore, based on the cross-correlation between the first merged data (the first pixel group represented by the number "1") and the second merged data (the second pixel group in the unit pixel group excluding the first pixel group), the analysis unit 30 can more appropriately analyze the frequency of the image signal in the vertical and horizontal directions within that unit pixel group. That is, the analysis unit 30 can more appropriately analyze whether the region formed by that unit pixel group contains more high-frequency components and exhibits moiré fringes.

[0113] (Another specific example 3)

[0114] Figure 10This is a diagram illustrating another portion of the image sensor 10 used in the first embodiment of this disclosure (another specific example 3). As shown... Figure 10 As shown, with Figure 3 Similarly, the Bayer units, consisting of green (G), red (R), blue (B) and green (G), are configured in a matrix.

[0115] In even-numbered row groups, as shown by the number "1", in the unit pixel group, the two pixels in the left half (first pixel group) are partially merged and the data is read (first part merged data), and the upper right pixel is added to the first pixel group, or partially merged separately and the data is read (append part merged data).

[0116] In the odd-numbered row group, as shown by the number "1", in the unit pixel group, the two pixels in the upper half (first pixel group) are partially merged and the data is read (first part merged data), and the lower left pixel is added to the first pixel group, or partially merged separately and the data is read (append part merged data).

[0117] In this way, within a unit pixel group, the first pixel group is partially merged, and then different pixel groups (the first pixel group plus another pixel or a single other pixel) are partially merged. Next, all unit pixel groups are merged, and all merged data is read.

[0118] exist Figure 10 In the example, since partial merged data is acquired from regions composed of multiple pixel groups with different centroids, multiple second-part merged data can also be acquired by subtracting the aforementioned partial merged data from the total merged data. Based on the first and second-part merged data obtained through various combinations, the analysis unit 30 can more appropriately analyze whether the region composed of the unit pixel group contains more high-frequency components and whether moiré fringes are generated.

[0119] As shown in this article, there are various methods for partial merging, but not limited to these. The pixels partially merged in a unit pixel group can be set regularly or randomly. For example, the analysis unit 30 can set the pixels partially merged in the unit pixel group according to the type and performance of the imaging device including the lens, image sensor, etc., the subject, the surrounding environment, and other shooting conditions, so as to be able to appropriately analyze unit pixel groups (regions) containing more high-frequency components and producing moiré fringes.

[0120] Furthermore, as mentioned above, a unit pixel group is not limited to consisting of 4 (2×2) pixels. For example, it can also consist of 3 (3×1) pixels, 8 (4×2) pixels, 9 (3×3) pixels, and 16 (4×4) pixels. Similarly, a Bayer unit is not limited to consisting of 4 (2×2) unit pixel groups. For example, it can also consist of 9 (3×3) unit pixel groups and 16 (4×4) unit pixel groups. The method for defining the partially merged pixels can be appropriately determined, or it can be determined using AI.

[0121] <Second Implementation>

[0122] Next, as an image sensor according to the second embodiment of this disclosure, a specific method for operating the combined dual conversion gain (DCG) and all pixel imaging plane phase difference AF (autofocus) will be described. The basic configuration of the image sensor in this embodiment is the same as that of the image sensor 10 in the first embodiment, and the pixel merging also utilizes the same method as in the first embodiment. Here, the specific configuration and operation of operating the combined dual conversion gain and all pixel imaging plane phase difference AF in the image sensor will be described in detail.

[0123] Figure 11 This is a schematic diagram illustrating the circuit configuration related to the signal flow, used to explain an example of combining dual conversion gain and all pixel-level phase difference AF in 4 (2×2) pixels. For example... Figure 11 As shown here, Figure 5 Each of the four photodiodes (PD1 to PD4) shown is divided into two for use in phase difference autofocus (AF) of all pixel imaging surfaces, becoming sub-photodiodes (PD1L / PD1R to PD4L / PD4R). The transmission transistors (TX1 to TX4) are corresponding to the sub-photodiodes and become transmission transistors (TX1L / TX1R to TX4L / TX4R) (L: left, R: right).

[0124] Furthermore, the circuit is equipped with a floating diffusion region (FD), a source follower amplifier (SF), a reset transistor (RES), a switching transistor (X), and a selection transistor (SEL).

[0125] Furthermore, to achieve dual conversion gain, an additional load capacitor is added to the pixel that can be electrically switched by a switching transistor (X). By increasing the load capacitance of the floating diffusion region (FD), the charge-voltage conversion gain can be set to be smaller; by decreasing the load capacitance, the charge-voltage conversion gain can be set to be larger.

[0126] Figure 12 It is used for explanation Figure 11 The diagram shows the operation of each element in a circuit consisting of 4 (2×2) pixels. Furthermore, each of the 4 (2×2) pixels is composed of two sub-pixels (L: left, R: right). While each pixel is composed of two sub-pixels, this is not a limitation; for example, it may be composed of three or more sub-pixels.

[0127] At time t1, the reset transistor (RES), the switching transistor (X), and the transmission transistors (TX1L / TX1R~TX4L / TX4R) are turned on, and the secondary photodiodes (PD1L / PD1R~PD4L / PD4R) are reset.

[0128] After a specified accumulation period for data accumulation, the process of reading data from the pixels constituting the unit pixel group begins. First, at time t2, the reset transistor (RES) is turned off, and the switching transistor (X) and the selection transistor (SEL) are turned on. Next, in the state where the charge-voltage conversion gain of the floating diffusion region (FD) decreases (LCG), the FD reset noise is converted from analog to digital and stored in row memory 1 (LCG / FD reset noise).

[0129] At time t3, the switching transistor (X) is turned off. Under the state where the charge-voltage conversion gain of the floating diffusion region (FD) increases (HCG), the FD reset noise is converted by AD and stored in row memory 2 (HCG / FD reset noise).

[0130] At time t4, the transmission transistors (TX1L) and (TX2L) are turned on to acquire the left part of the merged data for phase difference AF of the shooting plane in HCG state, perform AD conversion, and save it to the line memory 3 (HCG / phase difference AF L part merged data).

[0131] Furthermore, by subtracting the HCG•FD reset noise stored in the row memory 2 from the HCG•phase difference AF partial merged data stored in the row memory 3, it is possible to obtain the HCG state phase difference AF partial merged data (noise removal•HCG•phase difference AF partial merged data) with the reset noise removed.

[0132] At time t5, the transmission transistors (TX1L•TX1R) and (TX2L•TX2R) are turned on to acquire the partial merged data in the HCG state, perform AD conversion, and save it to the row memory 3 (HCG•partial merged data).

[0133] By subtracting the HCG / FD reset noise stored in row memory 2 from the HCG / partial merged data stored in row memory 3, it is possible to obtain the partial merged data (noise removal / HCG / partial merged data) of the HCG state after removing the reset noise.

[0134] In addition, by subtracting the noise removal •HCG • phase difference AF (L-partially merged data) from the noise removal •HCG • phase difference AF (R-partially merged data), it is possible to obtain the noise removal •HCG • phase difference AF (R-partially merged data).

[0135] At time t6, the switching transistor (X) is turned on. In the state where the charge-voltage conversion gain of the floating diffusion region (FD) is small (LCG), the transfer transistors (TX1L•TX1R) and (TX2L•TX2R) are turned on to acquire the partial merged data in the LCG state, perform AD conversion, and save it to the row memory 3 (LCG•partial merged data).

[0136] Next, by subtracting the LCG FD reset noise stored in row memory 1 from the LCG partial merged data stored in row memory 3, it is possible to obtain the partial merged data in the LCG state after removing the reset noise (noise removal • LCG • partial merged data).

[0137] At time t7, the transmission transistors (TX1L•TX1R~TX4L•TX4R) are turned on to acquire all the merged data in the LCG state, perform AD conversion, and save it to the row memory 3 (LCG•all merged data).

[0138] At time t8, by subtracting the LCG•FD reset noise stored in row memory 1 from the LCG•all merged data stored in row memory 3, it is possible to obtain the LCG•all merged data with the reset noise removed (noise removal•LCG•all merged data).

[0139] Thus, in HCG state, the phase difference AF partial merged data (L portion) and partial merged data (equivalent to the first merged data p1) are extracted from the image sensor 10. In LCG state, the partial merged data (equivalent to the first merged data p1) and all merged data (equivalent to all merged data a1) are extracted. Furthermore, as described above, in HCG state, the phase difference AF partial merged data (R portion) can be obtained through calculation.

[0140] As described above, according to the imaging apparatus and control method equipped with the image sensor of the second embodiment of this disclosure, in the LCG state, partial merged data (equivalent to the first partial merged data p1) and all merged data (equivalent to all merged data a1) are extracted. Therefore, similar to the first embodiment of this disclosure, an image can be generated while appropriately removing moiré fringes. By appropriately removing moiré fringes from the high SNR data in the LCG state, the generation of moiré fringes in fine images can be suppressed.

[0141] Furthermore, in this embodiment, not all merged data is read in the HCG state. However, if the transmission transistors (TX1L•TX1R~TX4L•TX4R) are turned on in the HCG state, all merged data in the HCG state is acquired and AD conversion is performed, then all merged data in the HCG state can also be acquired. Since the transistor switching or AD conversion process places a load on the processor mounted on the imaging device 100, by reducing the number of transistor switching or AD conversions, the increase in load and power consumption applied to the processor mounted on the imaging device 100 can be suppressed.

[0142] Furthermore, in this embodiment, it is possible to acquire partially merged phase difference AF data under HCG conditions. Phase difference AF data requires high SNR, and since it is possible to acquire data under noise-resistant HCG conditions, it is very effective for this purpose.

[0143] Furthermore, in this embodiment, in the LCG state, it is impossible to obtain merged data for phase difference AF, but by setting a dedicated pixel in the image sensor and masking a portion of that pixel, or by using a (2×1) on-chip microlens structure, the signal for phase difference AF can also be obtained.

[0144] Figure 13 This is a schematic diagram illustrating an image sensor with dedicated pixels configured to acquire signals for phase difference autofocus (AF). For example... Figure 13 As shown, a dedicated pixel is set among the multiple pixels arranged in the image sensor. This dedicated pixel, for example, has its left half (L region) or right half (R region) masked. Furthermore, here, the dedicated pixel is divided into two parts, left and right, but is not limited to this. For example, it can be divided into two or more parts, vertically, for masking, so as to be able to properly acquire the phase signal for phase difference AF.

[0145] In a dedicated pixel, under LCG state, if the phase difference signal is acquired optically in an unmasked area, LCG phase difference AF data can be obtained.

[0146] The embodiments described above are for the purpose of understanding this disclosure and are not intended to limit the interpretation of this disclosure. The elements, their configurations, materials, conditions, shapes, and dimensions included in the embodiments are not limited to the illustrated elements and can be appropriately modified. In addition, the configurations shown in different embodiments can be partially replaced or combined.

[0147] Explanation of reference numerals in the attached figures:

[0148] 1…control circuit, 2…pixel group, 3…signal line, 4…reading circuit, 5…digital signal processing unit (DSP), 10…image sensor, 20…data acquisition unit, 30…analysis unit, 40…moiré fringe removal unit, 100…imaging device, M100…control method, S10~S50…each step of control method M100.

Claims

1. A shooting device, characterized in that, have: The data acquisition unit acquires a first portion of merged data based on a first pixel group, wherein the first pixel group is composed of at least one pixel through merging within a unit pixel group consisting of multiple grouped pixels. The analysis unit analyzes the frequency characteristics of the image signal of the region composed of the unit pixel group based on the cross-correlation between the first portion of merged data and the second portion of merged data based on the second pixel group composed of pixels other than the first pixel group in the unit pixel group. as well as The moiré fringe removal unit removes moiré fringes generated on the image based on the analysis results of the analysis unit, wherein the image is generated based on the image signal of the region composed of the unit pixel group.

2. The shooting device according to claim 1, wherein, The moiré stripe removal unit removes the high-frequency components of the image signal using a low-pass filter, thereby removing the moiré stripes.

3. The shooting device according to claim 1, wherein, The moiré stripe removal unit removes the moiré stripes based on the image signal of the region consisting of unit pixel groups near the unit pixel groups.

4. The shooting device according to claim 1, wherein, The data acquisition unit acquires all merged data based on all pixels constituting the unit pixel group. The second part of the merged data is obtained by subtracting the first part of the merged data from the total merged data.

5. The shooting device according to claim 1, wherein, Each photodiode, corresponding to the plurality of pixels, is connected to a common floating diffusion region.

6. The shooting device according to claim 5, wherein, The floating diffusion region can switch between multiple charge-voltage conversion gains. The data acquisition unit acquires the first portion of the merged data at the low conversion gain among the multiple charge-voltage conversion gains. The analysis unit analyzes the frequency characteristics of the image signal of the region composed of the unit pixel groups based on the cross-correlation between the first portion of the merged data and the second portion of the merged data under the low conversion gain. The moiré stripe removal unit removes moiré stripes generated in the region composed of the unit pixel group based on the analysis results of the analysis unit under the low conversion gain.

7. The imaging device according to claim 1, wherein, Each of the plurality of pixels is further composed of two or more sub-pixels. The data acquisition unit acquires, based on first sub-part merged data of a first sub-pixel group consisting of at least one or more sub-pixels, and second sub-part merged data of a second sub-pixel group consisting of sub-pixels other than the first sub-pixel group, from the two or more sub-pixels. The first sub-part merged data and the second sub-part merged data are used for phase difference autofocus.

8. The shooting device according to claim 1, wherein, Each of the plurality of pixels is further composed of two or more sub-pixels. Any one of the plurality of pixels includes a masking pixel that masks at least one of two or more sub-pixels. The data acquisition unit acquires sub-partial merged data based on sub-pixels other than the occluded pixels in the pixels containing the occluded pixels. The merged sub-data is used for phase difference autofocus.

9. A shooting device, characterized in that, have: The data acquisition unit acquires a first portion of merged data based on a first pixel group, wherein the first pixel group is composed of at least one pixel through merging within a unit pixel group consisting of multiple grouped pixels. The analysis unit analyzes the frequency characteristics of the image signal of the region composed of the unit pixel group based on the cross-correlation between the first portion of merged data and the second portion of merged data based on the second pixel group composed of pixels other than the first pixel group in the unit pixel group. as well as The image generation unit recovers the high-frequency components in the region composed of the unit pixel groups and generates an image based on the analysis results of the analysis unit.

10. A control method executed by a processor included in a shooting device, characterized in that, include: The data acquisition step acquires a first portion of merged data based on a first pixel group, wherein the first pixel group is composed of at least one pixel through merging within a unit pixel group consisting of multiple grouped pixels. The analysis step analyzes the frequency characteristics of the image signal of the region composed of the unit pixel group based on the cross-correlation between the first portion of merged data and the second portion of merged data based on the second pixel group composed of pixels other than the first pixel group in the unit pixel group. as well as The moiré stripe removal step removes moiré stripes generated in the region composed of the unit pixel group, based on the analysis results of the analysis step.

11. A terminal, characterized in that, Equipped with a shooting device according to any one of claims 1 to 9.

Images