Imaging device and imaging method

Abstract

This method suppresses the bias of the pixel centroid after binning compared to conventional methods. [Solution] The binning processing unit 34 performs k × k (where k is an integer) binning. Furthermore, the binning processing unit 34 can perform uniform binning. In uniform binning, signals from at least one pixel of each color located relatively towards the center of the pixel group are excluded from processing.

Description

Technical Field

[0001] This specification discloses an imaging device and an imaging method.

Background Art

[0002] For example, as exemplified in Non-Patent Documents 1 and 2, binning processing is known as an image processing method. In binning processing, the values of a plurality of pixels of an image sensor are added or averaged. For example, the values of pixels of the same color are added or averaged, respectively.

[0003] By combining a plurality of pixels into one, the resolution of the image decreases. On the other hand, the brightness increases due to the addition of pixel values. Also, noise is reduced by averaging. That is, by capturing a plurality of pixels as one large pixel, the light receiving area is expanded and the sensitivity is improved. Based on such merits, for example, when the imaging environment is dark, binning processing is executed.

[0004] FIG. 23 and FIG. 24 illustrate binning processing according to the prior art. FIG. 23 illustrates binning processing of Gb pixels and B pixels. Also, FIG. 25 illustrates binning processing of R pixels and Gr pixels. Note that Gb pixels refer to pixels arranged under a Gb color filter. Similarly, B pixels, R pixels, and Gr pixels refer to pixels under a B filter, an R filter, and a Gr filter, respectively.

[0005] FIG. 23 and FIG. 24 illustrate 2×2 binning processing. The values of four adjacent pixels of the same color are added or averaged. Here, the pixel value after binning processing is the value at the centroid (geometric centroid) of the four pixels on the light receiving surface, as indicated by a white circle.

Prior Art Documents

Non-Patent Documents

[0006]

Non-Patent Document 1

[0007] Figure 25 conceptually illustrates the pixel arrangement after binning. Theoretically, the 2x2 binning process makes each pixel size equivalent to that of pixels in Figures 23 and 24, which are twice as large vertically and horizontally. In Figure 25, the centroid (geometric center) of each expanded pixel is indicated by a white circle.

[0008] For example, in a pixel group consisting of pixels R, Gb, Gr, and B, the distances d1 and d3 between centroids within the group are shorter than the distances d2 and d4 between adjacent pixel groups. This non-uniform arrangement of centroids may cause artifacts to appear in the image after binning.

[0009] Therefore, this specification discloses an imaging apparatus and an imaging method that can suppress the bias of the pixel centroid after binning processing more effectively than conventional methods. [Means for solving the problem]

[0010] The imaging apparatus disclosed herein comprises a pixel array, a color filter array, and a binning processing unit. The pixel array has a two-dimensional arrangement of multiple pixels. The color filter array is arranged on the pixel array. The color filter array also has RGB color filters arranged in a Bayer array. The binning processing unit performs k × k (where k is an integer) binning. That is, the binning processing unit performs binning on a pixel group consisting of k pixels vertically and k pixels horizontally in the pixel array. The binning processing unit is capable of performing uniform binning. In uniform binning, signals from at least one pixel of each color located relatively towards the center of the pixel group are excluded from processing.

[0011] With the above configuration, the uniform binning process makes the distance between pixel centroids after binning more uniform than before, as shown in Figure 6, which will be described later.

[0012] Furthermore, in the above configuration, the binning processing unit may be able to switch between normal binning, which uses signals from all pixels in the pixel group, and uniform binning.

[0013] With the above configuration, normal binning is selected when prioritizing the brightness and low noise of pixels after binning. Uniform binning is selected when prioritizing the reduction of artifacts. Thus, the above configuration allows for binning tailored to the specific requirements of the image.

[0014] Furthermore, in the above configuration, the binning processing unit may perform a 2x2 binning process. In this case, two row selection wirings are provided for each row of the pixel array. Also, for multiple pixels in the pixel array, the connection destination is switched between one row selection wiring and the other row selection wiring at intervals of two pixels along the row direction.

[0015] With the above configuration, uniform binning can be performed on the analog circuit.

[0016] Furthermore, in the above configuration, four column signal lines may be provided for each row of the pixel array. In this case, the color filter array has two color filters arranged alternately along the column direction. Also, a column signal line is assigned to each of the two color filters. Moreover, for a pair of pixel groups along the column direction, the column signal line connected to one pixel group is different from the column signal line connected to the other pixel group.

[0017] According to the above configuration, pixel signals can be acquired simultaneously from a pair of pixel groups aligned in the column direction.

[0018] Furthermore, in the above configuration, during video capture, the binning processing unit may shift multiple pixels that are grouped in a predetermined frame two-dimensionally between the pixel group in that frame and the pixel group in the next frame.

[0019] With the above configuration, the positions of artifacts occurring in a given frame and artifacts occurring in the next frame are shifted. As a result, artifacts are visually mitigated when the video is displayed.

[0020] Furthermore, in the above configuration, the imaging device may include a display and a tracking mechanism. The display shows an image formed by a pixel array. The eye-tracking mechanism measures the line of sight directed towards the display. The binning processing unit performs binning on areas outside the point of gaze fixation.

[0021] According to the above configuration, image processing is performed based on so-called foveal rendering, which reduces the computational load of image processing.

[0022] In addition, this specification discloses an imaging method. This imaging method is executed on an imaging device. The imaging device includes a pixel array and a color filter array. In the pixel array, a plurality of pixels are two-dimensionally arranged. The color filter array is disposed on the pixel array. In the color filter array, RGB color filters are arranged in a Bayer pattern. In such an imaging device, a binning process of k×k (where k is an integer) is performed. That is, a binning process is performed in which k pixels in the vertical direction and k pixels in the horizontal direction in the pixel array are grouped into one pixel group. Here, as the binning process, an equal binning process is executed in which signals from at least one pixel of each color, which are relatively arranged on the central side of the pixel group, are excluded from the processing target.

[0023] In addition, in the above configuration, as the binning process, a normal binning process using signals from all pixels of the pixel group and an equal binning process may be switchable.

[0024] In addition, in the above configuration, in video imaging, a plurality of pixels grouped in a pixel group in a predetermined frame and a pixel group in the next frame may be two-dimensionally shifted.

[0025] In addition, in the above configuration, the imaging device may include a display and a tracking mechanism. An image by the pixel array is displayed on the display. The eye tracking mechanism measures the line of sight directed at the display. Further, an equal binning process is executed for a region outside the fixation point of the line of sight.

Advantages of the Invention

[0026] According to the imaging device and imaging method disclosed in this specification, the deviation of the pixel center of gravity after the binning process can be suppressed more than before.

Brief Description of the Drawings

[0027] [Figure 1] It is a diagram illustrating an imaging system according to this embodiment. [Figure 2] This figure illustrates a pixel array based on a Bayer array. [Figure 3] This is a diagram (1 / 4) illustrating the uniform binning process. [Figure 4] This is a diagram (2 / 4) illustrating the uniform binning process. [Figure 5] This is a diagram (3 / 4) illustrating the uniform binning process. [Figure 6] This is a diagram (4 / 4) illustrating the uniform binning process. [Figure 7] This figure compares an image obtained by uniform binning (horizontal line image) with an image obtained by conventional normal binning. [Figure 8] This figure compares an image obtained by uniform binning (vertical line image) with an image obtained by conventional normal binning. [Figure 9] This diagram illustrates the signal-to-noise ratio (S / N ratio) in uniform binning and other image processing techniques, depending on the brightness (luminosity) of the subject. [Figure 10] This figure illustrates the configuration of the pixel array according to this embodiment. [Figure 11] This is a diagram (1 / 3) illustrating the uniform binning process using the pixel array according to this embodiment. [Figure 12] Figure (2 / 3) illustrates the uniform binning process using the pixel array according to this embodiment. [Figure 13] Figure (3 / 3) illustrates the uniform binning process using the pixel array according to this embodiment. [Figure 14] This diagram illustrates group shift in uniform binning. [Figure 15] This diagram illustrates the visual effects of group shifting. [Figure 16] This figure illustrates the range to which uniform binning is applied in image processing based on foveal rendering. [Figure 17] This is a diagram illustrating a standard 3x3 binning. [Figure 18] This is a diagram illustrating 3x3 even binning. [Figure 19] Figure 18 illustrates 3x3 uniform binning when unused pixels are changed. [Figure 20] This is a diagram illustrating standard 4x4 binning. [Figure 21] This is a diagram illustrating 4x4 even binning. [Figure 22] Figure 21 is a diagram illustrating 4x4 uniform binning when unused pixels are changed. [Figure 23] This is a diagram (1 / 3) illustrating the binning process related to conventional technology. [Figure 24] This is a diagram (2 / 3) illustrating the binning process related to conventional technology. [Figure 25] This is a diagram (3 / 3) illustrating the binning process related to conventional technology. [Modes for carrying out the invention]

[0028] The imaging apparatus and imaging method according to this embodiment will be described below with reference to the drawings. The shapes, materials, quantities, and numerical values ​​described below are illustrative examples for illustrative purposes. These shapes, etc., can be appropriately changed according to the specifications of the imaging apparatus. In addition, the same reference numerals are used for equivalent elements in all drawings below.

[0029] 1. Configuration of the imaging system Figure 1 illustrates an imaging system according to this embodiment. This system comprises an imaging unit 10, an image processing unit 30, a display device 40, an eye-tracking mechanism 45, and an input unit 47. The imaging device according to this embodiment is configured to include the imaging unit 10 and the image processing unit 30.

[0030] The imaging unit 10 includes a pixel array 12, a color filter array 14, a horizontal scanning circuit 16, a CDS-ADC circuit 18, and a vertical scanning circuit 20. For example, the imaging unit 10 is a CMOS image sensor.

[0031] The pixel array 12 contains multiple pixels arranged in a two-dimensional array. For example, as shown in Figure 10, which will be described later, multiple pixels are arranged in both the row and column directions.

[0032] The color filter array 14 is placed on the pixel array 12. The color filter array 14 has RGB color filters arranged in a two-dimensional array. For example, the color filter array 14 has RGB color filters arranged in a two-dimensional array according to the Bayer array.

[0033] Figure 2 illustrates the arrangement of the Bayer-arranged color filter array 14. For convenience, the green (Gb) in the blue (B) row and the green (Gr) in the red (R) row are shown with different signs. Strictly speaking, Figure 2 does not represent the color filter array 14 itself, but rather the pixel arrangement beneath the Bayer-arranged color filter array 14. For example, the pixel at coordinate (0,0) outputs a Gb signal (pixel value).

[0034] The horizontal scanning circuit 16 is a circuit that selects the readout row of the pixel array 12. As shown in Figure 10, which will be described later, row selection wiring RS (RS1a~RS4a, RS1b~RS4b) extends from the horizontal scanning circuit 16 into the pixel array 12.

[0035] The CDS-ADC circuit 18 holds the signal (voltage value) of each pixel in the pixel array 12 and performs analog-to-digital conversion (A / D conversion). The mechanisms of signal holding and A / D conversion are known, so a detailed explanation is omitted.

[0036] The vertical scanning circuit 20 instructs the CDS-ADC circuit 18 which column of the pixel array 12 to read.

[0037] The image processing unit 30 comprises an imaging signal acquisition unit 32, a binning processing unit 34, and a display image generation unit 36. The image processing unit 30 is composed of, for example, a computer. That is, these functional units are composed of the computer's CPU and memory working together.

[0038] The imaging signal acquisition unit 32 acquires digitally converted pixel values ​​from the CDS-ADC circuit 18. The pixel values ​​acquired by the imaging signal acquisition unit 32 are sent to the binning processing unit 34 and the display image generation unit 36. For example, as will be described later, the values ​​in the region where the pixel value is below a predetermined threshold are sent to the binning processing unit 34. The values ​​in the region where the pixel value is above the predetermined threshold are sent to the display image generation unit 36. Details of the binning process will be described later.

[0039] The display image generation unit 36 ​​receives the pixel values ​​after binning and the pixel values ​​of each pixel in the pixel array 12. The display image generation unit 36 ​​generates image data from these pixel values. The generated image data is sent to the display device 40, and the image is then displayed on the display device 40.

[0040] The eye-tracking mechanism 45 is provided in the imaging system for the purpose of performing foveal rendering, which will be described later. The eye-tracking mechanism 45 detects the gaze of a user viewing an image displayed on a display device 40, for example.

[0041] The input unit 47 is a user interface such as a touch panel, keyboard, or mouse. As will be described later, the user can select between uniform binning (Figures 3-6) and normal binning (Figures 23-25) through operations on the input unit 47. In other words, the binning processing unit 34 can switch between normal binning and uniform binning by operating the input unit 47.

[0042] 2. Principle of Uniform Binning Figure 2 illustrates a portion of the pixel array 12. To identify each pixel, numbers are assigned in both the row and horizontal directions. For example, the pixel position in the upper left corner is represented as (0,0), i.e., row 0, column 0, similar to XY coordinates. Figure 2 also shows the array of pixels that output signals for each of the RGB colors.

[0043] Generally, binning is an image processing technique used in dark imaging environments to improve image clarity by reducing resolution. For example, binning is performed on a k × k scale (where k is an integer). This means that k pixels in the vertical direction and k pixels in the horizontal direction are combined into a single pixel group. This binning process is performed by the binning processing unit 34 of the image processing unit 30 (see Figure 1).

[0044] For example, Figures 2-6 show a 2x2 binning process (k=2). In a 2x2 binning process, for example, for pixel Gb, the four pixels (0,0), (2,0), (0,2), and (2,2) are combined into a single pixel group.

[0045] In aggregated pixel groups, the individual pixel values ​​are averaged or added together. Averaging reduces noise (smooths out variations). Addition increases the pixel value (luminance) within the pixel group.

[0046] In this embodiment, the binning processing unit 34 is capable of performing uniform binning. In uniform binning, as illustrated in Figure 6, the pixel centroids after binning (G_Gb1~G_GB4, G_B1~G_B4, G_Gb1~G_Gb4, G_R1~G_R4) are distributed more evenly in a geometrically uniform manner compared to conventional normal binning (see Figure 25).

[0047] Figures 3-6 illustrate a 2x2 uniform binning process. The uniform binning process is performed by the binning processing unit 34 (see Figure 1). In the 2x2 binning process, four types of pixels are processed for each of the colors Gb, B, Gr, and R. For convenience, Gb and Gr will be described below as different colors (or different types).

[0048] Referring to Figure 2, the 64 pixels are divided into four pixel groups by a 2x2 binning process. The pixel groups are indicated by thick rectangles. Referring to Figure 3, in the uniform binning process, the binning processing unit 34 performs the binning process while excluding signals from at least one pixel of each color located relatively towards the center of the pixel group from the processing target.

[0049] In other words, the pixel signals shown by the hatched lines in Figure 3 are discarded and not used in the binning process. In this uniform binning process, Figure 4 shows the geometric centroids G_Gb1 to G_Gb4 of pixel Gb and the geometric centroids G_Gr1 to G_Gr4 of pixel Gr. Figure 5 shows the geometric centroids G_B1 to G_B4 of pixel B and the geometric centroids G_R1 to G_R4 of pixel R.

[0050] As shown in Figures 4 and 5, in a 2x2 uniform binning process, the three outermost pixels of each color are selected within a pixel group. The geometric centroid is then equal to the centroid of the triangle formed by connecting the centers of the three outermost pixels.

[0051] Figure 6 schematically shows the pixel arrangement after uniform binning. The spacing between the centroids of the pixels, G_B1~G_B4, G_R1~G_R4, G_Gb1~G_Gb4, and G_Gr1~G_Gr4, which have been theoretically extended by the uniform binning process, is more uniformly distributed compared to conventional binning (normal binning) as shown in Figure 25.

[0052] For example, comparing Figure 6 and Figure 25, the distances between the centroids d11 and d13 within a pixel group in uniform binning are wider than the distances between the centroids d1 and d3 within a pixel group in normal binning. Also, the distances between the centroids d12 and d14 between pixel groups in uniform binning are narrower than the distances between the centroids d2 and d4 between pixel groups in normal binning.

[0053] In this way, the centroids of the expanded pixels are evenly distributed, suppressing artifacts in the image. Figure 7 shows an example image of a bundle of lines extending mainly horizontally. Figure 8 shows an example image of a bundle of lines extending mainly vertically. In these images, one type of artifact (image distortion) is line image drift.

[0054] In both Figures 7 and 8, the image on the left is an image obtained using normal binning. The image in the center is an image that has been corrected after normal binning has been performed on the image sensor. The image on the right is an image obtained using uniform binning. Comparing these three images, it can be seen that artifacts are reduced in the image obtained using uniform binning. In the image in the center, weighting is applied to the target pixels after normal binning. For example, for pixel B, the corrected value h is applied in a 2x2 digital filter process. B The correction is performed as follows: =(49×a+7×b+7×c+1×d) / 64. Here, the top-left pixel of the 2×2 pixel group is a, the top-right pixel is b, the bottom-left pixel is c, and the bottom-right pixel is d. Similarly, for pixel Gb, the corrected value h is applied. Gb A correction is performed using the formula =(35×a+21×b+5×c+3×d) / 64. Furthermore, the corrected value h is applied to pixel Gr. Gr A correction is performed using the formula =(35×a+5×b+21×c+3×d) / 64. Furthermore, the corrected value h is applied to pixel R. R The correction is performed using =(25×a+15×b+15×c+9×d) / 64.

[0055] For pixels subject to uniform binning, the binning processing unit 34 (see Figure 1) averages or adds the respective pixel values. For example, referring to Figure 4, the binning processing unit 34 adds the respective pixel values ​​of pixel Gb(0,0), pixel Gb(2,0), and pixel Gb(0,2). Adding the pixel values ​​increases the brightness value of the expanded pixel. Alternatively, the binning processing unit 34 calculates the average value of the respective pixel values ​​of pixel Gb(0,0), pixel Gb(2,0), and pixel Gb(0,2). Calculating the average value reduces noise in the expanded pixel.

[0056] As illustrated in Figure 3, in uniform binning, the signal of at least one pixel is intentionally excluded from processing and discarded. This means that the accuracy of averaging, or in other words, the improvement in the signal-to-noise ratio, may be limited compared to using the signals of all pixels. Figure 9 shows examples of signal-to-noise ratios for different binning processes. The horizontal axis represents the luminosity of the imaging environment [cd / m²]. 2 The luminosity is shown. On the horizontal axis, the luminosity increases as you move to the right. The vertical axis shows the signal-to-noise ratio (S / N ratio). On the vertical axis, the S / N ratio improves as you move upwards.

[0057] Figure 9 shows characteristic curves for four types of images. Specifically, the signal-to-noise ratio (S / N ratio) for four types of images—normal binning, uniform binning, digital weighted binning, and no binning (4C)—is illustrated in Figure 9. In digital weighted binning, referring to Figure 2, the pixel value at coordinate (0,0) is multiplied by a weight of 2.25 for pixel Gb. The pixel values ​​at coordinates (0,2) and (2,0) are multiplied by a weight of 0.75. Furthermore, the pixel value at coordinate (2,2) is multiplied by a weight of 0.25.

[0058] Referring to the characteristic curve in Figure 9, it can be seen that there is no significant difference in the signal-to-noise ratio between normal binning and uniform binning, especially in dark environments.

[0059] Thus, in the uniform binning process according to this embodiment, signals from at least one pixel of each color located relatively towards the center of the pixel group are excluded from processing. However, the excluded pixels may be used in processes other than uniform binning, such as phase difference detection.

[0060] 3. Circuit configuration that enables analog uniform binning The uniform binning process according to this embodiment can be performed by calculating the digital value after A / D conversion. In addition, the uniform binning process according to this embodiment can be partially performed in the stage prior to A / D conversion, that is, in the analog stage.

[0061] Figure 10 illustrates a circuit configuration that enables analog uniform binning. In this figure, each pixel is numbered, corresponding to the coordinate representation in Figure 2. For example, pixel Gb00 in the upper left corner of the screen corresponds to pixel Gb at coordinate (0,0), i.e., row 0, column 0. Similarly, pixel Gr77 in the lower right corner of the screen corresponds to pixel Gr at coordinate (7,7).

[0062] In this diagram as well, each pixel in the pixel array 12 is arranged based on the Bayer array. That is, two-colored pixels are arranged alternately along the row direction, and two-colored pixels are also arranged alternately along the column direction. However, for convenience, Gb and Gr are treated as different colors.

[0063] Row selection wiring extends from the horizontal scanning circuit 16 into the pixel array 12. In the pixel array 12 shown in Figure 10, two row selection wirings are provided for each row of pixels. For example, row selection wirings RS1a and RS1b are provided for the first row of pixels. Similarly, row selection wirings RSna and RSnb are provided for the nth row of pixels (where n is an integer).

[0064] Multiple pixels in the pixel array 12 are connected to either row selection wiring RSna or row selection wiring RSnb every two pixels along the row direction. For example, referring to the first row, pixels B10 and Gb20 are connected to row selection wiring RS1b, and pixels B30 and Gb40 are connected to row selection wiring RS1a.

[0065] Furthermore, for each row of pixels, the row selection wiring to which they are connected is standardized to either row selection wiring RSna or row selection wiring RSnb. For example, all pixels in the first row are connected to row selection wiring RSna (RS1a, RS2a, RS3a, RS4a, RS5a, RS6a, RS7a, RS8a).

[0066] 4. Parallel Processing Structure The pixel array 12 illustrated in Figure 10 can simultaneously read out signals from pixel groups along the column direction. Referring to Figure 10, four column signal lines are provided for each column of the pixel array 12. That is, for the m-th column (where m is an integer) of the pixel array 12, four column signal lines Cma, Cmb, Cmc, and Cmd are provided. The CDS element and ADC element of the CDS-ADC circuit 18 (see Figure 1) are connected to each of these column signal lines Cma, Cmb, Cmc, and Cmd.

[0067] Furthermore, in a Bayer array, two color filters are arranged alternately along the column direction. A column signal line is then assigned to each of the two color filters. For example, in pixel group A1 in Figure 10, pixels Gb00 and Gb02 are both connected to column signal line C1d. Also, pixels R01 and R03 are both connected to column signal line C1b.

[0068] Furthermore, for pixel group A2, which is paired with pixel group A1 along the column direction, pixels Gb04 and Gb06 are both connected to column signal line C1c. Also, pixels R05 and 07 are both connected to column signal line C1a.

[0069] Thus, in the pixel array illustrated in Figure 10, column signal lines are assigned to each of the two color filters. Furthermore, for a pair of pixel groups aligned in the column direction, the column signal lines connected to one pixel group are different from those connected to the other pixel group. In other words, the column signal lines do not overlap for each color within a pixel group. Moreover, the column signal lines do not overlap for pixel groups aligned in the column direction.

[0070] By having such column signal lines and connection configurations, it is possible to simultaneously read out signals from pixel groups along the column direction, as will be described later.

[0071] 5. Analog Equalization Binning Process Figure 11 shows an overview of the uniform binning process. Referring to the dashed rectangular frame in Figure 11, the pixel groups are divided into four, similar to Figure 3. Pixels located relatively towards the center of each pixel group are indicated by diagonal hatching. The signals of pixels included in this hatching are excluded (discarded) during the uniform binning process.

[0072] The specific circuit operation is illustrated in Figure 12. This figure illustrates the operation of pixel groups A1 and A2. However, due to the symmetry of the circuit, the remaining pixel groups are processed in the same way as pixel groups A1 and A2.

[0073] During the uniform binning process, the binning processing unit 34 (see Figure 1) instructs the horizontal scanning circuit 16 to turn on the row selection wiring. In Figure 12, the row selection wiring that is turned on is underlined.

[0074] Specifically, for the upper row (1st row) and lower row (4th row) of pixel group A1, a pair of row selection wires RS1a, RS1b, RS4a, and RE4b are turned on. Also, for the middle row (2nd and 3rd rows) of pixel group A1, only row selection wires RS2a and RS3a are turned on. As a result, the signals of pixels Gr11, R21, B12, and Gb22, which are located in the center of pixel group A1, are not read.

[0075] Similarly, for the upper row (5th row) and lower row (8th row) of pixel group A2, a pair of row selection wires RS5a, RS5b, RS8a, RS8b are turned on. Also, for the middle row (6th and 7th rows) of pixel group A2, only row selection wires RS6a and RS7a are turned on. As a result, the signals of pixels Gr15, R25, B16, and Gb26, which are located in the center of pixel group A2, are not read.

[0076] When performing normal binning instead of uniform binning, a pair of row selection wires RSna and RSnb (where n is an integer) are turned on for all rows during row reading.

[0077] Regarding the uniform binning process, as described above, the connected column signal lines C1a, C1b, C1c, and C1d of pixel groups A1 and A2 do not overlap. Therefore, the readout of pixel groups A1 and A2 is performed simultaneously.

[0078] Focusing on the first column, the signals from pixels Gb00 and Gb02 are simultaneously sent to the column signal line C1d. Since the pixel signal is a voltage signal, and pixels Gb00 and Gb02 are in parallel in the circuit, the average voltage of pixels Gb00 and Gb02 is output from the column signal line C1d. In other words, the averaging of the pixel values ​​is performed on the analog circuit. Similarly, when pixels of the same color are simultaneously sent to a single column signal line, the average value of the voltages of these pixels is output from the column signal line.

[0079] Furthermore, focusing on the second and third columns, each column's signal line receives a signal from a single pixel.

[0080] The signal sent to the column signal line is transmitted to the CDS-ADC circuit 18 (see Figure 1). The signal (pixel value) converted to a digital value via the CDS-ADC circuit 18 is then subjected to uniform binning by the binning processing unit 34.

[0081] For example, the signals (pixel values) output from the first and fourth columns of the pixel array 12 are doubled because they contain signals from two pixels. The signals (pixel values) from the second and third columns of the pixel array 12 are processed at their original size. Referring to Figure 13, for example, for pixel group A1, the signal output from column signal line C1d is doubled. Furthermore, the signal output from column signal line C3d is output at its original size. These signals are added together to obtain the expanded pixel A1_Gb value in pixel group A1.

[0082] Thus, in the addition process, the pixel values ​​of pixels of the same color within a pixel group are added together. On the other hand, in the averaging process, the average value is obtained by dividing the value after the addition process by the number of pixels to be binned (for example, 3 pixels Gb00, Gb20, and Gb02).

[0083] 6. Even binning process during video recording In this embodiment, the imaging device may switch the pixel group framework for each frame during video capture. Figure 3 illustrates the pixel groups in a predetermined frame. Figure 14 illustrates the pixel groups in the next frame.

[0084] Comparing Figure 3 and Figure 14, the binning processing unit 34 (see Figure 1) sets each pixel group so that unused pixels (pixels whose signals are discarded) are complementary. For example, in Figure 14, the pixel groups are set shifted by two rows and two columns from Figure 3. By alternating between the groupings in Figure 3 and Figure 14, visual artifacts are reduced.

[0085] Figure 15 illustrates an image with uniform binning applied to a given frame (based on Figure 3) (left side of the page) and an image with uniform binning applied to the next frame (based on Figure 14) (center of the page). As is particularly clear from the lower image, by differentiating the pixel grouping between frames, the locations where artifacts occur become complementary. For example, even if an artifact occurs in a certain area of ​​one image, no artifact occurs in the same area of ​​the other image. Therefore, in a video displaying these images sequentially, artifacts are visually reduced, as shown in the right-hand figure.

[0086] 7. Uniform binning process based on foveal rendering Generally, the feasibility of binning is determined based on the brightness of the pixel values. In addition, uniform binning and normal binning may be performed based on so-called foveal rendering.

[0087] Referring to Figure 1, the imaging system includes an eye-tracking mechanism 45. The eye-tracking mechanism 45 tracks the gaze of the user viewing the image displayed on the display device 40. For example, the eye-tracking mechanism 45 includes a near-infrared light source and a camera. In other words, so-called non-contact eye tracking is performed by the eye-tracking mechanism 45.

[0088] The eye-tracking mechanism 45 identifies the user's point of focus. The binning processing unit 34 performs uniform binning on areas outside the point of focus. For example, even for pixels with brightness exceeding the threshold mentioned above, the binning processing unit 34 performs uniform binning. In other words, by intentionally lowering the resolution of images outside the point of focus, the computational load related to image display is reduced.

[0089] Referring to Figure 16, for example, if the central part in the height direction of the image displayed on the display device 40 is the gaze area 42, then the areas above and below it become the non-gaze area 44. The binning processing unit 34 performs uniform binning on the pixel groups corresponding to the non-gaze area 44.

[0090] 8.3 x 3 binning In the embodiment described above, a 2x2 binning process was exemplified as a kxk binning process. However, the binning processing unit 34 according to this embodiment can also handle other binning processes.

[0091] Figure 17 illustrates the process of normal 3x3 binning. In this example, 3 pixels are selected both vertically and horizontally for each color. The size of the pixel group is 6x6. Figure 17 illustrates the centroids G_Gb, G_B, G_R, and G_Gr of the expanded pixels after normal binning. As shown in this figure, the centroids G_Gb, G_B, G_R, and G_Gr are all positioned towards the center of the pixel group.

[0092] Figure 18 shows an example of 3x3 uniform binning. In this example as well, signals from at least one pixel of each color located relatively towards the center of the pixel group are excluded from processing (not used). For example, for pixel Gb, the pixels at coordinates (4,0), (0,4), (4,2), (2,4), and (4,4) are unused pixels. Similarly, for pixel B, the pixels at coordinates (1,0), (1,2), (1,4), (3,4), and (5,4) are unused pixels. For pixel R, the pixels at coordinates (0,1), (2,1), (4,1), (4,3), and (4,5) are unused pixels. Furthermore, for pixel Gr, the pixels at coordinates (1,1), (3,1), (5,1), (1,3), and (1,5) are unused pixels. By performing this uniform binning process, the centroids of each color are distributed more evenly compared to the centroids in normal binning (see Figure 17), as shown by the centroids G_Gb, G_B, G_R, and G_Gr.

[0093] Figure 19 shows an example where unused pixels are selected differently from those in Figure 18. In this example, the pixels at coordinates (1,0), (4,0), (0,1), (5,1), (0,4), (5,4), (1,5), and (4,5), which were unused pixels in the example in Figure 18, are subject to binning (become used pixels).

[0094] As a result, in the example in Figure 19, the centroids of each color are shifted towards the center compared to Figure 18, but are distributed more evenly compared to the normal binning process shown in Figure 17. Furthermore, in the example in Figure 19, fewer signals are discarded compared to Figure 18, indicating a higher efficiency in addition and averaging through binning.

[0095] 9.4 x 4 binning Figure 20 illustrates the process of normal 4x4 binning. In this example, 4 pixels are selected both vertically and horizontally for each color. The size of the pixel group is 8x8. Figure 20 illustrates the centroids G_Gb, G_B, G_R, and G_Gr of the expanded pixels after normal binning. As shown in this figure, the centroids G_Gb, G_B, G_R, and G_Gr are all positioned towards the center of the pixel group.

[0096] Figure 21 shows an example of 4x4 uniform binning. In this example as well, signals from at least one pixel of each color located relatively towards the center of the pixel group are excluded from processing (not used). For example, for pixel Gb, the pixels at coordinates (6,0), (6,2), (4,4), (6,4), (0,6), (2,6), (4,6), and (6,6) are unused pixels. Similarly, for pixel B, the pixels at coordinates (1,0), (1,2), (1,4), (3,4), (1,6), (3,6), (5,6), and (7,6) are unused pixels. And for pixel R, the pixels at coordinates (0,1), (2,1), (4,1), (6,1), (4,3), (6,3), (6,5), and (6,7) are unused pixels. Furthermore, for pixel Gr, the pixels at coordinates (1,1), (3,1), (5,1), (7,1), (1,3), (3,3), (1,5), and (1,7) become unused pixels. By performing this uniform binning process, the centroids of each color are distributed more evenly compared to the centroids in normal binning (see Figure 20), as shown in the centroids G_Gb, G_B, G_R, and G_Gr.

[0097] Figure 22 shows an example where different unused pixels are selected compared to Figure 21. In this example, the pixels at coordinates (1,0), (6,0), (0,1), (7,1), (0,6), (7,6), (1,7), and (6,7), which were unused pixels in Figure 21, become the pixels to be binned (used pixels). On the other hand, the pixels at coordinates (2,3), (5,3), (2,4), and (5,4), which were used pixels in Figure 21, become unused pixels.

[0098] As a result, in the example in Figure 22, the centroids of each color are distributed to a similar extent as in Figure 21. Furthermore, in the example in Figure 22, fewer signals are discarded compared to Figure 21, resulting in a higher efficiency for addition and averaging using binning. [Explanation of symbols]

[0099] 10 Imaging unit, 12 Pixel array, 14 Color filter array, 16 Horizontal scanning circuit, 18 CDS-ADC circuit, 20 Vertical scanning circuit, 30 Image processing unit, 32 Imaging signal acquisition unit, 34 Binning processing unit, 36 Display image generation unit, 40 Display device, 42 Gaze area, 44 Non-gaze area, 45 Eye tracking mechanism, 47 Input unit.

Claims

1. A pixel array in which multiple pixels are arranged in a two-dimensional array, A color filter array is arranged on the aforementioned pixel array, and the RGB color filters are arranged in a Bayer array. A binning processing unit performs a k × k binning operation on the aforementioned pixel array, where k pixels in the vertical direction and k pixels in the horizontal direction are grouped into one pixel group (where k is an integer), Equipped with, The binning processing unit is capable of performing uniform binning, in which signals from at least one pixel of each color, located relatively towards the center of the pixel group, are excluded from processing. Imaging device.

2. The imaging apparatus according to claim 1, The binning processing unit can switch between a normal binning process that uses signals from all pixels in the pixel group and a uniform binning process. Imaging device.

3. An imaging device according to claim 1, The binning processing unit performs the 2x2 binning process, Two row selection wirings are provided for one row of the aforementioned pixel array. The multiple pixels of the pixel array are configured such that, every two pixels along the row direction, their connection destination is switched between one row selection wiring and the other row selection wiring. Imaging device.

4. An imaging device according to claim 3, Four column signal lines are provided for each row of the aforementioned pixel array. The aforementioned color filter array has two color filters arranged alternately along the column direction. The column signal lines are assigned according to the color of the two color filters. For a pair of pixel groups along the column direction, the column signal line connected to one of the pixel groups and the column signal line connected to the other pixel group are different. Imaging device.

5. An imaging device according to claim 1, In video capture, the binning processing unit shifts a plurality of pixels that are grouped together in a two-dimensional manner between the pixel group in a predetermined frame and the pixel group in the next frame. Imaging device.

6. An imaging device according to claim 1, A display on which an image is shown by the aforementioned pixel array, An eye-tracking mechanism for measuring the gaze directed towards the aforementioned display, Equipped with, The binning processing unit performs the binning process on an area outside the point of gaze of the line of sight. Imaging device.

7. A pixel array in which multiple pixels are arranged in a two-dimensional array, A color filter array is arranged on the aforementioned pixel array, and the RGB color filters are arranged in a Bayer array. In an imaging device equipped with, In the aforementioned pixel array, k pixels in the vertical direction and k pixels in the horizontal direction are treated as one pixel group (where k is an integer), and a k × k binning process is performed. As the binning process, a uniform binning process is performed in which signals from at least one pixel of each color, located relatively towards the center of the pixel group, are excluded from the processing. Imaging method.

8. The imaging method according to claim 7, The binning process can be switched between a normal binning process that uses signals from all pixels in the pixel group and a uniform binning process. Imaging method.

9. The imaging method according to claim 7, In video capture, the pixel group in a predetermined frame and the pixel group in the next frame are shifted two-dimensionally. Imaging method.

10. The imaging method according to claim 7, The imaging device is A display on which an image is shown by the aforementioned pixel array, An eye-tracking mechanism for measuring the gaze directed towards the aforementioned display, Equipped with, The binning process is performed on the region outside the point of gaze. Imaging method.

Images